Discover RealityKit APIs for iOS, macOS, and visionOS

Hello, my name is Yidi, I'm an engineer on the RealityKit team. In this session, I'll show you some new RealityKit APIs for developing spatial computing apps.

RealityKit is a framework that provides high-performance 3D simulation and rendering capabilities for your apps, on iOS, iPadOS, macOS and visionOS. It offers a wide variety of capabilities for your 3D content to blend seamlessly with the real-world environment, enabling you to create immersive spatial computing apps and games. Since the launch of Apple Vision Pro, we have been getting amazing feedback from all of you. I sincerely want to thank you for all those great suggestions and feature requests, and today, I’m really excited to share that a lot of that feedback has made it into RealityKit as brand new features for you. In this session, let’s use some of those features to build a Spaceship game. You can download the sample project linked to this session and follow along. Our game recreates the feeling of kids playing with their favorite spaceship toy. The game starts in a hangar. It's a safe place where the player can inspect their spaceship up close. Then, the player learns how to fly the spaceship in their physical environment, using both of their hands as input. After getting familiar with flight controls, the player can then can fly out to the outer space, navigating through asteroid fields, and deliver precious space cargo. They can even fly through a portal that takes them to distant planets. I will take you through a series of new RealityKit APIs as we build this game. First, we will look into using hover effects for interacting with our 3D models, and then use hand tracking to build the core input mechanic.

Next, I will add physics simulation to our game using force effects and joints.

Then, I will add dynamic lights to illuminate the scene and use shadows to help the player perceive distance.

After that, I will use the portal API to open up a portal into far away outer space and take advantage of the new portal enhancements to fly our spaceship there.

Finally, I will talk about RealityKit's cross-platform capabilities and how we can easily bring our game to other Apple platforms. It's going to be an exciting journey as we go through all of these. Let's begin right away! I will start with hover effects and input.

On visionOS, you can display your app's content in a window, a volume, or space. Our game starts by showing the spaceship in the hangar with the volume style.

The player can inspect the spaceship up close by rotating it around using a simple drag gesture. The system also provides a base plate for this volume, which shows visual feedback to where the player is looking. You may notice though that, unlike the base plate, the spaceship does not show any visual highlights when the player looks at it. Let's add a HoverEffectComponent to the spaceship so it displays a subtle spotlight when it's looked at. It makes the interaction feel more responsive. But what if we want to customize that effect to match our game's art style? I’ve got some good news for you! This year, RealityKit introduces new APIs that will allow us to customize our hover effects. In addition to the default spotlight effect we just saw, we have two brand new styles to use: highlight and shader. The new highlight style applies a uniform highlight to the entire mesh. When you use the spotlight or highlight style, you can also customize their tint color and strength, and the new shader style unlocks more possibilities by integrating with a shader graph material.

Let's try adding a highlight effect with a tint color, to make it fit with the spaceship's look. I will also adjust the strength so it's visible without being intrusive.

To do this, I will create a HighlightHoverEffectStyle and set its color to an artist-defined yellow and set its strength to 0.8.

Then, I’ll create a HoverEffectComponent with this style and add it to the spaceship entity. That’s how simple it is to add the new highlight hover effect to your entities.

While the new customizations with spotlight and highlight offer a quick and easy way to add hover effects to your 3D content, the new shader style is even more powerful and flexible, which lets you use a shader graph material in Reality Composer Pro to compose advanced visual effects. Using this capability, I have added a custom hover effect that lights up the spaceship's window when you look at it. That looks great and fits really well with the spaceship model. To achieve this effect, I have used the new HoverState node in shader graph. The HoverState node provides several inputs, including an intensity value. The intensity value is animated from 0 to 1 when the player looks at the spaceship. The spaceship's window already has a shader graph material set up in Reality Composer Pro. I added the HoverState node to it and supplied the intensity value to a Mix node so that we can fade in the spaceship's emissive color as the player looks at it.

Finally, we can connect the output of the Mix node to emissive color of the PBRSurface.

In code, all we need to do is update the hover style to shader, with default inputs.

There are a variety of effects you can achieve with the HoverState node. For example, my colleague Adrian built a spatial drawing app, and he used a shader style hover effect to animate a sweeping purple glow along a 3D brush stroke. For more details on this sample, do watch the session “Build a spatial drawing app with RealityKit” from this year’s WWDC. HoverEffectComponent offers powerful ways to introduce rich visual feedback when a 3D entity is looked at, and it does all of this without compromising people’s privacy.

This year, SwiftUI is also introducing custom hover effects APIs. Check out the session “Create custom hover effects in visionOS” to learn more.

Next, let's define the game's input mechanic.

When the player is ready to fly the spaceship around, they exit the volume and enter a mixed immersive space.

Here, the player controls the spaceship using both their hands. Their left hand controls how fast the spaceship flies. As they bring their thumb and index fingertips closer, the spaceship flies faster. Their right hand controls the spaceship's orientation. When right hand is tilted up, the spaceship also tilts up. And when it is turned to the left, the spaceship also turns to the left.

To implement this input mechanic, we need to track the player's hands in 3D space.

With visionOS 1, it was possible to add custom hand tracking to your app with ARKit.

This year, RealityKit is introducing the new Spatial Tracking API to make this task even easier.

A SpatialTrackingSession helps us get the player's consent for hand tracking, and once it is set up, we can track the player's hands using RealityKit anchor entities. For the left hand, we create two anchor entities: one for index finger tip and one for thumb tip.

Inside of our custom system's update function, we query the position of the anchor entities using the Entity.position API.

We compute the distance between the two anchor entities, and we map the distance to a throttle value through a custom function, where a shorter distance produces a larger throttle.

Finally, we can accelerate the spaceship with this throttle value. We compute a force vector along the spaceship's forward direction and add the force to the spaceship entity.

For the right hand control, the implementation is similar to the code we just saw. Check out the sample code linked to this session for the detailed implementation.

Now, with both our left and right-hand custom gestures in place, the player can fly the spaceship using their hands as input! Nice! You can also learn more about SpatialTrackingSession API in Adrian's session "Build a spatial drawing app with RealityKit." Now that the player has a spaceship they can fly, let’s make their pilot experience a little more interesting. We will learn how to use new force effects and physics joints APIs as we do this.

To help the player practice their pilot skills, I have added planets and asteroids in the space but navigating around these static objects doesn't feel challenging enough. How about we make the asteroids orbit around the planet? This mimics how real-world physics work for astronomical objects.

The force effects API is a great way for introducing simulations like this. Force effects define a volume and continuously apply forces to physics bodies in that volume.

For the planet, we want a force effect that pulls the asteroids towards the center.

RealityKit offers four built-in force effect types. Let's see if any one fits our needs.

A constant radial effect applies a constant force to all physics bodies towards the center.

A vortex effect applies forces that circulate the physics bodies around the axis.

A drag effect slows down physics bodies with its volume by applying a force that's proportional to their velocity.

And a turbulence effect applies random forces to physics bodies.

The constant radial effect is really close to what we want but we don’t want our planet to apply the exact same force magnitude to all of our asteroids. We want the magnitude to vary based on their distance from the planet.

We can achieve this by defining a custom force effect in RealityKit.

Here I have defined a class called Gravity that will contain the logic for our custom force effect.

All custom force effects conform to the ForceEffectProtocol, which has three requirements: parameterTypes, forceMode, and an update function.

ParameterTypes help us tell the physics engine what parameters we need to compute this force effect. For the gravity effect, we will need position and distance.

forceMode controls how the force vector from this effect is interpreted by the physics engine.

For gravity, we simply set it to force.

Finally, the update function is where we compute the actual force vectors. The parameters struct provides information we need to compute the forces.

I'll unwrap distances and positions from it first. Then, I’ll loop through all the physics bodies that are affected by this force effect. In our case, these are the asteroids around the planet.

I'll define a custom function that computes a force vector with the given distance and position values, and I'll output the result by setting the force on the parameters struct. The physics engine will take this force and apply it to the physics body.

Alright, now we have a custom force effect. Next, let's activate it in our scene.

First, I’ll create a ForceEffect object with an instance of Gravity. Here, I also set up the spatialFalloff so it only affects physics bodies within an 8-meter radius from the planet and set mask so it only affects asteroids.

Finally, I’ll add the gravity force effect to the planet entity with a ForceEffectComponent.

Let's see the gravity effect in action! Hmmmm, the gravity effect is definitely working but it's pulling the asteroids straight towards the center, rather than driving an orbital motion.

To fix this, we need to give the asteroids an initial velocity that follows the orbit's trajectory. Gravity from the planet pulls the asteroids towards it. With the velocity offsetting the gravitational pull, we achieve orbital motion.

Let’s compute the orbit velocity when we spawn the asteroids.

Using the asteroid's radius and angle from the planet, I can compute the initial velocity. Then, I’ll set this initial velocity to the asteroid through a PhysicsMotionComponent.

Let’s see how it looks now.

Nice! The asteroids are finally orbiting the planet. With the power of force effects API, we are able to achieve this orbital motion with just one custom force effect. Now that we have a challenging enough asteroid obstacle course, let’s add a trailer to the back of our spaceship so that players can transport space cargo while flying through the course.

To add a trailer to our spaceship, the easiest approach is to make the trailer entity a child of the spaceship. However, with this approach, the connection between spaceship and trailer feels very rigid.

We can make the connection feel more playful using a physics joint. Joints connect two physics bodies together through constraints.

Let's make a joint that allows for rotations but limits translations. This simulates how real-world trailers work. With this joint, towing the trailer around feels much more realistic and fun.

To make a joint, we need to add a pin to our entity. Pins define a position and orientation that's relative to the entity. One joint connects exactly two pins. The spaceship's pin is at the back, and the trailer's pin is in front.

We want a joint that limits the translation between the two pins to zero but we still want to leave some room for rotations along all three axes.

Like force effects, RealityKit offers several built-in joints.

Without any constraints, the second entity in a joint can freely translate or rotate along any of the three axes. The built-in joints apply constraints for how the second entity can translate or rotate. Let’s take a look at them.

A fixed joint disallows both translation and rotation.

A spherical joint doesn't allow translation but allows for limited rotation around Y and Z axes and free rotation around the X-axis.

A revolute joint is similar to a spherical joint but more restrictive. It allows for rotation around the X-axis only.

A prismatic joint is also known as a slide joint. It allows for translation along the X-axis only. And finally, a distance joint allows free movement in all three axes but only as long as the distance between the two bodies is within a given range.

For our spaceship and trailer, a spherical joint is pretty close to what we want. The only difference is it doesn't limit rotation along the X-axis at all. But that’s all rght! We can use a custom joint type to satisfy our need.

Let's see how we can do this. I have added a Hook entity to the spaceship through Reality Composer Pro. It defines where the pin should be so I won’t need to hard-code a transform. I will find the hookEntity in code and get its position relative to spaceship.

Then, I'll create a hookPin on the spaceship entity, using the pins.set API and use hookOffset as the pin's position. I’ll do same thing for the trailer's pin.

Next, let's create a custom joint that connects the two pins.

A custom joint allows us to constrain the angular motion around all three axes separately. I’ll set the angular motions to be constrained in a small range and restrict the angular motion around X a little more so the trailer doesn't rotate up and down too much.

Similar to angular motion, we can control linear motion along three axes separately. Since we don't want any translations, I’ll set linear motion along all three axes to be fixed.

Finally, to activate the joint, I’ll add it to the physics simulation.

Using our custom physics joint, the trailer now follows the laws of physics as our spaceship drags it along. That looks great! Next, let’s improve the piloting experience for our player by adding a headlight to the spaceship. The headlight will not only light up nearby objects but also cast shadow, so players can easily tell how far away they are from obstacles.

To achieve this, we can use dynamic lights and shadows, which are brought to visionOS this year.

RealityKit offers three types of lights. A spotlight illuminates objects in a cone- shaped volume. It supports customizing its angle, distance, as well as attenuation.

A directional light lights up all objects in a scene.

A point light apply to objects.

You can customize its attenuation radius and fall off exponent.

All three types of light support customizing color and intensity but only spotlight and directional light can cast shadows.

As you introduce lights and shadows, we encourage you to check your app's performance frequently, as these can be expensive to render.

Let's attach a spotlight to the front of the spaceship. Having a powerful headlight will help the player tell which objects are close to the ship. You can add lights and shadows in two ways: through Swift API in code, or through Reality Composer Pro, which offers UI for tuning lighting behaviors. Let's walk through how we add this in Swift code.

Similar to how I had set up an entity in Reality Composer Pro for the spaceship's hook, I added a new headlight entity to define the light's transform. In code, I will find this headlight entity in the spaceship entity hierarchy and attach a spotlight to it.

The spotlight is configured with a yellow tint color. I will set intensity to ten thousand lumen and attenuation radius to 6. These values work well for our scene.

We want this spotlight to cast shadow, so I will also create and add a shadow component to the entity.

Let's take a look at the result.

With the headlight and shadows, we can now easily see when the spaceship is flying dangerously close to asteroids.

All objects that are lit by dynamic spotlights and directional lights cast shadow by default. If you don't want some objects to cast shadow, you can add the DynamicLightShadowComponent to them and set castsShadow to 'false'. This video shows asteroids with castsShadow disabled. Our game is getting exciting to play already. I think it would be really cool to add a portal that warps the spaceship to far-away outer space. Let's implement this feature using the portal API, which brings many new exciting enhancements. Again, thanks to your feedback! A portal opens a window to another world through a mesh surface. All the entities in this world are masked by the portal geometry. Notice how the asteroids surrounding this planet only render inside of the circular portal surface.

To learn more about how to build a portal, check out the session "Enhance your spatial computing app with RealityKit" from WWDC23. On visionOS 1, an object is either fully inside a portal or fully outside.

This release of RealityKit adds a portal-crossing enhancement, which makes it possible for an object to smoothly cross the portal surface, to either go into the portal or come out of it.

In this video we see two spaceships. The one on the left has portal crossing disabled, so it gets masked as it flies out. And the one on the right has portal crossing enabled, and it can fly out of the portal smoothly.

Enabling portal crossing requires setups in two places. First, I’ll set up crossing mode when creating the portal component.

You set the mode to a plane to enable crossing or set it to disabled to disable crossing. If you choose to use a plane, make sure it coincides with the portal geometry itself. In our game, the plane that coincides with the portal is the positive Z-plane.

In addition to portal crossing, you can now also define a clipping plane for the portal. I'll also set it to the positive Z-plane.

The other piece of setup is around the spaceship entity. Entities in the portal world don't have portal crossing enabled by default. To enable it for the spaceship, I’ll add a portal crossing component to it.

And that’s it! Our spaceship can fly into the portal now.

Hmm... I do notice there is one strange thing about the spaceship as it crosses the portal. The lighting on it has a pretty harsh transition right at the portal-crossing plane. You can see it on the spaceship's wings.

The harsh light cutoff is caused by the different lighting environments inside and outside of the portal. For a model that has portal crossing enabled, when it's inside the portal, it is lit by a custom ImageBasedLightComponent. Let's use a star field image as an example.

When the model is outside the portal, it receives additional lighting from an environment probe, which approximates the lighting in your physical environment.

When an object is just crossing the portal, it exists both inside and outside of the portal at the same time. The inside part is lit by the star field from the portal's ImageBasedLightComponent, and the outside part of it is lit by the combination two sources: the star field and the environment probe.

To make the lighting transition smoother, we can use an EnvironmentLightingConfigurationComponent.

This component controls how much lighting an entity receives from the environment probe, specifically for the part of it outside of the portal.

It exposes an environmentLightingWeight value. A value of 1 means the object receives full lighting from the environment probe, and a value of 0 means the object receives no lighting from it. When the spaceship is outside the portal, you can set environmentLightingWeight to 1, so it receives outside lighting. As the spaceship approaches the portal, you can gradually reduce this value from 1 to 0, so that when the spaceship starts crossing the portal, it receives lighting from inside of the portal world only.

To do this in code, I’ll create an EnvironmentLightingConfigurationComponent. I’ll keep track of the spaceship's distance from the portal surface and use a custom function to map the distance value to an environment lighting weight value. As the distance gets smaller, which means the spaceship is getting close to the portal, I’ll reduce the weight.

Every time the ship moves, I’ll need to recompute the weight and update the component to our spaceship entity. Let’s see this code in action.

Nice! The spaceship now gets a smooth lighting transition as it approaches the portal surface. The EnvironmentLightingConfigurationComponent is a powerful way for configuring your app's lighting. It's useful even when your app doesn't use a portal.

To dive deeper into how the new portal enhancements work, I highly recommend checking out the documentation for PortalComponent on developer.apple.com Great! At this point, we have the fully-functioning Spaceship game running on visionOS. We can see the hand-based input, force effects, joints, lights and shadows, and portals all working together.

Wouldn’t it be great to have our spaceship experience run across platforms? With this release of RealityKit, you can now bring your spatial- computing experiences from visionOS seamlessly to iOS, iPadOS and macOS.

With minimal changes to our code, it’s now possible to build our experience for other Apple platforms, and we have added this support for a number of RealityKit features to make this process seamless for you. This includes RealityView, ShaderGraph, particle emitters, portal, hover effects, text, and more. It's also a great opportunity for you to develop cross-platform tools and pipelines to accelerate your app development. Now, let’s use some of these cross-platform features to bring our Spaceship game to iPadOS. With such extensive cross-platform support, most code in Spaceship does not have to change at all.

All the code we've written for hover effects, force effects, joints, light and shadows, and portal crossing, can all stay the same. So what do we need to change? First, on visionOS we can layout our UI in separate windows around the player. On iPadOS, we place the UI directly on screen.

Secondly, on visionOS, we use ImmersiveSpace as the entry point of our spatial experience. On iPadOS, we can just show the spatial experience on the screen directly through a RealityView.

RealityView is available on iOS, iPadOS and macOS with new enhancements, including controls for camera mode, post processing, and more. We want to use world tracking with camera feed as background. So I’ll set the RealityView's camera mode to world tracking to achieve this.

Finally, on visionOS we use hand tracking-based input to fly the spaceship. On iPadOS, multi-touch gestures work better as an input paradigm that matches the strength of this platform.

So I’ll transition to multi-touch-based input with a new multi-touch control view. It has a slider on the left for controlling throttle and a virtual joystick on the right for controlling pitch and roll. This maps to our visionOS input scheme pretty well by separating the responsibilities of two hands.

With these changes in place, the Spaceship game works great on the iPad. All the RealityKit features still work here, similar to how they work on visionOS. We can go to the hangar view, interact with the spaceship with the same drag gesture, and we can fly the ship in our physical environment with world tracking and camera feed as background.

With RealityKit's feature parity across platforms, it's really easy for you to make your app run across visionOS, iOS, iPadOS and macOS.

Great! Now we have a feature-rich Spaceship game that works not only on visionOS but also on iPadOS.

We focused on building out the simulation and rendering aspects of the Spaceship game, but it doesn't have any sound yet. My colleague James will walk through how he creates spatial audio for this game. Check out the session "Enhance your spatial computing app with RealityKit audio," where he covers not only how to use RealityKit's audio APIs but also best practices for creating spatial audio.

There are also many more RealityKit features that we didn't have time to cover in this session. Here is a quick overview of some of them.

LowLevelMesh and LowLevelTexture give you low level access for constructing and updating mesh and texture resources. The animation system has unlocked a lot of new capabilities, including creating animation timelines in Reality Composer Pro.

BillboardComponent introduces a privacy-preserving way to make entities always face the user.

PixelCast enables pixel- perfect entity selection through a rendering-based approach. Subdivision surface allows you to render smooth surfaces without creating a dense mesh. You can learn more about these features on developer.apple.com.

Wow! That was a lot! Let’s recap what we learned today. We started with adding custom hover effects to our spaceship and built a custom hand gesture-based control mechanic using the new spatial tracking API to fly it around.

We used force effects and joints to add physics to our scene and make it more playful.

Then, we used dynamic lights and shadows to make the experience look vibrant. We also introduced portal crossing to transport our spaceship into outer space. Last but not least, we brought Spaceship to run on an iPad using RealityKit’s cross-platform API support.

Remember to check out "Build a spatial drawing app with RealityKit" and "Enhance your spatial computing app with RealityKit audio” sessions to learn about many other features that are new in RealityKit this year. I am really excited to see the amazing experiences you are going to build with these features. Thank you for watching!