Take SwiftUI to the next dimension

♪ Mellow instrumental hip-hop ♪ ♪ Hi, my name is Mark, and I'm a SwiftUI engineer. I'm pleased to invite you on a journey out of the plane and into space using SwiftUI. To demonstrate how SwiftUI works in a new dimension, and to show how SwiftUI and RealityKit work together in the system to enable incredible experiences, we'll talk about a handful of the APIs powering a sample app we've built called World. World demonstrates several different forms applications can take on the system, from the familiar window to the new volume, allowing for bounded 3D experiences to full space, which allows for the display of unbounded virtual content. While most of the APIs we'll discuss in this talk work whether you're adding just a dash of 3D in your primarily 2D window app, or creating an immersive 3D experience leveraging full space, we'll focus on using Volumes as a container since they provide a great way to explore the realm of 3D using SwiftUI in your app. To learn more about the other containers for SwiftUI content, please refer to the "Elevate your windowed app for spatial computing" talk and the "Go beyond the window with SwiftUI" talk. Once we cover the basics of volumes, we'll discuss how you can create and position 3D content with volumetric views and layout, how to integrate SwiftUI views into the new RealityView, and finally how to bring everything together with 3D gestures. But before we get too ahead of ourselves, let's talk about what volumes are and how they can help us take our first step into the realm of 3D. World uses volumes to emphasize its 3D content. This scene has no main glass window. Instead it places its 3D content directly into the scene with a panel of controls in front. But the 3D content is the star of the show here. Volumes provide you with a fixed scale container. Unlike windows that dynamically scale based on the distance they're placed from you, volumes maintain the same size at any distance. Volumes are horizontally aligned and support viewing from any angle. Volumes are a great way to display 3D content in your app without taking over the entire space, and creating one is incredibly simple. All you have to do is use the new volumetric window style when creating a new scene, such as a window group, and you'll get all these features automatically. Now that we have our volume set up, let's get some content inside it. To help us do that, RealityKit offers a new API called Model3D, a SwiftUI view that makes it simple to load rich 3D assets, such as USDZs, while providing phases to handle different parts of the asset loading lifecycle. Think of Model3D as a counterpart to AsyncImage. It handles all the work of loading complex geometry while keeping your app running smoothly. Let's use Model3D to show one of the other models from the World app, the moon. Now, I've already added a moon USDZ file to my project, and I can just pass that name to the Model3D initializer. Now we can handle the phase of the model. Now, this isn't a phase of the moon. It actually indicates the loading status of the asset, and it has a couple different states we can switch over. Before the model has loaded, I can show some text or another piece of UI indicating to the user the content isn't ready yet. I'll use a progress view here. If the model failed to load, I can show an error message using the localized description from the error. And if the model succeeds, I can use it in my UI.

Similar to images, I need to use the resizable modifier to tell the layout system that the model can be resized according to its available space. And I want the model to fit within its available space, so I'll use the scaledToFit modifier. Now I have a moon that loads asynchronously in my app. Let's keep building on this example to explore more 3D functionality in SwiftUI. Let's inspect some of the other models that make an appearance in the World app. I want to show all of them at once in a sort of display. We can repurpose the MoonView we just wrote to handle displaying any USDZ file. Let's just rename it. And pass in the name of a USDZ file to load. Now I can replace the hardcoded Moon string. Now let's make a data structure to represent a celestial object. I'll give each one a name and a size. I'll list out a few different objects that I have in my project: the Earth, Moon, and sun. Now I can display each object in an HStack with a ForEach. For each object, I can use the new CelestialObjectView we just created using the object's name. Let's vary the sizing of each model using a frame modifier using the object size.

Our models look great from the front, but remember we're in 3D. A change of perspective reveals that our objects are back face aligned, as if their bounding boxes were flush against a plane behind all three of them. This is the default alignment for 3D content in SwiftUI. If we want to change that, we can specify the depth alignment to use in a frame depth modifier like so. I can pass the object's size to use for the depth and specify that I want the models front faces aligned instead of the back. Now the views all have their front faces aligned instead. Now I want to give each object a label. I can do that by giving each Model3D an overlay. Inside I'll make a text label with a glass background effect to make sure it's always readable. I'll also make these labels are aligned to the bottom of the model, so I don't block the content. The display looks great so far, but it feels a bit static. Finally, let's explore some of SwiftUI's new additions to geometry effects. I can use a TimelineView to animate a change in time.

Now I'll give our views some spin using the new Rotation3DEffect. I'll make one and give it an angle based on the current date with a scale factor.

And I'll have the objects spin around the y-axis. And just like that, we've got our objects spinning. Model3D is great for loading and displaying simple assets in your view hierarchy. But for more involved 3D models, scenes, or experiences, RealityView is SwiftUI's entry point to unleash the power of RealityKit in your app. And indeed, the World app uses RealityView to break down its more complex visualizations using the RealityKit Entity-Component system. This allows for rendering the individual models, as well as advanced rendering effects like lighting and orbit paths. Thanks to the new RealityView, SwiftUI and RealityKit go hand in hand on the platform. For more information about RealityView and more new RealityKit features, please see "Build spatial experiences with RealityKit" and "Enhance your spatial computing app with RealityKit." For now, let's talk about how you can make the most of SwiftUI within your RealityView using the new attachments API. Attachments allow you to pair a tagged SwiftUI view with an entity that you can use inside your RealityView. Attachments are great for adding annotations or editing affordances relative to specific entities.

If you've used the Canvas API in SwiftUI, attachments will probably feel familiar. The key difference with attachments is that they are live views, not just snapshots. This means they can respond to state changes, run animations, and handle gestures. Let's explore attachments inside the World app by adding on some extra functionality that will let me place some markers tracking my favorite places around the world. In the Earth view, I've created some state tracking an array of some of my favorite places, each with a name and location. For each place, I can create some text with its name. And I can give it a glass background effect so that it's always legible. And now I'll give it a tag so that I can reference it later in my RealityView. This tag can be any hashable value, but I'll use a unique identifier I've added for each place. Now I can look up an entity hosting each attachment view using the tag I specified. I can add this entity to my RealityView content. And then I can use the lookAt function to position and orient the label along the surface of the globe. We have a couple interesting places pinpointed now. But let's not get too attached to this app quite yet. I have some more ideas for how we can take it to the next level. So, now we know how to position 3D content and make the most of SwiftUI inside of RealityView using attachments. Now let's talk about how to interact with all this content. The platform brings the gestures you're already familiar with into the third dimension with support for hands and eyes, as well as new trackpad mechanics. Let's use these new capabilities to extend our progress on the World app. We have a pretty good thing going with our "Favorite Places" extension. However I don't think I'm satisfied with the number of places we have mapped out. Let's make a way to add some more using a tap gesture on the globe. Before we jump into that though, let's talk about how to configure an entity for input. Let's say we're configuring our RealityView with our content. Here I've already added a model of the Earth. An entity hierarchy needs an InputTargetComponent to receive input inside RealityView. If the component is added to an entity, all that entity's descendants can also receive input unless otherwise specified. In RealityKit, CollisionComponent is used for defining the shape of an entity's interactive region. Let's use a sphere for the Earth model so that we can get an accurate interaction point on its surface. This is all we need to allow for our entity to handle SwiftUI gestures inside RealityView. At this point, I can add a SpatialTapGesture to my RealityView.

But to make it even easier to use SwiftUI gestures with your RealityKit content, we've added a new gesture modifier called targetedToEntity that I can use to target my earthEntity specifically. If the tap doesn't occur on this entity or one of its descendants, the gesture will fail. Now let's handle the gesture value. SpatialTapGesture has a new location3D property, allowing us to get an accurate tap point on the surface of the globe. The 3D location is in the RealityView's local SwiftUI coordinate space in points, not meters. To figure out where we want to place our new label, we'll need to convert the location into the RealityView's scene. The targetedToEntity modifier makes this super easy by adding on some coordinate space conversion helpers to the gesture's value itself. We can use that to convert from SwiftUI local space to the scene's coordinate space. Finally, I can add the data for the new place using the location we just calculated. I'll also scale up the location just a tad so that the labels float slightly above the Earth's surface.

We now have a way to add more favorite places to the globe with just a tap. But now we have a problem: we need to discover more places! To do that, let's launch a satellite that can help us pick out more exciting spots around the world. One way to add a satellite model to our globe would be to load a model using RealityKit, but let's use some of the other techniques we've learned about. I can add a Model3D as an attachment. Specifying a frame makes resizing the satellite model to a reasonable size super easy. I'll also give my model a tag so I can reference it in my RealityView. Just like with the labels, I need to add my model to my RealityView. Now, let's define a gesture that returns a 3D transform that will let us define scale, rotation, and positioning of our satellite. I'll start with dragging using a DragGesture. To convert from a drag gesture to a transform, I'll use a map. DragGesture has several new properties to handle manipulation in 3D. We can use the DragGesture's new translation3D property to get how much the drag has moved since it started. Now I'll create the transform. I can pass in the translation to the initializer and return it from the map. Now I can use our manipulation gesture that we just wrote to transform my satellite. I'm going to use the updating modifier to track when the gesture is active. I can use this state to shrink down all our labels during the interaction so that they don't block our view of the Earth. It's important to use updating to track any transient gesture state I have because it guarantees that my gesture state will automatically be reset if the gesture fails. When my gesture value changes, I can set the state's new transform and then use offset modifiers to position the satellite model. I'll also animate changes to the transform using a spring animation, so that when we let go of the satellite, it animates back to its original place. Now we can drag the satellite around. This is a great start, but we need to see this thing in more detail. Now that we have everything hooked up, let's add on some scaling action. For this I add a MagnifyGesture that recognizes simultaneously with the drag. I'll also add the new RotateGesture3D, which can measure unconstrained 3D rotation of the user's hands.

I'll plug in these new values to our transform, and finally I need to update the rotation and scale of the entity. I'll use the rotation3DEffect and scaleEffect for this. And there we have it! We can now freely drag, scale, and rotate our satellite entity. Our satellite looks like it's ready for voyage. The gestures we've added work with all the input devices and modes you'd expect, including direct interaction with hands, indirect pinch, trackpad, and accessibility features. Using familiar SwiftUI gestures, along with the new targetedToEntity modifier, you can quickly build interactions within intricate entity hierarchies. We're now ready to explore the planet with our satellite, but now it's time for you to explore SwiftUI's new 3D capabilities in your apps. New scene types like volumes and full spaces allow you to consider what an application can be in all new ways. The powerful layout and rendering systems in SwiftUI have been extended to make SwiftUI not only a powerful way to build apps on iOS, macOS, tvOS, and watchOS, but on this all-new platform as well. The new attachments API opens up incredible new opportunities to integrate SwiftUI views into 3D scenes. And finally we've explored how to put the stories you tell people right into the palms of their hands, using familiar and powerful gestures in SwiftUI. Thanks to SwiftUI and RealityKit, you have an exciting journey ahead beyond the bounds of the 2D plane. And we're only getting started. Welcome to the platform! ♪