Meet SwiftUI for spatial computing

♪ Mellow instrumental hip-hop ♪ ♪ Hello, and welcome to "Meet SwiftUI for spatial computing." I'm Andrew, an engineer on the SwiftUI team. We're so excited about spatial computing. It opens up a world of new possibilities for building groundbreaking applications. In this talk, I'll walk you through what you need to know to build your own great apps for the platform and the powerful new features in SwiftUI that make it all possible. When you build an app for spatial computing, the best way to build it is using SwiftUI. We're bringing SwiftUI into a bold new future, with all-new 3D capabilities like volumes, immersive experiences with Full Space, new 3D gestures, effects, and layout, deep integration with RealityKit, and so much more. These new capabilities only exist in SwiftUI. If you have existing code for your apps, much of it will still work on the platform. For all new code, the best way to write it is with SwiftUI. We believe so deeply in SwiftUI that we built the system from the ground up using SwiftUI. From core building blocks of an app like Buttons, Toggles, and TabViews, to core elements like the Home View and Control Center, to familiar apps like TV and Safari, to all-new experiences like 3D boards in Freeform and immersive rehearsals in Keynote, SwiftUI powers all of these things and so much more. With SwiftUI, you describe your app's user interface, leaving the system to choose intelligent defaults for you so you can learn these concepts once and apply them anywhere. This is even more useful with an entirely new platform. Your existing SwiftUI knowledge transfers over seamlessly to an entirely new look and feel. Consider a button. Buttons on the system have a lot in common with the buttons you already know and love from other platforms. For example, like on macOS, buttons on this platform use a bordered style by default. But buttons also have some key differences to adapt to the idioms of the platform. Let's zoom in to our simulator and take a closer look. Bordered buttons like these use a vibrant material background. And all buttons gain rich hover effects that react to your eyes, hands, and to pointer input. They can also scale down and provide audio feedback when pressed. And with buttons in places like navigation bars, they can automatically display a tooltip with your button's label when people look at them. Or consider TabView, which hangs off the side of your app, providing quick and easy navigation without interfering with your app's content. And TabView expands to display more detail, like each tab's labels, just by looking at it. TabView is a prime example of how SwiftUI can take advantage of what makes the platform unique, like having your app's interface react to where you're looking. We've applied this same thoughtful design to all of the core building blocks of an app, from navigation and presentations to controls and interactions. We provide these intelligent platform defaults so your app can fit right in from the start, allowing you to focus on what makes your app great. But we didn't stop there. SwiftUI also includes a whole suite of brand-new APIs purpose-built for a 3D environment. And those APIs start with scenes. Recall that an app in SwiftUI is made up of scenes, and scenes are made up of views. Scenes form the top-level entry points to your app, like WindowGroup for displaying one or more windows. With spatial computing, there are three kinds of scenes that make up an app: windows, volumes, and Full Spaces. Windows are great for building traditional and familiar interfaces, ranging from rich and complete apps like Safari or Freeform, to menus that serve as an entry point to more immersive experiences like Mindfulness. Volumes are a new 3D window style purpose-built for displaying objects or experiences in a bounded space. Volumes can also be displayed alongside other apps, making them a great tool for building lightweight 3D experiences, from previewing 3D models with Quick Look to playing board games over FaceTime with your friends. Finally, we have Full Spaces, a brand-new way to build rich, immersive apps. Full spaces give your app complete control, hiding windows from other apps and allowing you to place your content anywhere. Full spaces can augment the real world while keeping people grounded in their surroundings, or they can fully immerse people in incredible new experiences. From realistic rehearsals in Keynote to thrilling new kinds of games, Full spaces unlock a world of new possibilities. And these scene types are designed to be used together. Each one is purpose-built for different use cases, and you can mix and match them in whatever way makes sense for your app. You can present a volume from a window, like to show a lightweight 3D preview of an architectural model. Or you could present a Full Space from a volume to completely immerse people in that model. You can also use multiple scenes of the same type, which is a great way to organize your app into separate pieces that people can open and close independently. Let's dig a little deeper into each of these scene types, starting with windows. I'm a big fan of astronomy, and I've been working on an app with some of my fellow stargazers to help others learn more about the Earth and solar system. Now, they've already gotten a great start on this app, building out this gorgeous introductory page in a window. But I have big plans to go further. To show our window, I'm using the same WindowGroup API that works on all other platforms By using WindowGroup, I also automatically support creating multiple windows, just like on macOS or iPadOS. Now, I've been really keen to add a new feature to the app: a library to collect all kinds of fun facts about our solar system and universe so my fellow space students can keep on learning. But I've been stuck trying to figure out where in the app to put this feature. To help answer that question, let's break down the structure of a window into its building blocks. Windows on the system start with a beautiful glass background, keeping people grounded by letting your surroundings show through while keeping your app's contents easy to see. Within my window, I can use the same navigation containers that I'm already familiar with on other platforms. For example, I can use a TabView to organize my app into several top-level components displayed on the leading edge of my window. I can also use navigation stacks and split views to help organize my app's hierarchy and display rich information, following a similar structure as iPadOS. Lists are another powerful tool for organizing information with a beautiful new style that fits right in. And of course, I can add interactivity to my app using built-in controls like buttons, toggles, and pickers. Now, I think this Library view would be a great use of a TabView, which is perfect for making the top-level entry points to my app accessible at all times so deeper knowledge is always just a tap away. To do that, I'll wrap my content in a TabView, and provide a tabItem with a label for each tab. Looks great! TabView has a unique appearance, hanging off the edge of my window. TabView is a great example of a new concept we've added to SwiftUI: ornaments. Ornaments allow you to place accessory views relative to your app's window, and they can even extend outside the window's bounds. These are great for displaying additional controls that don't belong in the window itself and can avoid interfering with your app's content. And we can even create our own ornaments using the new ornament modifier. Let's take a look at the Library detail view I've been working on. I've already gotten started on this page, broken up into a few sections: some summary text, a grid of interesting stats, and a horizontally scrolling list of fun facts. But these sections are missing something. To figure out what, let's take a closer look at our app's materials. Windows by default are grounded in a beautiful, new glass background. Glass is designed to compose with new, vibrant materials that help provide a visual hierarchy for your app. Glass adapts automatically to the environment to keep content legible no matter where it's used, allowing your app to look great in any setting. This means that on this platform, there is no dark or light appearance for your app. Materials do the hard work for you. Let's see how we can use materials to make my Library tab fit in and improve legibility, starting with the stats grid. My stats grid section is currently just made up of a VStack with a title and the grid itself. Let's try giving it a material background and make this card really pop. To do that, I've used the regular material, which adds a vibrant, darker background on top of our window's glass that grounds the shape of my card and improves legibility for the detail text. I've used a rounded rect shape and added some padding to make the card feel just right. I'll apply the same treatment to our fun fact cards as well. That's looking much better! Next, let's turn to our fun fact cards and zoom in for a closer look. This card is built as a button that can show more details about the fact when I press it. I've also used a custom button style that provides that nice material background. The button's contents are composed of a title, detail body, and a footnote inviting people to tap to learn more. The "Learn more" text is looking a little too heavy, though. To fix that, I'll use the secondary foreground style for my text. Notice that this secondary style automatically adapts within our background material to use a beautiful new vibrant treatment, giving greater visual weight to my content while still keeping that secondary text legible. These hierarchical shape styles are great for expressing relative visual weight, letting SwiftUI take care of automatically adapting the specific materials to use based on context. Use them and the other rich shape style and material APIs in your apps to create gorgeous interfaces that look great in any context and on any platform. Next, let's turn to interaction. With spatial computing, there are brand-new ways to interact with apps. The most common form of interaction is to simply look at an element and use an indirect pinch gesture to perform a tap. You can also directly interact with apps using your hands just by reaching out and touching them. And with a connected trackpad or a hand gesture, you can use a pointer for the most precise input. The system also works with a connected hardware keyboard, with support for keyboard shortcuts, Focus, and key modifiers to supercharge productivity in your apps. Finally, the system supports the same great accessibility technologies that you're already familiar with from other platforms, like VoiceOver and Switch Control. The best part is that SwiftUI controls are built to be accessible out of the box, doing much of the heavy lifting for you, which allows everyone to get the same great experience when using your apps. The gestures you're already familiar with in SwiftUI work great, adapting automatically to each form of interaction, like TapGesture and DragGesture. And we've added some brand-new gestures to enable all-new kinds of rich 3D interactions like RotateGesture3D to rotate a view in all three dimensions, using two hands or a connected trackpad. And of course, all of the same accessibility APIs and technologies from other platforms work great too, from VoiceOver and rotors to Dynamic Type and Invert Colors. And many of these features have been reimagined for the platform. Like Dwell Control, which allows people to use and navigate your apps using only their eyes. To learn more about these great features and how to make your app accessible to everyone, check out "Create accessible spatial experiences." A critical tool to making interaction feel easy and intuitive with spatial computing are hover effects. For example, simply by looking at an interactive view, the system shows a subtle highlight effect over the view to provide feedback that you can interact with it. These effects adapt automatically to each form of input, giving people confidence in what they're interacting with. Hover effects are useful not only to provide responsive feedback, but also to assist with targeting. And hover effects are the only way to make your app react to exactly where people are looking. These effects are applied outside of your app's process to respect people's privacy. Hover effects are added automatically to most controls, like buttons, toggles, text fields, and more. If you use the built-in styles provided by SwiftUI, you'll get these effects in your app automatically. If you're using a custom control or style, make sure to add hover effects to make them feel responsive and easy to use. Let's return to those fun fact cards I've added to my app and check out how they work with hover effects. Those cards are looking great, but there's one big problem. Because I'm using a custom ButtonStyle, I'm now in charge of providing my own hover effect. Without one, there's no feedback that these cards are actually interactive. The good news is that's super easy to fix. My ButtonStyle currently just adds some padding and a custom material background to the button's label. To fix that missing hover feedback, I'll add the hoverEffect modifier to my ButtonStyle. This selects an automatic effect for me that is appropriate for the context. In this case, that'll be the highlight effect. Let's take a closer look. When I look at these card buttons, I now get a nice, subtle highlight effect that makes it clear that they're interactive. The effect even automatically matches the shape of my button's background, ensuring that nobody misses a chance to learn a fun fact. There's a lot more to cover about building great windowed apps. To go further, check out "Elevate your windowed app for spatial computing," where you'll walk through updating a multiplatform app, learn how to add ornaments, dive deeper on hover effects, materials, and more. Next let's take our app to the next dimension with volumes. I want to help my fellow outer space observers gain a new perspective on the planet that we call home. And a volume is perfect for doing just that. To add a volume to my app, I'll use the same WindowGroup scene that I used for my main window, and simply specify a volumetric window style. I can also provide a default 3D size for the window so it can fit our content. Let's check out our volume in the simulator. Within my volume's content, I'm using the new Model3D API from RealityKit to display the 3D Earth model our designer has put together. And with Model3D, it's really just that easy to add 3D content to my app. A Model3D is similar to an Image, making it easy to load and display beautiful 3D content. Unlike an Image, a Model3D always loads asynchronously, as 3D content can take time to load and be ready to display. Similar to the AsyncImage view, Model3D can automatically display a placeholder view while we wait for the content to load. Or I can take full control and display my own placeholder instead. It's worth taking a moment to notice here that Model3D is just another SwiftUI View. Bringing 3D into your SwiftUI app builds on the same concepts you're already familiar with with natural extensions to the layout system, visual effects, gestures, and more. Let's build on this example so we can understand how. I want to add some controls to help bring the Earth to life. I've already built out my control panel UI, and I'd like to place it in front of my Earth model. To do that, I can simply use a ZStack layout. Layouts like ZStack are automatically aware of the depth of your content, just as they are with width and height. And Model3D by default is sized to fit its content in all three dimensions, just like an image is in 2D. In fact, the entire layout system is aware of your content's depth and its available space and adjusts your layouts accordingly. And there are even new modifiers to control how depth behaves with layout in your apps, like the new padding3D modifier to add spacing between your SwiftUI views along their front or back faces, which we can use here to give our controls some room to breathe. Remember that volumes are designed to be viewed from any angle, so it's important to think about how your app's content is arranged in all three dimensions. Finally, to make my controls look great and feel grounded, I'll make sure to use the new glassBackgroundEffect modifier, giving it the same beautiful glass treatment we had in our standard window before. Now these controls are looking ready to rock. Another thing I'd love to add to our 3D globe is an easy way to spin the globe to a random place, which is my favorite way to plan my next vacation destination. To do that, I'll add a 3D rotation effect to my Model3D, in this case about the y-axis. And I'll use a state variable to track the rotation to use. Then I'll add a tap gesture and when it fires, I'll change the rotation by some random amount with a new bouncy spring animation. Let's give this a spin. Ah yes, the western hemisphere. That really narrows down my summer plans! Notice that the rotation effect we just applied to our globe is truly 3D. We've upgraded these geometry effects you already know how to use with new 3D capabilities, including scales, offsets, and custom 3D transforms. Now this is already a great way to check out the globe, but I think we can make it even better. To do that, I'll use RealityView, a new SwiftUI View that provides easy access to the full power of RealityKit. With RealityView, I provide a closure to load and make my RealityKit content. To display my Earth model, I'll create a ModelEntity and await until it loads. And once it finishes loading, I'll add it to my RealityView's content to display it. Notice that I can use async-await directly in my RealityView's closure. Just like with Model3D, it will automatically display a placeholder until my content has loaded. Now that I've loaded my Earth entity, I have access to the rich library of RealityKit APIs to make my content really shine. In this case, I want to add some lighting to the Earth to give it a sunnier disposition. I've already written some RealityKit code to add my special image-based lighting, so I'll call that here. Now that is looking delightful! There's so much more to cover with RealityView. With RealityView, it's never been easier to blend the worlds of user interfaces and rich 3D experiences. RealityKit allows you to add all kinds of rich behaviors like custom materials and shaders, physics and complex animations, and so much more. To learn more about RealityView and RealityKit, we've prepared several talks that go much deeper. Check out "Build spatial experiences with RealityKit" when you're ready dive in. For now, let's focus on two cool features of RealityView: gestures and attachments. With RealityView, SwiftUI gestures work automatically, making it easier than ever to bring your 3D content to life. I want to build on the tap gesture we added before to also place a marker on the globe based on where I tapped so I know exactly where to travel to next. To do that, I'll use a SpatialTapGesture, which now gives me the full 3D location of the tap. To help me identify where in the Earth's entity I tapped, I'll use the new targetedToAnyEntity gesture modifier. This provides me with the context I need, like the entity I tapped on and the location relative to that entity, which I'll use to look up the location for my pin. To display the pin on my globe, I can use RealityView attachments. Attachments are great for mixing custom SwiftUI views together inline with RealityKit entities. I can add other SwiftUI views directly within my RealityView's attachments closure, making them available to place as RealityKit entities anywhere in my RealityView. I'll add a pin attachment here and give it a tag that I can use to identify it. Then in my update closure, I'll look up the entity for my attachment and add it to my RealityView's content to display it. And then I'll place that entity to match the tap location. Let's give this another spin! Looks like I'd better get packing! We're so excited about the possibilities for building amazing volumetric apps that we've prepared an entire talk to help you take your SwiftUI apps to the next dimension, where you'll learn more on using SwiftUI together with RealityKit, building rich 3D interactions, adding depth to your layouts, and more. Finally, let's take this app beyond the window with Full Spaces, a brand-new way to build rich, immersive 3D experiences using SwiftUI. With a Full Space, you take full control. You can place content anywhere in the environment to augment people's surroundings in creative new ways. Or you can fully immerse people by hiding their surroundings to create stunning new experiences. Let's dive into space and bring our solar system to life. I've already gotten a head start on this page in our main window, inviting our armchair astronauts to view outer space for themselves. Now I just need to create my space and wire up this button for liftoff. To add a Full Space to my app, I'll simply add a new ImmersiveSpace scene as I did with my WindowGroups. Within its body, I'll provide a root view for my space's content. I'll also provide an ID for my space so I can programmatically open it from our main window. To open this space, I'll use the new openImmersiveSpace environment action. Within my button, I'll call that action and pass in the ID for our space. And just like that... ...we have liftoff! This is a great start. I can get up close to the Earth in a way that I never could before, letting me really appreciate all the rich detail, like those realistic clouds. But there's something missing here. I want to feel truly immersed in this space. To do that, I'll use a powerful tool of spaces: immersion styles. A Full Space can come in one of several immersion styles, and you can transition between these styles on the fly. With mixed immersion, your space's content coexists with the real world, which makes it great for lightweight experiences and augmenting people's surroundings. With full immersion, your app can become fully immersive and hide people's surroundings, transporting them into stunning new worlds. And progressive immersion is a great middle ground for experiences that keep people grounded in the real world in their periphery. With progressive immersion, people can also use the Digital Crown on the device to dial in exactly how much immersion feels right to them. Now I think a fully immersive space would be the perfect fit for my app to make it truly feel out of this world. To do that, I'll return to my ImmersiveSpace scene and add in the new immersionStyle modifier with the full immersion style. Here I provide both a list of supported styles and a current selection, allowing me to change the style on the fly. When I use full immersion, the system completely hides people's real-world surroundings. So I need to provide a virtual environment in which to immerse the people using my app. And of course, what better environment to use than outer space itself? We'll reach for the stars with our environment by creating a new RealityView to display our star field. Within our RealityView's make closure, I'll load our star field entity, and when it's ready, I'll add it to the RealityView's content. Then I'll add the Starfield view to our solar system's body alongside our Earth and Sun. Let's zoom in for a closer look.

Now this is making me feel truly starstruck.

To take my space to the next level, I can use ARKit. ARKit is a powerful framework deeply integrated with the system, providing access to rich real-time understanding of people's real-world surroundings, with APIs like world tracking and scene understanding that allow you to place content on real-world surfaces. And hand tracking, an incredible new tool that you can use to build custom hand gestures, interact with your content using realistic physics, and so much more. To learn more about how you can use ARKit in your spaces, check out "Meet ARKit for spatial computing." Now I'm so excited about the possibilities that ARKit unlocks with my app that I wanted to give you a sneak peek of one more feature that I've been working on. By integrating ARKit into my space, I was able to implement a new hand gesture to summon the Earth. So you can hold the world in the palm of your hand.

We're just scratching the surface of what you can do with Full Spaces. And we've prepared another talk for you to launch into space. Check out "Go beyond the window with SwiftUI" to dive deeper on the fundamentals of spaces and learn about advanced tools like adding effects to your surroundings, displaying virtual hands, integrating with SharePlay, and more. And if you need full control over rendering the content in your space, you can do that too using Metal and the new CompositorServices framework. Check out "Discover Metal for immersive apps" to learn more. We can't wait to check out all the amazing new apps you're going to build using SwiftUI. From beautiful windows to an entirely new dimension with volumes, to incredible immersive experiences with Full Space, a world of new possibilities await you. If you're ready to dive deeper, we've got even more great talks for you to check out, like "Principles of Spatial Design," where you can learn more about how to design your app to really shine. And if you have existing UIKit code you want to bring to the platform, check out "Meet UIKit for spatial computing." Thanks for watching, and have a blast with spatial computing! ♪