Design spatial SharePlay experiences

♪ Mellow instrumental hip-hop ♪ ♪ Carnaven Chiu: Hi, I'm Carnaven. Jay Moon: And I'm Jay. Carnaven: We're from Apple's Design team. We are excited to talk to you about how you can design great SharePlay experiences for spatial computing. In a SharePlay session, Spatial Personas let you feel present with others in a way technology has never enabled before. You can look your friends in the eye, express yourself naturally with movement and body language, and best of all, enjoy a shared activity together using your app. These shared activities are a key part of what makes this an exciting new computing platform and people will expect your app to fully support the features I'm going to talk about today. Shared activities in FaceTime build on the existing foundation of SharePlay, which is already a great way for groups of people to enjoy content together while apart. This includes simple UI patterns for understanding what everyone is doing, shared control of the content, and independent control for how the experience appears on your own device. Shared context, available in SharePlay sessions with Spatial Personas, allows for an even more immersive SharePlay experience. Picture yourself sitting in a living room in front of a TV with friends. If someone is to your right, you are to their left. Everyone is free to move around the room, and anyone can control the TV. It's always clear who is controlling the TV because everyone is seeing the same thing as everyone else. This is what we mean by shared context. Shared context also allows us to communicate effortlessly. For example, you can see Jay is not paying attention to the movie. He's clearly distracted by his phone. He can point at things or use gestures to better communicate. He can communicate his emotions, thoughts, and intentions, using his facial expressions. You can even tell by his body language and actions whether he's calm and relaxed or stressed and anxious. Maintaining a shared context in FaceTime with Spatial Personas allows us to communicate more the way we have become accustomed to in the real world. In this presentation, we're going to cover how to set up the shared scene for your activity, key considerations when starting SharePlay experiences with Spatial Personas, how to pick the seating arrangement that suits your shared activity, interactions in shared context and the role of UI, and how to present shared content when your app is in a Full Space. Setting up a SharePlay activity with Spatial Personas starts with understanding the type of experience you are trying to recreate. Each type of experience requires its own considerations for content placement and interactivity. Enjoying media together might look different from collaborating on a presentation together, which might be different, again, from an engaging game that deepens social connections. Your app can share up to one window and one immersive space at a time through SharePlay. Once you choose the type of content your shared activity will be built around, you need to think about how to organize and present it in the most socially desirable way. Consider the following when deciding what form your experience will take. If you are presenting your content in a window, you can think of your window as being like a communal device or tool like a TV, a speaker, a whiteboard, or a table. People can interact with your windowed app in their surroundings or even pull up environments like the Moon. When creating SharePlay activities in a Full Space, you can create a more immersive experience, but people won't be able to interact with their other apps at the same time. You can think of these experiences like a dedicated room -- this degree of control and focus is useful for content like games and media experiences. And keep in mind, on this platform, people can easily have many windows open. In SharePlay, every window is marked clearly by the system with "shared" or "not shared." Putting your activity in a window is a great way to support multitasking and collaboration and doesn't require people to leave their surroundings. Using a windowed shared activity enables workflows like dragging and dropping items into your activity. You could even have more than one window from your own app, enabling for example, dragging and dropping songs from a private music library to a public player window. Next, let's talk about how someone would join your shared activity. As on other platforms, you can include entry points to SharePlay within your UI. For example, in the TV app, any Play button can be used to start a shared activity. The platform makes it easy to start or stop a shared activity from the top of any window. People can share any window with no additional effort from your app, similar to macOS. But for richer, more volumetric visuals or group interactivity, you will want to adopt SharePlay. When possible, don't launch any extra windows unrelated to the shared activity. If your app requires an account, consider making shared activities work without them. Minimize the amount of setup involved in joining. If your shared activity does need additional steps to get started, start the activity for people who have the right permissions and present a placeholder window to those who don't. This signals what the rest of the group is doing. The window should allow the person to open anything needed to resolve any issues. As soon as the person has met the minimal requirements to participate, close any extra windows and reveal the activity to the participant, exactly where the placeholder was the whole time. When someone begins a shared activity in FaceTime, everyone already on the call automatically joins, which adjusts where everyone is seated and makes your app visible for everyone. Where your app is positioned, where people sit around it, and where a new person shows up when they join are all defined by one of three seating arrangements that the platform offers and your app can choose from. We call these Spatial Persona Templates. For scenarios where it's important for everyone to have a good view of your app's content, choose a side-by-side template where people sit shoulder to shoulder. For an activity that resembles a tabletop experience, choose the Surround template, where your app's content is placed in the middle of a circle of people. People can converse over your app while looking at its content from their own unique perspective. And for scenarios where your app is ambient or doing something in the background and conversation is front and center, choose a Conversational template. Not everyone gets a good angle of your app's content, so make sure if you choose this, your activity doesn't rely on everyone getting to interact with it. Your app can create a SharePlay experience that supports up to four other Spatial Personas. Your app is responsible for scaling what it presents to any number of participants ranging from two to five. For example, if you choose a Surround template, you should design your content around the fact that there might be up to five seats around the table. And of course, life isn't just about work or content consumption. You can just hang out in a relaxing place with friends and family and enjoy quality time together. You should also choose the surround template when there's nothing to present in the middle of the group. Next, my colleague Jay will take you through how your content and user interface should adapt when you bring people into a SharePlay experience. Jay: Thanks, Carnaven. Conveniently for you, the platform takes on the responsibility of making your app, your participants, and their spaces share a common coordinate system and frame of reference that we call "shared context." This means you don't need to manage the synchronization of the size, position, and orientation of your scenes. While we maintain the illusion that everyone is looking at the same app, your app is actually running separately on multiple devices, and each instance could display audio, visuals, and UI differently. Your app manages the interactions, feedback, and choice of content displayed on each device to make everyone feel like they're looking at the same thing. Your app needs to synchronize everyone's view of content, showing animations and any state changes in lockstep. This is key to making gestures like pointing make sense to everyone on the call. It's possible that your shared activity might work both as a window and as a more immersive experience. You should encourage everyone to be in the same mode by having everyone follow the first person to transition.

Spatial Audio is a great tool for adding more shared spatial context to your app. When everyone is in the same environment, everyone shares the same audio context so the group can hear the gentle lapping of a lake coming from the same direction. A shared music player window will sound as if it's a speaker positioned in the space with you. In your app, you, too, can make sound come from your app window or as part of a more immersive soundstage that everyone can navigate together. Once you've established shared context, we can actually get away with less UI. If you're used to SharePlay on iOS, you might be familiar with how the system communicates when someone has done something that affects everyone's experience. We do this to simulate a sense of presence while apart and to understand why something might have changed without your intervention. Remember that we can perceive more of each other's actions naturally in FaceTime on this platform, especially when they involve large physical motion, and we have the ability to verbally resolve small moments of confusion. You can lean on in-person communication to close any knowledge gaps, so you don't need much UI, if any, to communicate why something has changed. SharePlay activities feel most like their real-life counterparts when people feel free to interact with everything they see. Avoid the temptation to create complicated permissions systems or turn-taking mechanisms if they aren't a natural part of your experience. Consider instead being very permissive with interactions, and impose a consistent and predictable way to resolve conflicts. You can lean on the group settling on norms that arise from bumping up against these rules. While it's important to share as much context across devices as possible, look for opportunities for software to improve on the physical limitations of real-world social experiences, especially when it comes to comfort and accessibility. For example, some people have different volume preferences, or different abilities to parse language. So in TV, I can set my own volume level without affecting someone else's. And people who have trouble hearing dialogue can still enjoy a show at the same time as someone who'd rather not see subtitles. Throughout these personalizations, the same content will still be played back at the same time, so no one is missing context. Collaborative work can take on a different form on this device. By necessity, we're all used to sitting at different devices even when we're working on the same thing. We look at our own view of content and edit one area of a document while a coworker edits another, often leaving notes along the way to call out changes.

On this platform, collaboration experiences can be more lively with a shared view that everyone works on. This also allows you to guide the group's attention with gestures. In real life, we can use different tools at the same time. So while people may be looking at what feels like the same app window, it's important to allow people to be in different editing modes. This means the tool each person has active should also be personalized, so just because you're marking in red doesn't mean someone else is stuck doing the same. As always, keep the main viewport -- the shared context -- the same for all participants. If individuals want to separately dive in on another part of the document, you could always allow each person to open a personal window, as if they brought their own iPad to the meeting. Opening multiple windows can allow individuals to focus on their own edits while still keeping the shared context open for everyone else. Finally, we'll discuss when it's appropriate for a participant to leave a shared context and enter a Full Space on their own. There's a ton of content designed for this medium where there's only one ideal viewing position. In these cases, it may be appropriate to separate people into different contexts. For example, the magical spatial captures you can photograph in Head Set can be shared as a window side by side with everyone else's Spatial Personas. But to get the full effect of the capture, you can enter a Full Space. This Full Space view is best seen head-on, so we don't allow side-by-side viewing and hide other people's Spatial Personas. Think of these moments as if each person has temporarily entered their own room.

Similarly, panoramas, 180-, and 360-degree videos are designed to curve around a single point of view, so Spatial Personas should also not be visible when you enter this mode. You should make sure context remains shared by continuing to synchronize everyone's position in the video. Another good reason to leave a shared context is for individual safety. Someone may want to step out of an immersive experience temporarily. People can always press the Digital Crown to return to passthrough. When this happens, find a way to continue representing the shared context in a window and make it easy for the participant to rejoin shared context once they're done. When done right, even when someone has to dip out, they should never feel lost about where the rest of the group is. In summary, design how your activity begins for everyone. While the system provides a stable coordinate system for all participants, your app is also responsible for sharing as much context as possible. Depending on how complex your app interactions are, personalized control can be necessary for comfort and accessibility. In-person nonverbal cues can take the place of state-change notifications in your UI. And designing a great shared spatial experience often comes down to knowing when and how to diverge from shared context. Apps on this platform really shine when they can be enjoyed with others. We encourage you to build your app with these social experiences in mind. Carnaven: Thanks for watching. ♪