Design interactive experiences for visionOS

Hi, and welcome to "Design interactive experiences for visionOS". I’m Rose Thomas, an Experience Designer at Apple. And I’m James Cooper, Creative Director. We’re excited to help you get started designing interactive narrative experiences for Apple Vision Pro.

Interactive experiences are a new medium that bring together the storytelling possibilities of linear entertainment like film and television, the interactivity of video games, and the immersion and presence of live theater. These experiences invite audiences into the adventure, where they might come face to face with characters in the story.

Where audiences become an active participant and can explore narrative-driven worlds without fear of failure.

Where they can immerse themselves in the magic of this new medium as it transforms their space.

This medium is rapidly growing, and these experiences are an incredible new way for you to connect with your audience and tell impactful stories that take advantage of the unique possibilities of spatial computing.

And Apple Vision Pro is an amazing device for telling these stories.

In this session, we’ll share some design considerations when creating your own interactive experiences for this platform.

First, we’ll share how you can set the stage for your experience and immerse people in new worlds.

Then, James will explain how you can use interaction to draw the audience into your story.

And I’ll share some learnings around designing experiences for everyone.

Let’s get started with setting.

On Apple Vision Pro, interactive experiences have a special super power: They can bring the world of your story into the audience’s physical setting. Encounter Dinosaurs, an interactive narrative experience we designed for Apple Vision Pro, does exactly this.

It allows audiences to peer into the Cretaceous Period, and have the creatures of that period peer back! It’s available for free from the Vision Pro App Store, so you can try it whenever you like. There’s a lot to think about when you’re bringing an entire universe into someone’s space. Fortunately, visionOS gives you a wide variety of ways to tell your story and immerse people in your world.

You might want to tell your story within a piece of familiar UI, like a window. You might want 3D content to come to life in a volume.

You may want to surround the audience in your environment, bringing people further into your narrative. Or you may want to combine all three! How you choose to tell your story is up to you. In Encounter Dinosaurs, we bring audiences face to face with some intimidating creatures, but fully immersing someone in that world can be a lot for people not used to interacting with dinosaurs! You can’t just suppress 65 million years of gut instinct, so we blended the audience’s world with our own. We designed a custom RealityKit portal that allows people to look into that world, but keeps a boundary between audiences and our giant carnivore.

The rectangular shape of the portal feels reminiscent of cinema and television, and offers the audience a delightful glimpse into another world. It also allows us to focus the audience’s attention in front of them. When designing your experiences, consider which building blocks of visionOS work best to help you tell your story.

When you go the movies, or watch TV, you’re not asked to place the screen yourself before every viewing. Similarly, in visionOS, windows and volumes are automatically positioned in space when they first open.

If you're designing an experience for an immersive space, the origin of the content defaults to your head position.

But if you create something custom like we did, you decide how that content appears.

With visionOS, you can make this process feel like magic for your audience.

For example, when someone opens Encounter Dinosaurs, they’re greeted with an initial credits roll and a start screen. It's a nice way to onboard the audience, but it also serves a second purpose.

Our experience uses ARKit behind the scenes to get a sense of the audience’s space and find a great place for our portal. When designing for the placement of your content, think about using features like scene understanding, head position, and more to help ground your content in someone’s world and let them focus on your story, rather than the setup.

One of the best things about Vision Pro is someone can take it anywhere, so your audience can experience your story at home, at the park, or even on a plane. But all those options can affect how you choose to display content.

What works indoors in someone’s living room might not work outdoors in a park.

When we designed Encounter Dinosaurs, we imagined the portal opening against an open wall. And if someone starts the experience in a large room or outdoor space, that's exactly how it works, the portal opens as large as 4m across along a flat plane. But if you don't have as much space, the experience changes. In smaller rooms with more narrow wall space, the portal wraps around the audience to keep the experience engaging without running through a wall and breaking immersion. And in limited spaces where there might be less room in front of the participant, or in settings such as a bus or a plane, the experience places the portal a set distance from the participant and dims passthrough to maintain immersion. You can’t design for every possible space as you never have control, that’s the illusion, but you can consider common settings where someone might engage with your experience and how it might change.

And finally, a very important tip when it comes to designing spatial content, everything changes once you’re testing in device. You may imagine how things are going to look and feel while designing, but you can’t know for certain until you test them. Prototype and test your ideas in device as often as you can. Things you thought would work often don’t, and things that you thought might not work actually turn out great! Now, I’m going to hand it over to James to discuss how you can really bring your story to life! Now that we’ve set the stage, let’s talk about interaction. The fundamental thing that sets interactive art forms apart from any other medium is the agency of the audience. You are making them an active participant in your story through interactivity.

The best interactive stories are a harmony of storytelling and interactive agency. You must maintain a balance between empowering your audience and the story you are trying to tell. One does not triumph over the other. This becomes your audience’s story as much as it is your own. In this section, we’ll talk about how to find that balance.

We’ll start with how to introduce interactivity, as spatial content can be overwhelming for many people.

It's important to ease your audience into the experience, set the tone of the world, and help people understand how they can interact.

Originally, we hoped participants of Encounter Dinosaurs would naturally discover interactivity, but in early testing, we found many people thought it was a passive experience and didn't realize they could interact at all.

Rose mentioned our opening credit sequence, which helps to cue the audience that the experience is about to begin.

After the credits, we added a series of onboarding graphics, which encourage people to explore and provide clear direction that the creatures in the experience can see them and may react to them. Since this is a new medium, there is no precedent for interaction, so these kind of clear directions can be essential. You can then use implicit interactions to build upon those directions. Encounter Dinosaurs is a character-driven experience, so it felt natural to teach people how to interact using a character: our butterfly. It flutters directly towards the audience, and its motion draws their attention. It then circles just in front of them, its proximity naturally inviting them to reach out. And if they extend a finger or a hand, the butterfly will briefly land.

Your first interaction can set the stage for the entire experience.

You can invite people to interact by introducing a familiar object, character, or pattern that they recognize, then use cues like proximity, motion, and sound to pull them in. And just make sure that your first interaction is with something small, simple, and welcoming to ease your audience in. Let’s talk about characters. A good story is all about the characters within it, and adding interaction gives you new ways to help audiences connect with those characters.

When designing spatial characters, you can utilize eye contact, proximity, and hand and body interactions to really make them come alive in someone's space.

The butterfly may be the first character the audience meets in our interactive experience, but the real stars of the show are the dinosaurs. After the portal opens, the first dinosaur you meet is Izzy, our baby Isisaurus. Because we don't have too many dinosaurs roaming around these days, we often compared her to a puppy or a kitten, and thought about how one might interact with a baby animal in the wild. Izzy is watching and listening to you the whole time. We animated her to be endearing, creating moments where she scrambles and slips, and makes adorable cooing noises. People naturally want to get close to Izzy. If they approach as she’s chasing the butterfly, they can even coax her onto a rock in the scene, where she climbs up to a comfortable height for participants to begin interacting.

Clever girl.

Interactions here are natural by design: we developed a subtle gesture and motion detection system that allows Izzy to identify and sniff a person’s hand, and allows participants to pet her, and build trust like one might with a real animal.

Of course, it might not come naturally for audiences to want to interact with a bigger dinosaur that enters their space, particularly when you can tell it’s a carnivore! To try and alleviate any sense of fear the audience might be feeling, we designed Raja - our Rajasaurus - to approach with a sense of curiosity.

He tracks head and hand movement, and keeps his eyes on participants as they move around.

He backs up and approaches as they step forward and back, and even snaps at them if they push a little too far.

And if you don’t move, well, he’ll still see you if you don’t move.

When designing your own characters, a great starting place is to consider how someone might interact with a comparable creature in the real world. VisionOS has a number of great features that let you design authentic interactions for your characters, even something as simple as getting the audience's head position so a character can maintain eye contact, can have a big impact on how someone experiences your story. Once you’ve designed for these unique hand and body interactions, you need to clearly signpost when participants can or can’t interact.

We set a rule where interaction should only happen when a character is right on the boundary of the portal, or in the audience’s space.

Before Raja gets to that point, he can look at the participant and track hand movements, but interactions beyond this are limited.

As he crosses the portal threshold and enters the participant’s space, he gets close to them, and bows his head as if to say, “I dare you to pet me.” We kept this pattern consistent for all characters in our experience. The character must also be looking at the participant: eye contact at close proximity naturally cues interaction. When Raja’s attention is drawn elsewhere - by the sound of prey, or an invading contestant to his territory, audiences naturally don’t expect to interact.

When he re-enters the portal, his interactive moment is over, and he returns a safe distance from the audience.

When you design custom interactions, it's important to create consistent rules about how and when they work, and when they don't, within your story world. You can use cues like proximity to implicitly guide your audience on any limitations of interaction.

These rules become the audience’s verbs for how they interact with the characters and world in your story.

And that consistency, for how and when to interact, is so key in maintaining immersion. Encounter Dinosaurs introduces hand interactions in the very first moment with the butterfly. Audiences can then interact with both Izzy and Raja using their hands.

If suddenly those hand interactions did not work, the experience would feel broken, and the audience’s sense of immersion would shatter.

Interacting with virtual characters should be fun. When encouraging your audience to engage, think about authentic behaviors, what comes naturally, and how you can enable that through intuitive gestures, clear signposting, and consistent interactions. You can then breathe life into your interactions with spatial audio, to deepen audience immersion, and to direct attention with subtle cues.

Encounter Dinosaurs is filled with spatial audio sources that brings the world to life, with multiple emitters for background ambience like the wind, and key moments like the thunder.

Each character also has multiple spatial audio sources, with omni directional emitters on the feet and tail, and directional emitters on their mouths.

And music is spatialized with a surround mix that draws focus to the scene before the portal opens, and after it closes.

We can also use spatial audio to direct attention.

When Izzy arrives, you hear her chirp coming from inside the fissure. When Raja is first introduced, you are cued into his entrance with loud, booming steps coming from the left.

And when Raja gets distracted by an Isisaurus chirp from behind the participant, it feels like it really is coming from inside the room.

Spatial audio can bring your world to life and create that extra layer of immersion for your audience. Consider where you can use audio in your story to give characters or objects an incredible sense of presence, set the scene, and direct attention Agency is what sets these experiences apart from other media, and the way that people express themselves in these worlds is through interaction. When you find that right balance of story and agency, and follow these interaction guidelines, you can bring your story worlds to life in truly delightful ways. Back over to Rose. Thanks, James! For an experience to be interactive, you need someone to interact with it. You need an audience. Great interactive experiences are designed with audiences in mind, and you can offer multiple ways for them to be part of the world.

If there’s one thing you take away from this talk, it’s that you need to design around your audience. When we started designing Encounter Dinosaurs, we had plans for what we wanted the audience to experience. We wanted participants to enjoy the perfect narrative journey, to do specific things at the right time, to interact in a certain way. But this medium is all about putting the audience in the driver’s seat: it’s their experience, their story. The more we tried to be prescriptive, the more we found ourselves fighting against audience intention, and the less intuitive the experience became. When we started watching how our teammates tried to interact with izzy without prompting, it opened up more freedom in our experience. Consider how you can create experiences that support audience choice, even if the action someone wants to do isn't something you had originally considered. And test often. Get participants to try out your experience and pay attention to how they want to interact with your world, then lean into that. Once your world and characters respond to the things your audience naturally wants to do, you may find they enjoy your story even more.

And remember, not making a choice is also a choice. Sometimes your audience may not want to interact, and that's okay: Part of giving agency to your audience is letting them decide how engaged they want to be with your experience.

Encounter Dinosaurs supports a range of audience experiences. For participants who prefer a more traditional cinematic experience, they can sit back, watch, and still experience a full narrative with moments of conflict and resolution. They'll witness the butterfly, meet Izzy, watch Raja stomp on in, and conclude their peek into the portal with a tense showdown. If the audience wants to be more involved, the world recognizes their choices. Izzy and Raja both behave very differently depending on whether the audience is passive, respectful, or pushy.

And for the most adventurous participants, we built even more to discover. The way they interact can change the story and its ending.

Remember, the story you want to tell may not match the story your audience wants to tell. While you can't anticipate every choice, your experience should respect an audience's decisions if they try to do something that doesn't fit with the story you're trying to tell, consider designing in-world boundaries that gently steer them back on track. When designing your experience, it's also important to think about the audience's emotional journey throughout your story.

I want you to think on your favorite movie for a moment.

It probably alternates between moments of tension and relief, before concluding with a satisfying ending. It can be exciting to have back-to-back action scenes, but it's important to provide moments of respite to avoid leaving audiences fatigued by the end. In interactive experiences, nonstop interactions can also be physically exhausting, so it's even more important to offer the audience breaks.

We built high moments of action and interaction, and low moments of relief, into our Rajasaurus sequence. Try to find variation in the type of emotion you want to elicit in your audience, and alternate between slow and fast moments.

When you design for your audience, you're making your experience more resonant and accessible for everyone.

Apple Vision Pro offers several ways you can support people's accessibility needs, like closed captioning for audio tracks, VoiceOver navigation, audio descriptions for on-screen content, and Dynamic Type for UI on screen. You should adopt as many of these as possible in your experience. Encounter Dinosaurs is a highly audiovisual experience, so it was crucial that we support audio descriptions.

Four more baby Isisaures pop up and climb out. The Audio Descriptions add an extra layer of richness to the content, and the information adjusts based on the choices the audience makes, so these dialogs are just as dynamic as the content. Now, most interactions in Encounter Dinosaurs require physical movement and gestures. But to make the experience accessible for everyone, we designed a custom interaction system.

It works like a modal in a video game, and responds to system gestures and accessibility features like VoiceOver or Dwell.

We wanted to maintain a sense of discovery and magic for everyone as they go through the experience, even if they choose the alternative interaction system. We designed language that spoke broadly to the audience's intention, and asked them how they wanted to interact. If a participant selects “Be friendly" or “Be forceful," they'll get the same reaction from the dinosaur that they would if they moved that way. If you turn on this option, you can also still access the primary interaction system: The dinosaurs can still see the audience and watch them in their space, and they'll still react to movement. Audiences can choose what works best for them at any point during the experience.

If you're designing custom interactions for your experience, consider how you can design alternatives that make it accessible to all. I'll pass it back to James to wrap us up.

Thank you Rose, We covered a lot of topics to help you build awesome interactive experiences, and although it can be challenging, we’re sure you’ll find that with these learnings, life finds a way. We hope you feel the same level of confidence, and can’t wait to see what interactive stories you tell on Apple Vision Pro! Thank you.