Qualities of great AR experiences

♪ Mellow instrumental hip-hop music ♪ ♪ Welcome to "Qualities of great AR experiences." I'm Alli Dryer, and I'm part of the Apple Design team working on augmented reality.

AR lets you deliver experiences that blend virtual objects with the real world, creating the illusion that these objects actually exist.

AR can be surprisingly immersive, and yet at the same time, it's always grounded in what's really happening.

It allows you to visualize things that would be impossible, risky, or hard to do from a practical standpoint.

Blurring the line between imagination and reality, an AR experience really feels like magic.

You can completely transform people's surroundings through visuals and sound so they experience something new.

You can also add new layers of information to the environment to enable quick and lightweight interactions.

Designing for augmented reality can be very different from designing for 2D applications, but don't worry, we'll break it down here.

Today I'm going to share criteria you can use to decide if AR is right for the experience you're building, along with tips for handling some of the unique aspects of augmented reality when you're designing your first AR experience.

OK. So you're designing an app or feature and wondering whether AR makes sense for you.

Here are a few things to consider.

AR can help you deliver a true representation of things.

To show what I mean, let's look at an example of something AR does really well, which is to present the real-world size of an object.

Say you're trying to educate people about how big a dinosaur is.

You could just write it out, or show a picture, or you could unleash a virtual T. rex on the scene.

In that case, there's no need to imagine how big the dinosaur is.

The AR experience gives the viewer an instant, visceral sense of scale in a way that's truly immersive.

This example comes from the app Monster Park, a game that lets people explore and control the movement of animated three-dimensional dinosaurs.

It's fitting that the entire app revolves around this large-as-life AR experience, also emphasizing the creation of photos and videos for sharing.

AR is at its most delightful when it affects and responds to what's immediately around you, involving physical space in a meaningful way.

Say you're trying to design an app that helps with interior design, and you want to help people choose a paint color for their walls.

You could use swatches or stock photography, but those aren't very accurate.

Even better, you could show someone how their very own room would look with freshly painted walls.

The Color Snap app does just that.

It allows people to choose a paint color and then tap a wall to preview the effect.

Because it's AR, people can move around to see how the new color will affect their space from different vantage points.

Next, augmented reality allows you to create experiences that visualize virtual objects in 3D, which makes it easier to understand and evaluate them.

With AR, you can build features that help people try on things in a way that's virtual yet realistic enough for someone to make a confident purchasing decision.

For example, the Warby Parker app lets people try on virtual glasses so they can place an order knowing exactly how their frames will fit.

It's faster to switch between frame styles in AR than it would be to swap glasses in real life.

Grounding virtual objects in the scene is also really important.

The quality of lighting and shadows in your experiences can have a huge impact on people's feeling that things are really there, even when an object isn't very realistic.

The IKEA Place app allows people to place a piece of virtual furniture in their home and get a more accurate sense of what the updated room would look like.

IKEA Place takes advantage of ARKit's built-in lighting for realistic shadows that really settle the virtual furniture into the room.

Visualizing in 3D can refer to objects, but don't forget about using people's surroundings as a canvas for your experiences.

You can create new features with advanced visual effects, provide maps and 3D models that fit your brand, and even bring people's spaces into the game you're building with iOS's new RoomPlan API.

We've got a session on creating 3D room scans that goes into detail about RoomPlan if you want to learn more.

AR lets you streamline actions by attaching digital capabilities to physical things, making your apps and features more efficient.

You might have already experienced how using the real world as input can speed things up if you've ever scanned a code to order at a restaurant.

You can use AR to build compact yet incredibly useful features that save people time.

In iOS's Measure App, centering the camera on a person automatically displays her height in just the right spot.

The experience feels lightweight because there's very little UI, the interaction is based on device movement, and the measurement information disappears after you've seen it.

This AR experience would be too limited to offer as an app on its own, but it really complements the other capabilities of the Measure app.

So, now you know you can use AR for your apps and features when you want to deliver a true representation, involve physical space in a meaningful way, visualize in 3D, and streamline actions by making use of what's in the environment.

And as you can see, there are a lot of different ways to incorporate AR into the experiences you're creating, whether it's as the main focus of the app or with a lighter hand as a feature that reduces friction.

So, you've decided AR is right for your purposes and now it's time to get started with design.

When you're designing for AR, you'll need to consider some of the aspects that make it different than traditional 2D interfaces.

AR is spatial.

AR is movement-based.

And AR is tied to the physical environment.

You'll need to think about how to blend virtual elements into the scene, about ergonomics, and about working with a limited field of view.

Now I'll go through tips you can keep in mind when designing AR experiences.

People using your app might be in places that are difficult for AR, so what you can do to help is coach them through different ways to get to a better experience.

The Mission to Mars app does a great job of getting people set up for success with a strong coaching sequence.

Let's take a closer look at this example.

As you're getting started, Mission to Mars highlights three different requirements you need in your environment to have a great session.

First, there's a reminder that AR can be immersive, and it's wise to find a safe place to engage with it.

This can be a particular concern for experiences that involve wayfinding, where you'll need to take extra care not to draw people's attention to the screen for long periods of time.

Next, the app suggests that AR performs best on surfaces with texture, rather than on glass or smooth white featureless planes.

LiDAR can help overcome some of these challenging environmental conditions, so you might be able to omit this suggestion for some devices.

And last, Mission to Mars advises people to find a bright space because AR works best in environments that are properly lit.

What's great about this coaching sequence is that it's quick, easy to navigate, and helps people understand how to set up their environment for the best AR experience.

The next tip is to take advantage of screen space.

You'd be surprised how large a role it can play in AR experience design.

So, what is screen space? Think of screen space as a 2D layer that sits on top of the three-dimensional world that's captured by your camera view.

It's best to place text and interactive elements like buttons here in screen space, rather than in the 3D world of the camera to preserve legibility.

Here's a great example from the Mission to Mars app that illustrates how to handle text in screen space.

High-contrast text and buttons sit in a 2D screen space layer on top of the 3D scene below.

The text updates as the camera moves to help you understand how it relates to the 3D objects in the scene.

If you do need to lock your text to something in the world, try to keep it billboarded or parallel to the screen when possible.

Focus on increasing contrast, bumping up type sizes and putting text on a background so you can maintain accessibility.

Design for constant movement.

It's important that people understand how and when to move their devices.

Sometimes they'll need to move their bodies as well in order to experience AR.

Try to provide real-time feedback with visual effects and sound as people move, so that even if the action is taking place out of view, people can still connect with the experience.

Use simple, glanceable instructions and animations to coach people how to move.

It helps to place instructions in screen space and to show them on an as-needed basis, rather than all up front.

It's great to leverage the built-in coaching animations, but you can also create your own onboarding instructions that relate to your app, like this example from the app DoodleLens that shows an iPhone panning back and forth in front of a doodle.

Ergonomic considerations are really important for AR.

It can be exhausting to hold your arm out for long periods of time, and uncomfortable to reach for buttons that aren't positioned properly for one-handed use.

Emphasize legibility at arm's length for the entire interface.

Simplify interactions so that they can be completed with minimal effort.

Use oversized buttons with high-contrast icons that are easy to tap one-handed with a thumb.

Here's a nice example at the bottom of the screen in DoodleLens.

People are viewing AR experiences with a limited field of view offered through a handheld device, so there's a good chance they might not be able to see the full extent of something large.

There's also a possibility that a virtual object might be located out of view.

Allow people to adjust the scale of an object, in case they're not able to back up far enough to see the whole thing.

This example from AR Quick Look uses a pinch gesture for direct manipulation.

There's also a haptic so that you can feel it when you scale something past 100 percent.

When something is out of view to the right or to the left, use sounds and haptics and provide simple indicators or text instructions in screen space.

It can also be helpful to provide a map or bird's-eye view showing someone's orientation and illustrating that the object is located in the opposite direction.

In this example from the RoomPlan experience, the small 3D model drawing at the bottom of the view helps you preview your results and keep track of what has been scanned so far.

It can be difficult for people peering through a screen to understand the location of virtual objects in space unless they're behaving realistically.

Depth cues that help people see how far away things are can help with this.

The size of objects, perspective effects like diminishing towards the horizon, realistic shadows and lighting, the proper amount of detail in your textures, and overlapping objects create a feeling of depth and help people perceive spatial relationships.

I want to highlight one depth cue that's tricky to work with but can really help people understand where things are located relative to each other: overlapping, also known as occlusion.

Here's an example of overlapping you can see in AR Quick Look.

The virtual airplane seems like it's positioned behind the wooden blocks on the desk because the lower part of it is hidden.

Finally, craft experiences that last no more than a minute or two.

This is for the ergonomic reasons I mentioned earlier, but also because AR is a resource-intensive superpower that has a big impact on battery and thermals.

If you do end up creating a longer experience, make sure that you build in breaks.

For All Mankind: Time Capsule is an AR experience that allows you to explore the For All Mankind universe.

The app complements the TV show, telling stories through interactive objects.

Time Capsule is presented in chapters that provide moments of rest and a place to pause then return to the experience.

Today I shared tips to help overcome some of the challenges with working on AR features.

You've learned to guide people to the right environment, to take advantage of screen space, to design for constant movement, to think about ergonomics and a limited field of view, to use depth cues, and to limit the duration of the experience to keep people from getting fatigued.

Augmented Reality feels like magic when it's useful, delightful, and relates to the physical world.

AR allows you to see virtual objects or even to transform your environment.

You can attach digital capabilities to physical things to create a layer of useful information and actions.

The AR features and apps that you create will transform how people work, learn, play, shop, and connect with the world.

I can't wait to see what you'll build next in augmented reality.

Thank you! ♪