Explore immersive sound design

♪ Mellow instrumental hip-hop ♪ ♪ Welcome to "Explore immersive sound design." I'm Danielle Price, and I'm a sound designer on the Design team.

The content in this talk will include spatial audio examples, so for the best experience I recommend wearing headphones.

While not everyone watching this talk is a sound professional, we all have some expertise in spatial audio just from being human.

I'll illustrate this with an example you might be familiar with.

Let's say I can't find my iPhone.

I'll ping it from my Apple Watch and track it down.

I hear it out of my right ear...

It's getting louder; I'm getting closer.

Found it! I'm able to locate the sound because I'm aware of the direction and the volume.

As you can see, we use spatial audio constantly to navigate the world.

And in this case, I used it to locate my iPhone in the room.

Like we saw in the Find My example, in the real world, sound reverberates through and around our space, bouncing off walls and furniture before arriving at a person's ears.

Well, the system can mimic this behavior because it understands the spaces around you.

Now I'm going explain more about how Spatial Audio works on the headset and the different ways that we can design for it.

First, I'll break down how Spatial Audio works within the system.

Then I will talk through some examples about how to design sounds for the UI and immersive apps.

So back to this idea of recreating sounds from the real world.

The device adapts to different spaces and adds reverberation from your space to make things sound like they're really in the room.

Spatial audio sources will sound like they are closer or farther away depending on where they're placed.

I'm going to play the Find My sound a couple different ways so you can hear how this works.

First, here's the original sound that I exported.

Notice how it sounds clean and front and center.

OK, now here's the same sound but played in a different space, like this living room.

Also notice that the sound is coming from in front of you, but also that it's directionally coming from the right side of the couch.

The system considers the size and materials in your room to allow accurate playback of audio.

I know this is a simple example but this concept helps define how we hear sound on this platform.

You may already be familiar with Spatial Audio for music and media, and now you're able to extend that by playing sounds in your apps as if they are coming from where you place them.

I'll be referring to these sounds as spatial audio sources.

Let's look at how this applies to interaction sounds.

When we interact with the system using our hands and eyes, we're not touching anything physical.

So by adding subtle sounds to each interaction, we help give the user a sense of familiarity and confidence.

Here's an example of the virtual keyboard.

Listen for the sound of each key press and how it comes from its location on the keyboard in front of you.

Each time you interact with the keys, the familiar sound confirms your action.

A quick note on repetition: you may notice each key press sounds a little different.

When you know a sound will be played often, it helps to slightly randomize some elements of the sound like its pitch and amplitude.

It feels more natural and less static when keys are pressed in rapid succession.

Consider this to make your app feel more natural.

Next, let's look at another example of how we designed the UI sounds you hear in the system.

When we started designing the Photos app, there were certain interactions we wanted to emphasize with sound, like selecting a photo from the grid and transitioning from 2D to 3D.

Or when you're browsing through your photo album.

So, where do we start our sound design exploration? Well, we could record the sound of a window opening.

Or would that be a little too literal? Let's have a listen.

And if we go that route, maybe we can use the sounds of moving through projector slides for the browsing interaction.

While certainly interesting sound design examples, they don't really fit the visual aesthetic of the experience.

We wanted the sounds in Photos to fit in with the rest of the system sounds, and also highlight a sense of depth.

We think a good UI sound has to be subtle and give just enough feedback to be helpful.

That said, for your own app you can add some character, of course, but strive to make it fit with the overall feel of the UI.

With this in mind, let's have a look again, this time with the new sounds we designed.

Much better.

The sounds align with the rest of the OS sounds and match the timing of the transitions.

When you begin to add interaction sounds to your own app, keep in mind people may hear them quite often, so keeping those sounds subtle is important.

OK, now that we've learned about the role of UI sounds on the device, let's talk about using sound in some more immersive experiences.

Let's look at one of our environments, Mount Hood.

These environments are fully immersive experiences in the system.

Each place has a light and dark version.

And both have a realistic spatial soundscape to match.

The overall goal for these soundscapes is to be peaceful representations of reality.

They could play underneath the various apps you're using, and are meant to be subtle, not distracting.

Let's talk about some different ways to design, record, and mix sounds for these experiences.

As we began to think about the sound design for each place, one of the first things we asked ourselves was, how accurate should they sound? We went to the various locations to record, and the actual soundscapes often surprised us -- and not always in a good way.

Let's take a look at some of my personal footage from the Mount Hood location.

I just love this shot.

When we did preproduction planning, I would look at the images, close my eyes, and imagine the peaceful soundscape of birdsongs and soft winds in the distance.

Of course it was beautiful and those things do exist, but that wasn't all there was.

Let's take a look at footage from just to the right of where I took this photo.

What you're seeing and hearing is actually a large water drainage system.

And the noise is so loud you can't really hear any other nature sounds.

That noise reminds me of something else; let's listen again.

Not really what we were looking for.

As designers and developers, we have the freedom to curate the best user experience for our app.

In this case, we wanted to stay as close to reality as possible.

But if we have the chance to make something more pleasant than what actually exists, we should.

So feel free to recreate and curate the best of reality, making the sound of your application complement its visuals in the best way possible.

Now let's talk more about the audio recording process.

For the ambient background audio, we used a series of different microphones that record immersive audio, like the one in this photograph.

The goal for these recordings was to capture how the air sounds all around you in a place.

We would record all through the day and night to make sure we captured the entire range of sound activity.

Generally, we set out in search of places with as little noise pollution as possible, which usually required going off the beaten path.

These natural recordings would become the base layers that play under the spatial audio sources.

We would use high-sensitivity directional microphones that could zoom in on that specific sound we were looking for.

If you work with a professional sound designer, they can record sounds for you, but there are also plenty of professional high-quality sound libraries that could be a great source of sounds for your app.

Now, once we've recorded our sounds, how do we create a realistic soundscape? Remember the Find My sound playing from one point in the room? We used that same thinking for placing audio objects in environments.

Once again, we can draw from real-life experiences.

When we go outside, there are many different types of animals that make sounds from different locations and they all layer together to make a soundscape.

Our job is to recreate that with the right distance and placement.

So let's take everything we've learned and create a mix for Mount Hood.

There are two main categories of sounds we use when putting together a soundscape.

Spatial audio sources, which are sound elements that occupy a point in space like birds, crickets, frogs.

These sounds were generally placed in the foreground and midground of an environment, but you can place them anywhere in your app.

And the ambient background audio, this is the overall ambience of a place.

They are surround audio files anchored to play all around you and loop continuously without being noticeable.

I'll use these emojis to represent where we placed them in environments.

First, let's try placing the sounds of the crickets and frogs on the left and the right.

Now let's adjust this.

It's way too loud and feels too close, so we need to do two things.

First, turn down the volume of each a few decibels, And then push the location of the sound into the distance so it can sound further away.

Let's listen again.

It's starting to feel more natural.

Now, let's add a couple of frogs in the foreground, on the shore line.

That sounds really great, but we don't want to hear the same frog over and over again from the same spot.

So if we use randomization, we can create a more natural soundscape.

We could achieve that by alternating between a collection of different frog recordings as well as the location they are playing from.

From there, we can randomize the timing of when they are played.

OK, now let's listen to everything together.

There's still one more thing we need to add.

Remember the ambient backgrounds we mentioned earlier? Let's listen with the visual.

These sounds are played in surround, softly adding ambience to the space around you.

Now, let's listen to all the sounds together as it's experienced in the mix.

So let's recap our visit to Mount Hood.

Randomization will be a recurring theme.

Like we mentioned with the virtual keyboard typing, and with the foreground frog sources, be careful when repeating sounds over long periods of time.

Use randomization to affect how things play and to mix it up.

Consider placement.

Putting sounds in the distance on the system is a tool to move things back in space and sound farther away.

And in parallel with distance, sometimes you need to turn down the volume of an asset to achieve the right balance.

Now let's look at the same concepts we used for Environments and apply them to an example that doesn't use a fully immersive space.

As an opening moment to the onboarding experience, we wanted to fill the space with sound but also call attention to the animation writing “hello.” Here, too, we make use of the foreground and background.

Here's what it sounds like with just the foreground sound.

The animation indicates the source of the sound.

It's important to note that even if your app takes place in your physical room, you can still use an immersive soundscape to fill the space with sound.

Like with the environments, we used a combination of a spatial audio source and an ambient background sound.

Let's take a look at the video again with the ambience added to the mix.

You'll hear it echo in the background and fill up the space.

Whether you're moving on quickly or staying in an experience longer, this layered approach keeps it interesting and interactive.

In your own app, this could work well for a menu or title screen.

Let's recap.

We looked at how Spatial Audio enables us to attach sounds to the UI we see around us in the system, how sound can play a big role in making the UI feel responsive and confirm users' interactions, and how we can design an immersive spatial experience that's lively and captivating.

I hope this helped inspire you to use sound to enrich the experience of your app, making it more engaging and fun to use.

Thanks for listening.

♪