Advances in CarPlay Systems

Hi, this is Tom from the Car Experience Team at Apple and welcome to our talk on Advances in CarPlay Systems.

As you know, CarPlay is the smarter, safer way to use your iPhone in the car. iOS 13 has a ton of new updates to CarPlay including the all-new CarPlay Dashboard for a simple view of your tunes, turns and Smart Siri suggestions. In this session we will introduce you to vehicle integration features newly available in iOS 13. All of the features we will talk about today will require the vehicle system to adopt new APIs available in the CarPlay Communication plug-in R15. Today we will introduce new display and audio features in CarPlay that you can take advantage of in future vehicles.

First up, let's look at how CarPlay supports irregularly shaped displays.

The vehicle declares the physical size and resolution of the display and iPhone will setup an H.264 video stream of that size which will be rendered to the display.

Prior to iOS 13, the CarPlay UI would be drawn up to each edge of the video screen. However, some car displays are fitted with pieces of trim that may obscure parts of the CarPlay user interface, restricting access to some touchable controls and preventing the user from being able to see certain parts of the UI.

With iOS 13, CarPlay now supports irregularly shaped displays. First, your system defines a view area. This represents the rectangle that CarPlay can draw a user interface image into.

A view area can also contain a safe area.

The safe area is also a rectangular area and must be a subset of the view area.

The safe area represents the portion of the display that will contain interactive content. This includes user interface elements such as buttons and lists and important status information like time, signal strength and route guidance information. iPhone will draw interactive content within the boundaries of the safe area.

Modern car displays can come in many different shapes and sizes. Displays themselves may be non-rectangular or portions of the display may be occluded by other elements. With iOS 13, CarPlay can use safe areas to better support non-rectangular shaped displays. First, the system defines a view area. The view area rectangle in this case would be made up of the widest and tallest points of the display. This allows the CarPlay image to render to all parts of the display.

Then a safe area rectangle will be declared for the part of the display where a CarPlay UI is guaranteed to be visible and interactable to the user. And that was our update on irregularly shaped displays. Next up, let's talk about second screen support in CarPlay.

Prior to iOS 13, iPhone provided a single H.264 video screen to display the CarPlay UI on the center console display. This provides a great way for users to set a navigation destination, interact with music apps and more. However, this limits the user's CarPlay experience to the vehicle's main display.

Many vehicles have rich displays directly in front of the driver to show important information in the form of an instrument cluster. In iOS 13, we now support multiple H.264 video streams to display CarPlay content in the vehicle's instrument cluster. This is a great way to display additional information from CarPlay apps directly in the instrument cluster.

And different types of information can be displayed in different areas of the instrument cluster using more than one video stream. In iOS 13, CarPlay will support a maneuver instruction card provided by Apple Maps, as well as a map view which can be rendered in parallel with the instruction card in two separate video streams.

Each video stream will have an independent night mode, allowing the vehicle system to specify an appearance mode for different content types. Also, the vehicle chooses what type of content is shown in each stream. For instrument clusters, car play will only draw the type of content the vehicle has selected. But please note for heads-up displays, information from iAP2 Route Guidance will continue to be used. So with this in mind, let's look at how to best utilize the different available protocols to integrate CarPlay into a vehicle instrument cluster.

Users can be provided with simple status information from apps in CarPlay such as navigation, audio and phone calls. Firstly, map or turn-by-turn information provided as a video from iPhone can be displayed.

Additional information about the navigation route can also be provided as metadata using the iAP2 Route Guidance feature. This, like other iAP2 features, requires the vehicle to draw the metadata in its own native UI. The currently playing track or podcasts can be rendered using iAP2 Now Playing along with any information about ongoing phone calls using the iAP2 Communications feature. Control of those phone calls can be supported using iAP2 Call Controls, allowing the system to offer specific user actions during phone calls like mute, swap call and end call.

Displaying CarPlay content in the instrument cluster is another part of the vehicle system that benefits from using safe areas and view areas. In this example, the view area would be the rectangle between and behind the virtual tachometers. While aesthetically it may be desirable for CarPlay content to fill this region, some of it would be obscured by the virtual tachometers. So a safe area would be declared in the area between the two tachometers and CarPlay will position the important information where it will be visible to the user. And multiple streams can be set up to display different types of content from CarPlay within the instrument cluster. For example, the second stream could be a map view and the third stream an instruction card. Displaying content within a virtual tachometer will also make use of view areas and safe areas. The rectangle containing the entire circle would be the view area, and the rectangle contained entirely within the circle is the safe area. Again, CarPlay will position the important information where it will be visible to the user.

So that was our update on support for second screens.

Now let's take a look at the last of our new features that affects displays in CarPlay, dynamic screen resizing.

Prior to iOS 13, the size of the CarPlay image remained fixed while the CarPlay session was active. Now the size of the CarPlay UI can change on the fly at the user's request by declaring multiple view areas for that display. This is ideal for systems that support configurable horizontal split screens as shown here, and larger portrait screens that are normally divided into distinct context-specific regions. CarPlay can provide an Always Available button for the user to trigger a resize request from within the CarPlay UI.

This is useful to show CarPlay in a full-screen configuration without reserving space in the native UI for dedicated control for the user to exit that configuration.

Resizing is not only supported on the center console display but also on the instrument cluster as well.

So let's use the instrument cluster from the previous section as an example of how dynamic resizing could be used in CarPlay.

Let's say this instrument cluster offers the user a way to move the virtual tachometers into one of two preferred positions.

For this to work well, the CarPlay image needs to be able to adjust its size based on where the virtual tachometers are positioned.

Multiple view areas can be declared for this display to make this happen. Let's declare the first view area for the widest configuration and include a safe area as discussed earlier.

For the narrower configuration, a second view area would be used with its own narrower safe area.

The vehicle system can also specify how long the transition between those configurations should take, allowing the CarPlay content to transition sizes in the same timeframe.

Let's take a look at how this transition would work.

The user starts in the wider configuration and uses some native UI control, possibly a steering wheel button, to begin the transition.

During this three-second transition, CarPlay is telling the native system the size of the encoded image in the H.264 frame, allowing the vehicle system to synchronize the position of its own tachometer UI elements. So in summary, dynamic resizing is supported on center console and instrument cluster displays. A UI resize can be triggered by the user either through the native UI or a button within the CarPlay UI.

The vehicle system declares how long transitions between view areas should take. And CarPlay provides an update on how far the view area transition it is so the vehicle system can synchronize transition animations in its own UI. So that's all for our new display-related features. Thanks for listening so far. Now over to Tanya for an update on Siri in CarPlay.

Thanks, Tom. My name is Tanya Kancheva and I'll talk to you about enabling Hey Siri support in future vehicles. So what do we mean when we talk about supporting Hey Siri in your car? First, of course, the driver can launch Siri completely hands-free without needing to press a button or take their eyes off the road.

And since Hey Siri is detected using the car's microphone, iPhone can be out of sight, left in a pocket or even in the trunk. Siri will still be available.

We also updated audio handling so that activating Siri through a button press is instantaneous. Just hold the Siri button on the steering wheel and say your request without waiting for an audible trigger to speak. Siri will be back with your answer immediately.

Lastly, we looked at how Siri and Music interplay in the car and enable the smoother and more familiar experience. Music continues to play in the background while interacting with Siri.

Here is a system overview for audio handling in CarPlay. Some of the components should already be familiar to you, but let's go ahead and look into the details.

To support Hey Siri, the in-car microphone needs to be always active and undergo continuous voice processing.

The echo cancellation module runs all the time and removes any audio playing through the car's speakers as well as reduces background driving noise.

The process microphone data is sent to an audio ring buffer within the CarPlay communication plug-in which holds a couple of seconds of historical audio. This buffer is stored within the car until there is a trigger to send it to iPhone. So what is this trigger? In fact, there are two modules which can lead to audio exchange with iPhone. Both constantly analyze the process microphone data. One can trigger when the drive says Siri. That's the keyword detector. The other one triggers when the drive starts talking. That's the voice activity detector.

Both detectors must be available in the car as iPhone determines which one is used for a particular scenario.

Now let's say the keyword detector triggered so the car notifies iPhone through the communication plug-in.

iPhone opens a new audio stream called Aux In, requesting some historical audio data to be sent.

The received audio is reanalyzed by a voice trigger detector on iPhone which verifies that the user said, "Hey, Siri." The second pass detector uses offline keyword detection on iPhone similar to how Hey Siri is detected when the user is not driving.

Only after the second pass detector on iPhone triggers Siri is activated to process the data and respond to the user.

Now, let's look at how this architecture is helping us improve triggering Siri with a button press. As we already saw, the mic, the ECNR unit and the audio ring buffer are processing audio within the car system all the time.

Now the user presses the Siri button on the steering wheel that is the trigger for iPhone to request audio data from the time when the user pressed the button. Since this is the button press, Siri activates immediately. And as the buffered audio is sent faster than real-time, Siri can be ready with a response right after the user stops talking.

Okay, now that we have discussed the three ways in which Siri can trigger from the car through a detected keyword, voice activity or a button press, let's see what happens when Siri is active and helping the driver.

Typically, music will be playing in the car. In this case, let's say that music is coming from iPhone. The car will receive the audio, mix it and play it through the speakers. In the past, when Siri started in CarPlay, music had to stop and there was a bit of silence before Siri prompts played.

With this new architecture, media playback is separated from Siri prompts so the music can keep playing in the background.

When Siri launches, an additional audio stream dedicated to Siri output, Aux Out, will be opened. The car will then mix the two streams, reduce the volume of the music as needed and play Siri prompts quickly and without big interruptions.

But please don't forget that in addition to Siri prompts, there might be active route guidance, so the mixer has to be able to mix three parallel audio streams coming from iPhone.

All right. Now that we have gone through the details of the audio architecture, let's quickly summarize what is needed within the vehicle system.

First, a microphone which is always processing audio in your system.

Then, an echo canceller and noise reduction module to clean up microphone input. A detector that triggers when the driver is talking. A detector that triggers when the Siri keyword is heard.

The ability to mix three audio streams from CarPlay.

And that's how our update on Hey Siri.

Thank you for your time. For more information, check out developer.apple.com. And enjoy your drive with CarPlay.