Meet the next generation of CarPlay architecture

Hi there! My name is Tanya, and I’m here with you today on behalf of the car experience team. We are so excited to finally share with you all, what’s behind the scenes of the next generation of CarPlay.

I hope you’ve already seen the amazing video by my colleague, Ben Crick, on the design system and you are ready to dive into how it all works! But before we get into the details, let’s start with an overview.

Cars have changed a lot since CarPlay first launched.

With larger screens, more of them and with a ton more features. People love bringing their favorite features from iPhone into the car with CarPlay.

iPhone users have the choice to use CarPlay, or this next generation of CarPlay that provides content for all the driver’s screens. A fully integrated interface, for the most seamless, consistent, and feature rich experience possible.

An experience that also fits perfectly into your specific vehicle’s configuration. It’s the very best of your car, combined with the best features and familiar interaction patterns of your iPhone.

And with that, today, we are going to go over the Architecture.

Next generation of CarPlay UI. And finally, how to enable custom features.

Ready? Let’s find out what features have been added to enable this new experience.

We start by building on the existing CarPlay architecture. Advanced CarPlay features like enhanced Siri, buffered media playback, and support for HEVC video streaming are essential. The next generation of CarPlay is a wireless-only experience. It reconnects and appears seamlessly if you were using it on your previous drive. The stability and performance of the wireless connection are essential for a smooth experience while driving.

And just as a recap...

and this is basically how CarPlay works today. You might have already seen this before. The Communication plug-in serves as the main interface between your system and iPhone. It delivers video and audio content, and accepts user input from the vehicle. In a typical CarPlay configuration, that means iPhone provides a single video stream which is then handed to your system to decode and composite before it’s displayed on the screen. Easy! Now… if we look at the next generation of CarPlay UI and everything that it shows, we might need couple more things.

We will go through the architecture focusing mostly on the cluster display since it comes with some very specific considerations. But of course, the same concepts apply to all displays in the vehicle.

So what do we actually see here? Let’s break down the UI into layers.

We refer to the frontmost layer as the Overlay UI. And yes, Overlay UI includes some pretty essential UI elements. The remaining layer includes all the rest. Both layers are composited by your system’s compositor and are displayed directly on the screen.

So what exactly is the Overlay UI? Overlay UI includes vehicle-specific telltales and essential indicators. They may be redesigned, and in some cases repositioned to match the new UI experience. But the key point is, that they are rendered entirely by your system.

Now, what’s in this remaining layer which holds the majority of UI elements? Again, we can split it. And each layer has a very specific purpose. We refer to the bottom one as Remote UI.

Remote UI brings a rich and familiar iPhone experience to the vehicle. Maps and audio apps come to life here in addition to features such as tire pressure or trip information. And as expected, remote UI extends to any other display bringing additional iPhone content such as media, climate or more. To achieve all of this, we now need one iPhone video stream for each display in the vehicle. So what does this mean for the architecture? From what was a single display, we now need to display content on additional displays. There’s not much new here. In principle, this is already supported in CarPlay today! So what’s next? Well next is Local UI. Local UI is a little bit more interesting. It actually shows driving-related gauges and indicators! What does local UI give us? It allows to have UI which is rendered with low latency which can respond to high frequency car signals. It’s robust and not affected by Wi-Fi interference or disconnects, since data stays local to the vehicle and finally, it is fast to start. Remember, the next generation of CarPlay starts instantaneously if you were using it on your previous drive. As soon as the displays are lit, or maybe when the door opens or maybe when the driver is approaching the car, content is ready to show. This may be even before iPhone has been detected or reconnected as you hop in and drive off quickly. All of this is achieved with the Local UI which is obviously rendered directly, or locally, on your system.

So how does the Local UI work? It starts with an OpenGL-based renderer that enables rendering on the vehicle itself.

We augment it with an asset package, which includes images as well as behavior scripts. These scripts implement logic to help generate the UI. Asset packages are specific for each vehicle, and are transferred from iPhone during pairing.

Before using any new asset package, your vehicle verifies its authenticity. This mechanism brings all the benefits of local rendering, but also opens up the possibility for the UI to be refreshed over time. Cool, right? Let’s add these components to our architecture.

We start with the local renderer and add the assets and scripts for your specific vehicle. And of course, as you see we do that for each individual display. That’s it so far. Let’s move on.

The final layer is something we call Punch-through UI. Punch-through UI allows you to incorporate vehicle-specific UI into the experience. Features such as driver assistance, external cameras, or any other visually rich feature, on any display, can be expressed as Punch-through UI. Without a doubt, these pixels are all rendered by your vehicle! But the key here is, the UI still feels integrated! Punch-through UI is elegantly revealed when displayed, and it participates seamlessly in transitions and animations. It just fits into the overall experience. So how is this achieved? Before we get to that, let’s review what we’ve covered so far.

First, we looked at Overlay UI and how it’s composited directly by the vehicle. Then we looked at Remote UI, Local UI and Punch-through UI. These three layers are processed by a dedicated compositor, which uses OpenGL to provide a single unified output frame to the system’s compositor for display on each screen.

So back to our architecture, let’s add that compositor.

The compositor receives the video stream from iPhone, as well as the locally-rendered UI.

And now it directly receives Punch-through UI as well.

As you have gathered so far, the next generation of CarPlay incorporates content rendered both locally on your car, as well as remotely by iPhone.

Yet, the experience includes seamless layout transitions and animations, just like what we’re familiar with on iPhone. To achieve this, we use frame level synchronization.

As in CarPlay, each remotely-rendered, iPhone video frame carries a presentation time stamp that specifies when it should be actually shown to the driver.

The next generation of CarPlay adds a dedicated, low-latency channel from iPhone to each display called UI sync.

With that, iPhone carefully orchestrates the transition of UI elements as they move across the display, irrespective of the rendering source. The behavior scripts interpret this information and match frames from Local UI, Punch-through UI, and Remote UI, one frame at a time, at 60 frames per second. Or even faster, if that’s what your displays are capable of. Magical isn’t it? And you play a crucial role in this. The vehicle system ensures timely delivery of video frames and UI sync packets throughout the system, and preserves the associated time stamps.

Lastly, let’s see the full graphics architecture come together by adding the Overlay UI, which, as we learned, goes straight into your vehicle’s compositor. And that’s it! Now we have a full picture of how the UI for the next generation of CarPlay is created.

Now let’s learn what’s behind the scenes.

What drives this UI? And that’s the state of the vehicle. The next generation of CarPlay relies on the vehicle system to always provide up-to-date state information which is then reflected in the UI.

And depending on where a specific information is rendered, or where it’s available within the system, multiple interfaces are available.

And of course, we at Apple believe strongly in protecting the user’s privacy and with this approach to vehicle state we can ensure data is used only where it’s needed, and in many cases, it never leaves the vehicle.

There is a dedicated vehicle state interface for the UI displayed on instrument cluster display, and a similar interface for Local UI on the center display. And for Remote UI, the Communication plug-in has been extended to support vehicle state. Again, where vehicle state information is actually provided depends heavily on integration within your vehicle and the availability of information. And with that, we’ve completed an overview of the architecture that enables the next generation of CarPlay.

Next, let’s understand how this new user experience comes to life and what it takes to interact with it.

When we talk about interacting with the next generation of CarPlay UI, there are three major tasks to consider. First, you configure the UI to feel at home and match the unique capabilities of your car. Second, you ensure the UI is always aware of the state of the car. And finally, you display UI in response to vehicle conditions or user interactions. To accomplish any of these tasks, you use the vehicle state protocol.

Let’s start with configuring the UI. The next generation of CarPlay is highly flexible, and adapts to the feature set of your car.

UI elements can be added or removed to match the availability within the car. Different models, trim levels, or detailed configuration options can be easily mapped to specific features using the vehicle state protocol.

All that UI configuration happens when vehicle states are initialized, and remains fixed for the duration of the session.

Let’s look at a specific example - climate.

The next generation of CarPlay includes a full-featured climate control UI that’s designed to accommodate a wide range vehicle models with different features and capabilities. But, let’s erase it all, and start with a blank state and configure it as we wish.

First, we add a custom image that represents the interior of your car. Let’s imagine you need to support a 5-seater sedan with three distinct climate zones. You use Vehicle Resources and in this example you specify an identifier, "5Seater," that matches a variant to an interior image. It’s entirely up to you how many variations of this image you want to support. For example, if the car can be outfitted with sport seats, or red seats, and you want to express that detail here, you can! It’s super easy to add new identifiers and matching image assets.

Now, we configure the climate feature. Then, add controls for each supported zone. Let’s start with the driver zone. Here we imagine the driver can adjust the temperature, fan level, and vent directions.

We create an instance for each one of them. Temperature, and we make it nice and cozy - and fan, but how do we know that all this is for the driver? VehicleLayoutKey does that for us. In this car, the driver is sitting on the left, so indeed this is our driver zone. We also initialized these controls when we created them. Similarly, we create instances to represent the three vent controls in this example. First, the upper vent. Then the middle one, which happens to be on. Then the lower vent. Of course if you have seat heater or fan, you can add them in the same way.

The passenger zone follows the same pattern as it has the same controls, so we won’t get into much. We just configure it in the same way. Notice the passenger had set their climate a bit differently.

Let’s skip ahead to the rear zone.

In this example, passengers in the rear seat can only adjust the temperature and the fan level. So we just configure two instances and we assign them to the rear seat by using "seat_2nd_row" and provide initial values. Our rear climate zone is ready! Finally, let’s configure controls that apply to all zones. In this example, we’ve added a recirculation control to toggle between in-cabin and outside air, a cabin control to turn the A/C on or off, and a selector to synchronize zone settings. Well, in this example, when the SYNC button is pressed, just the passenger zone gets synchronized to the driver zone. So we configure that by using the vehicle layout key again. And that’s it! With this, we have configured climate control for this amazing car! And just like that, with what we just did, the driver also gets climate controls in the dock, even when the full climate UI is not showing. Tapping the controls provides easy access. Now that we’ve discovered how to configure the UI, how do we keep it up to date? To keep the UI responsive and accurate, vehicle information needs to be updated all the time.

When the driver presses a hardware button or some on-screen control, you will get notified of requested change of state. Act fast, execute the action and adjust state as requested.

When a change of state happens within the car, inform the UI immediately. Delayed responses are not welcomed here.

And finally, any values you provide should represent exactly what you want the driver to see. Any smoothing or adjustments are executed by the vehicle before forwarding to the UI.

The next generation of CarPlay simply shows exactly what you send.

So, how does this work in practice? Here are two locally-rendered gauges which require high frequency updates. On the left - the speedo. The speedo expects to receive the speed in three different units at all times. The highest precision value, meters per hour, the top one, is used for smooth gauge animations. The other two are used for numerical presentations of the speed. All units are needed at all times, we don’t want the UI to do math for us. The UI simply shows what you provide.

As we start driving, the speed increases, the gauge animates. Simple! And on the right you see the RPM gauge. The RPM gauge gets updated with a single value and an associated state. The state is set to Normal most of the time, but if the driver manages to exceed the maximum engine speed, it’s up to you to update both the value and the state to be reflected in the UI. As you can imagine, this is how it looks in practice.

So far, we looked at a couple of examples, but everything we learned applies to all features.

Configure what you want to see, and keep it up to date.

Next, how do you request some content to show? When the driver interacts with the car you might want to show something in response.

Maybe when the driver presses a button or as a response to a specific vehicle signal.

So what's available? In instrument cluster, you can show different types of notifications and warnings, a custom driver assistance view, or a custom picker for user selection.

On the center display, remote notifications are as flexible and versatile as you need them to be.

Let’s look at an example of displaying a tire pressure warning.

Here, we’ve already configured the tire feature with four instances of TirePressure, and a RequestContent instance for display in the cluster.

You keep updating Pressure and PressureState while driving, but when the state goes into low pressure, you want to actually alert the driver. You trigger the tire pressure UI to show. Notice that here you are allowing the driver to manually dismiss the warning. Now the tire is getting really flat, the state goes to deflation and the UI reflects it. Again, same basic principles apply when you want to show other warnings or notifications. Same for the center display as well.

Well now, where would we be without a backup camera?! All cameras shown in a car are a fun challenge! And the solution is interesting because it’s a mix of Punch-through UI for your actual camera stream and Remote UI for your custom user controls. Here we are looking at a reverse camera. It could be triggered by the driver shifting into reverse, or by pressing a hard key or button in the UI. In all cases, you’ll use two different interfaces: AutomakerInputStreams to show the Punch-through UI, and AutomakerExteriorCamera to show the user controls you’d like to see. Of course, these same APIs can be re-used to show a birds-eye camera view, parking assistance features, and many more. And be aware, the camera punch-though will show quickly when the driver jumps into the car, even if iPhone is not connected yet.

Remember: punch throughs are locally rendered.

And this is how it works in practice.

You call the APIs at same time, and you are all set to back into your parking spot.

Alright. Now that we’ve learned how to use the features included in the next generation of CarPlay, let’s take a look at how you can customize further and create some special experiences for each of your cars.

You have a wide range of options when it comes to customization. You might want to add some new experiences by showing highly visual and specialized UI. You might want to offer specific settings and settings menus, or you might want to display custom alerts or notifications to interact with the driver. All this is possible. Let’s just dive right in and take a look how to do all this.

The most flexible way to showcase your car’s features in the next generation of CarPlay is to use an automaker app. Automaker apps run on iPhone, which means you can leverage the full power of iOS to create your own rich, and custom UI. And of course, you can keep updating the experience over time. If you wanted to show custom settings, the driver can find them under the vehicle settings app. Let’s tap on it.

You see a very typical list of settings options.

The key here is - you define absolutely everything.

The type of setting you are using, the labels, icons, functionality, the structure, the order...

It’s all entirely up to you. And you simply use the fully customizable automaker settings APIs to do so.

Next, you might want to highlight some settings or features you just defined and make them easily accessible.

You just use the ProminenceInfo API, change the look and feel a bit if you’d like and your settings will show up at this top level.

Some of your settings might actually belong to one of the next generation of CarPlay experiences, maybe under climate or under audio or media settings. Easy to do as well! Let’s tap and see what’s there.

All of these are custom settings you’ve defined. And as you see, here you can also highlight what really matters to your drivers.

But what else can we do? You want to add a way for the driver to set a climate schedule. And then, you remember, you already have this. it’s in your iOS app. So can we just re-use it? Well, if your answer is yes, then you can re-use it. Simply configure this setting to be a deep link one, and it will take the driver straight to your app.

Small pause here.

With this capability, now the driver can access the functionality of your iOS app from custom entry points which you defined and added throughout the next generation of CarPlay UI.

Well that’s huge! Right? Now, to continue, some features or settings may be most applicable to a specific climate zone within the car.

Again, the VehicleLayoutKey enables you to make these massage settings handy for the driver.

Now when the driver taps on the button you can choose if you want to link to a sub-menu with more options, or, if the built-in UI already has advanced visualization of the seats, you can link to it using Punch-through UI. Of course, these are just two examples. No matter where within the UI you add a custom setting you can always choose what to show once the user selects it.

Similarly, as with your automaker app you can now basically add entry points to your custom settings menus, or built-in Punch-through UIs within the UI of next generation of CarPlay.

Next, let’s move to showing custom notifications. First, let’s look at the instrument cluster display. You can fully customize these notifications - labels, icons, buttons - they are designed to allow you to easily bring your existing notifications to life. You decide which one to show. You decide when to show one. You decide when to remove it. You can also choose if these notifications are rendered locally or within remote UI based on your use-case.

Now looking at the center display, the same highly customizable functionality is available.

You can fully configure notifications as you wish, again, using automaker notifications. And you can do even more using exactly the same APIs. Let’s say, as an example, the driver presses a drive mode button in the car.

You show what the current setting is and allow some more options with the configure button.

Now again, when the driver taps on it, you can choose what to show after that button is clicked.

Some already familiar options appear here! Link to your app, or to a custom menu, or to a built-in UI Punch-through. Your choice. Again, these are some simple examples, but they open a lot of possibilities and we honestly can’t wait to see how you use these options to bring the next generation of CarPlay into your vehicle! We’ve seen how the next generation of CarPlay architecture builds on a modern CarPlay foundation, and adds new capabilities such as in-vehicle rendering, composition, and support for vehicle state.

It’s easy for you to configure and build a UI that matches the features in each of your vehicles. And it provides a wide variety of options for you to bring your special features to life with customized UI. If you’re interested in learning more, check out the linked videos, and for access to technical documentation, enroll in the Apple MFi Program.

And that’s it! Thank you for going through the architecture with me. Enjoy bringing this next generation of CarPlay into your vehicles.

Can’t wait to see what you do with this!