Discover Reference Mode

Hi, I'm Ben Bodner from the Display and Color Technologies team at Apple.

In this video, I'll explain the capabilities, use cases, and APIs of the new Reference Mode on iPad Pro.

To help you understand Reference Mode, I should first describe what a reference display is.

Many displays intentionally manipulate the color, contrast, and sharpness of images.

For example, TVs usually have different modes that each target some unique aesthetic appearance, and mobile devices often adapt to ambient lighting.

While these sorts of features may be desirable in most situations, they can be problematic for content creators who usually want to know exactly how their display is rendering images.

For that reason, reference displays render color precisely to the specifications for a media format.

That means if you have different reference displays, they will all render the same content consistently.

This consistency gives content creators the ability to use multiple displays with the confidence that their work will look the same on all of them, assuming they are viewed in the appropriate lighting environment for their media format.

Now that we have an understanding of reference displays, let's get into the details of Reference Mode on iPad.

First, I'll explain when and how to use Reference Mode.

Next, I'll cover the specific capabilities of Reference Mode including which media formats are supported.

And last, I'll give a rundown of the APIs associated with Reference Mode.

So, when is it appropriate to use Reference Mode and how do we enable it? Reference Mode enables or enhances a variety of content creation workflows.

It adds support for color-critical tasks that require a reference display, which historically have not been possible on mobile devices.

This includes color grading, along with review and approval of graded content.

It also brings color accuracy to mobile video editing and compositing apps, improving the experience for users who typically would not have access to a reference display.

Finally, Reference Mode supports both Metal and AVFoundation-based color rendering, so apps serving these workflows can use either of them.

Reference Mode is similar to the reference display presets already available on some macOS systems but with an important difference.

On macOS, multiple preset modes are available that each deliver reference color rendering for different formats.

On iPad, Reference Mode is just one setting that is either turned on or off.

That means you can get reference color rendering for multiple formats without the need to change settings.

This mode can only be accessed through the iPad settings by a user and cannot be programmatically enabled or disabled by an app.

Reference Mode is supported on the 12.9-inch fifth-generation iPad Pro.

It can be used with an iPad as a standalone device, but it also brings new capability to Sidecar, a feature that allows an iPad to be used as a secondary monitor for a macOS device.

Previously, Sidecar did not support EDR.

But now, if Reference Mode is enabled on an iPad used as a Sidecar display, it will support EDR in addition to enabling reference color rendering for supported formats.

This new functionality also requires an Apple silicon Mac with macOS Ventura to be used as the host for the Sidecar display.

Enabling Reference Mode manipulates several other items in the Settings menu.

To achieve consistent color rendering, the brightness slider is locked and dynamic display features are turned off.

This includes disabling and locking True Tone and Night Shift, along with the Auto-Brightness feature.

Now that I've explained when and how to enable Reference Mode, let's take a closer look at what it means to run the reference color, along with the specific capabilities of Reference Mode.

Like the macOS presets I mentioned earlier, Reference Mode uses color management to support several different media formats.

This ensures that color is rendered accurately based on the specification for each format.

Of course, to determine how colors should be rendered, media needs to be tagged to indicate which format it is.

For reference color management to be valid, the display should only be viewed in the reference environment specified by the format standard.

The video format supported by Reference Mode require a dim environment, while graphics should be viewed in a bright indoor environment.

So what sort of color management does Reference Mode use? All formats of color managed to a similar spec.

This includes a D65 white point and a peak luminance of 100 nits for SDR media or 1000 nits for HDR media.

For HDR formats, tone-mapping is disabled so the display will precisely track the format's transfer function.

And colors will be displayed correctly as long as they don't exceed 1000 nits and are within the P3 color gamut, which is the iPad's native gamut.

Colors that are out of gamut or brighter than 1000 nits will be clipped with no tone-mapping.

The result of that clipping may vary based on the format.

Speaking of formats, here's a list of the ones supported by Reference Mode, along with the corresponding macOS reference display preset for each.

On iPad, Reference Mode will determine the appropriate color management based on color tagging, so there's no need to switch between presets.

One example of tagging are the ITU H.273 code points included in this table, which you may know as NCLC color tags from QuickTime.

But there are other forms of tagging that can also be used.

If your app uses a format that isn't on this list, don't worry.

Reference Mode still color manages unsupported formats; they'll just be rendered in a similar way to the default display mode.

You may notice that some of these rows include multiple formats.

That's because the formats listed in the same row should result in the same rendering on a reference display.

For example, HDR10 and BT.2100 PQ both use the BT 2020 color primaries, the PQ transfer function, and the BT 2020 YCC matrix as described by the code points 9-16-9.

But HDR10 also includes HDR metadata that can be used to inform tone-mapping.

In the iPad's default display mode, this could result in different renditions of the same content depending on the format.

But since Reference Mode does not tone-map, any HDR metadata is ignored and both formats are rendered identically.

Generally speaking, reference displays are calibrated with an optical measurement device to ensure color accuracy.

Like macOS presets, Reference Mode includes a fine-tuned calibration feature that allows the white point chromaticity and luminance to be calibrated at a single point.

This color adjustment is intended to account for display aging and differences between measurement devices so that your iPad's white point and peak luminance will match other displays that have been calibrated with the same device.

That wraps up the specifics of Reference Mode's capabilities in color rendering.

Next, let's discuss the UIScreen APIs associated with Reference Mode.

But before doing so, I should quickly recap what EDR is.

EDR stands for extended dynamic range and is Apple's HDR technology.

It includes both rendering technologies and pixel representations.

EDR pixels can represent both SDR and HDR content.

An important component of EDR is headroom, which is the HDR peak luminance divided by the SDR peak luminance.

In Reference Mode, HDR peak is always 1000 nits and SDR peak is always 100 nits, meaning EDR headroom is always 10 times.

EDR ensures that SDR pixels, which are represented with values from zero to one, are always rendered.

Any values between one and the EDR headroom will be properly rendered without tone-mapping, while values that are brighter than EDR headroom will be clipped.

However, there may be cases where you decide that the headroom is not high enough to justify rendering your content as EDR and instead want to implement an SDR path.

There may also be times when you prefer to make use of the available EDR headroom by applying your own tone-mapping.

In either case, UIScreen has new query properties and notifications you can use to make rendering decisions in your app.

More specifically, you can use UIScreen to register for notifications about changes in Reference Mode status and then query for changes in that status.

You can also use UIScreen to query the device's current and potential EDR headroom properties.

I'll talk more about that a bit later.

For changes in display mode status, register for the UIScreen.referenceDisplayMode StatusDidChangeNotification.

For query purposes, you can use this notification to get the new status and the new potential EDR headroom, both of which may inform rendering decisions in your app.

For example, you might want your app to apply its own tone-mapping based on the EDR headroom when Reference Mode is disabled and not apply any tone-mapping if Reference Mode is enabled.

Once you receive a notification for a change in display mode status, you can use the reference display mode status property of UIScreen to get the current status.

This will be one of four enumerated types.

If StatusEnabled is returned, then Reference Mode is enabled and is rendering as expected.

If StatusLimited is returned, then Reference Mode is enabled but temporarily can't render a reference response due to EDR headroom being limited to less than 10 times.

If this status occurs, a UI notification will also inform your users that the reference rendering is compromised.

If StatusNotEnabled is returned, then Reference Mode is supported on this device, but it hasn't been enabled by the user.

And lastly, if StatusNotSupported is returned, then Reference Mode is not supported on this device.

Next, let's take a look at some Swift code to see how you can take advantage of UIScreen notifications.

Here, we'll start by getting a default notification object from the NotificationCenter.

Then, we'll add a notification observer to your default notification object.

In this example, the screenChangedEvent selector determines the message that the receiver sends to the observer to alert the observer that the notification is posted.

The name of the notification to register for delivery to the observer is UIScreen.referenceDisplayMode StatusDidChangeNotification.

Now whenever the status changes, the function screenChangedEvent will be called.

In this example implementation of screenChangedEvent, we'll first get a UIScreen from the window scene.

Then, we can get the new referenceDisplayModeStatus property from the UIScreen.

You can also get the new potential EDR headroom value and use it in your application's rendering paths.

That covers Reference Mode status-change notifications, so let's move on to talking about EDR headroom.

Earlier, I mentioned two types of EDR headroom: current and potential.

These properties are relevant to any developer working with EDR and are not specific to Reference Mode.

Potential headroom returns the maximum possible headroom of your display.

For example, you may decide to use an SDR rendering path in your app if potential headroom doesn't exceed 1.0 or some other threshold.

The current headroom is dynamic and may be less than the potential headroom for a variety of reasons.

You may choose to apply tone-mapping in your app so that your content won't exceed the current headroom, which would cause it to clip.

Let's take a look at a simple example.

As in the previous example, we'll start by getting a UIScreen from windowScene.

From there, we'll get the potential EDR headroom property.

Then we can compare the potential EDR headroom value to a threshold and decide whether to implement an SDR rendering path or image processing path.

Here is an example of how to use the current EDR headroom property of UIScreen.

In a draw call, we can again get a UIScreen from windowScene and then grab the UIScreen property currentEDRHeadroom.

You could use this value to determine an appropriate tone map that will avoid clipping and apply it in your app.

Now that I've covered UIScreen APIs and reviewed some code snippets, we've reached the end of this video.

To recap, we learned how and when to enable Reference Mode on iPad.

We discussed reference color rendering and the specific capabilities of Reference Mode, including which media formats it supports.

And finally, we learned about UIScreen Reference Mode notifications and new properties for querying EDR headroom.

If you want to learn more about EDR and color technology on Apple platforms, here's a list of videos on related topics from the last few WWDCs.

The "Explore EDR on iOS" video from this year is especially relevant if you're working with EDR on iPad, either with or without Reference Mode.

And that brings us to the end of our talk.

I hope that you'll take advantage of Reference Mode and its APIs in your app.

Thanks for watching.