Explore the USD ecosystem

♪ Mellow instrumental hip-hop ♪ ♪ Hi, I'm Dhruv Govil, a 3D Software Engineer.

And in this session, I'll be walking you through what's new and exciting in our 3D Content Technologies ecosystem on our platforms.

3D is an amazing form of media, and it's even more important with the introduction of xrOS, where the third dimension will allow for even more immersive experiences.

It's become far more accessible for everyone to get started with creating amazing visuals on Apple computers they have at home or even the ones in your hands like an iPad.

To enable this wave of creative content, we've been building on the foundation based on the key technologies that power the creative industry worldwide.

The first of these technologies is Universal Scene Description, or USD for short.

It allows for a standard form of representing 3D content to share between applications.

And we're now also introducing support for MaterialX on xrOS to portably represent the visual appearance of objects.

USD is an open source project, created by Pixar, recently renamed to OpenUSD.

It is production proven and scales from creators making single assets to large studios working on AAA games and films.

USD allows for expressing complex and flexible relationships between asset data via composition.

Apple was an early adopter of USD, adding it to our platforms in 2017 and growing support since.

Today, USD is at the heart of 3D Content on xrOS.

MaterialX is also an open source project, created at Industrial Light & Magic for their work on Star Wars, now developed in conjunction with others like Autodesk and adopted by the Academy Software Foundation.

It allows artists to combine shader logic into a node graph within their 3D applications, all without needing to know how to code.

This graph can also be embedded inside USD, so it travels with your scene data.

We're supporting MaterialX first on xrOS with RealityKit, Apple's real-time 3D rendering framework for building immersive spatial experiences.

We have four areas to cover as we discover how these 3D content technologies can be used.

To start, we'll go over how our applications are enabling new uses for 3D content.

Next, we dive deeper into how we support MaterialX as well as color management in RealityKit.

Finally, we'll return to USD and go over the changes that you should be aware of since last year.

Let's begin by looking at the new ways you can use 3D in our applications, before we get into the technologies that power them.

Building on the success of Quick Look for USDZ files on iOS, we're now bringing Quick Look to xrOS.

It's so simple to just open up a USDZ file and see it beautifully and spatially represented, walking around it and connecting it to your surroundings.

We have a great talk as well on how to easily author content that will look great on this new platform.

Last year, we introduced Freeform, a powerful and easy-to-use brainstorming app.

Freeform now gives you the ability to embed USDZ content, so you can now collaborate on your 3D assets just like you would for other content types, across all supported platforms like macOS, iOS, iPadOS, and now xrOS.

Safari also introduces new support for 3D Content with the Model element.

The Model element is available on all Apple platforms with Safari and can be enabled under the Developer menu on macOS or Settings on other platforms.

Just as easy to use as the audio, video, and image elements, the Model element lets you embed a USDZ file into your web page with an interactive view, with the ability to rotate around objects.

This allows creation of more interactive websites, providing your users incredible experiences no matter which of the supported devices they're on.

We're working with the W3C Immersive Web Working Group on standardization.

We also have Reality Composer Pro, a new macOS application joining our suite of tools for creators and developers.

Using USD's composition features, it lets multiple people work in parallel, enabling each person to tackle a different aspect of the scene and see them combined together as each asset updates.

Assets can be authored to USD in your own content creation applications, and Reality Composer Pro lets you prepare them for use in xrOS apps and experiences.

Check out our Reality Composer Pro sessions to learn more.

As important as the 3D models you make is the shaders and materials that give them the individual visual aesthetic.

MaterialX gives you the power to have bespoke visuals that can be transported inside your USD scenes into RealityKit applications on xrOS.

Reality Composer Pro introduces a Shader Graph that authors MaterialX nodes embedded inside of USD files.

This enables the creation of shaders using an interactive node graph to compose the shader logic.

MaterialX shader graphs are how creators will be constructing custom shaders for xrOS content.

In addition to many of the standard MaterialX nodes, Reality Composer Pro also has a few custom MaterialX nodes that enable a range of xrOS-specific platform features.

Some of these shader nodes are: RealityKit PBR, our physically based rendering shader which enables realistic-looking 3D content; RealityKit Unlit, an unlit shader that lets you do more stylized shading; Geometry Modifiers that allow us to modify surface deformations; as well as several custom utility nodes.

We have a great developer session on how to use the Shader Graph to get you up and running with creating custom looks for your assets.

Support for MaterialX with USD grows across established third-party 3D applications, like SideFX's Houdini and LookdevX in Autodesk's Maya, shown here.

Reality Composer Pro then lets you preview your shaders to see how they'll look before deploying to your device.

As I mentioned, MaterialX is an open source project which enables creators and developers to make use of it in their own workflows and applications directly.

The MaterialX project includes support for generating shader code from your MaterialX node graphs.

Apple has added support to the project for creating Metal shader code to make the most use of our powerful GPUs.

This is now available in the MaterialX 1.38.7 release.

For developers who use USD, this also enables the future use of MaterialX within USD on macOS.

Having confidence in how your shaders behave is important, but so is having confidence in what your colors look like.

So we've expanded our handling of color space management for RealityKit, so you can make accurate-looking 3D content capable of using our wide gamut color range.

But first, a quick recap of why color spaces are important.

Color spaces are how graphics programs understand how to represent colors from digital values, including the range of colors available.

This allows multiple applications to reliably display and edit the same colors.

Apple displays primarily use Display P3 as their color space, whereas many other platforms may use the more commonplace sRGB.

DisplayP3 marries the wider primary range of digital cinema with the sRGB gamma curves used by computer displays.

In fact, it's capable of displaying up to 25 percent more colors than traditional sRGB, shown here as the black line within the greater Display P3 color space.

This allows for representing more of the colors we see in the real world, such as the vibrant feathers of a parrot, or the deep colors of your favorite clothes.

Most color and image workflows tend to use sRGB, which represents a standard range of colors used by many monitors for decades.

These are still capable of creating good-looking content, but can't take full advantage of the beautiful, wider-range displays that Apple products ship with.

If you instead make use of the Display P3 color profile, you're now able to use the wider color range that RealityKit can provide on these displays.

Textures authored with Display P3 in mind, and appropriately tagged, are able to express a much larger range of colors, with deeper and more saturated tones.

This effect may be subtle, but allows for the creation of more vibrant and authentic-looking assets.

Notice the richer reds and the lush greens.

Most importantly, color space tagging means your colors are accurate to what you saw when you made your content, avoiding accidental color shifts.

MaterialX and Color Management are great new additions to our USD ecosystem.

We also have more to share about what's new in USD for our platforms and the greater ecosystem.

All our first party applications, such as Motion and Quick Look, now benefit from an updated and more efficient USD version.

USD supports a wide range of object types, also known as schemas.

This update also allows the Storm renderer in Preview on macOS to support rendering even more USD schemas and features.

We also continue to update our documentation to highlight features and schemas that are newly supported on our renderers and which platforms support them.

Additionally, we make note of other changes in behaviors, so we recommend that you periodically refresh yourself on the documentation linked in the video description.

USD also allows for custom schema types.

And this year, RealityKit is introducing new Component schemas for its Entity Component System, or ECS for short, on xrOS.

RealityKit's ECS splits the systems that process data from the Component data itself, allowing it to live alongside your 3D content.

Thanks to these custom schemas, you can now use RealityKitComponent for built-in Components, and RealityKitCustomComponent for your own Swift custom components.

Your Swift components structs and dictionaries can be represented by the equivalent RealityKit USD schemas.

RealityKit also builds upon USD's spatialAudio with RealityKit's Audio File, Audio File Group, and MixGroup to create even more immersive audio.

Let's take a look at a USD file that represents some custom component data, as a USD prim, which is what USD calls objects in its hierarchy.

This allows your custom component data to live alongside your other prims such as geometry.

Since this is all in USD, you can use any app that lets you author USD prims directly to create them alongside the rest of your scene.

This aligns with your corresponding Swift Component that can then be used to read these values from your USD components, such as here where different objects in your app may have specific Engagement Points associated with them.

Please check out our talks on building applications with Reality Composer Pro to learn more.

We continue to contribute to growing these technologies in the industry, as a growing ecosystem makes content creation easier for everyone.

So we've worked with Pixar and the community to list many of the software packages that now support USD on the OpenUSD webpage.

This allows creators to see how easily they can create USD-based content with their own existing workflows on a Mac.

We've also been working on making it easy to build USD for our portable platforms like iOS, so that as developers, you, too, can integrate USD into your own applications.

This enables you to create applications that can author immersive USD content.

Additionally, we're collaborating to reduce dependencies in the USD library so that it's easier to get up and running with USD across all platforms, with minimal work.

USD includes a technology called Hydra, an abstraction framework for renderers, and Storm, a real-time renderer that makes use of Hydra.

Apple has worked with Pixar to add Metal support to Storm, making use of our modern graphics API that enables developers to create high-performance, GPU-based applications on our platforms.

Let's take a look at one scenario where this performance helps.

The Animal Logic ALab scene is representative of many feature film-level assets.

When set to full resolution, this scene can take over 26GB of graphics memory, previously requiring desktop workstation GPUs.

Now, with Metal in Hydra Storm, and unified memory on Apple silicon, a MacBook Pro lets you work on the go, even on demanding scenes like this, while retaining interactive performance.

This high-performance rendering in Storm also enables Blackmagic Design to add fast viewport rendering of USD to Fusion in DaVinci Resolve.

Building on the collaboration from previous years, we continue to work with Autodesk on their open source Maya USD plugin.

We've made several contributions to the project, including enhancements to the export of geometry and materials for your assets.

We also have improvements to animation import.

All of this helps you have more seamless workflows to create spatial content for xrOS.

Some of these features may be released in later releases of Maya USD, so we recommend keeping your plug-ins up to date.

We've also been working with the Blender Foundation, many individuals and partners like AMD, NVIDIA, and Unity, to deliver updates to Blender, an open-source 3D application.

This collaboration has enabled significantly improved USD import and export in Blender 3.5.

This includes USDZ support for the first time, allowing you to go directly to Quick Look.

Apple also worked with the Blender Foundation to introduce Metal support for their Eevee and Cycles renderer.

Now with Blender 3.5, you can run Blender as a fully native Metal application, speeding up your UI, viewport, and final render.

Final renders can make use of your GPU to complete in up to half the time when compared to CPU based renders, and the viewport can now render up to 4 times faster than OpenGL in certain scenes.

That wraps up this overview of what's new in the USD Ecosystem.

xrOS adds a whole new, exciting dimension to apps and experiences, built on a foundation of technologies like USD.

I recommend continuing by checking out these related talks about Quick Look and Reality Composer Pro.

By collaborating on these open standards, we're building a strong ecosystem to make it faster and easier to create content for xrOS and all our platforms, and we can't wait to see what you create.

♪