Explore enhancements to RoomPlan

♪ ♪ Alex: Hi. My name is Alex. I am from the Video Engineering team. Today, my colleague Antoine and I will be sharing with you what's new in RoomPlan. RoomPlan uses sophisticated machine learning algorithms powered by ARKit to detect walls, windows, doors, openings, and other room-defining objects. Our RoomCaptureView API allows you to integrate a scanning experience directly into your app. Once finished scanning, your app can present the resulting 3D model and export to a USDZ file. In this session, we will talk about what's new in RoomPlan. We will start with a new way to combine RoomPlan with ARKit by using a custom ARSession. Then, we will turn our attention to our MultiRoom support. We provide a new MultiRoom API to merge individual room scans into one larger structure. Next, we will show our new VoiceOver support for RoomCaptureView. Finally, we will go over some improvements in RoomPlan representations and enhancements to our export functions to enable new workflows.

Let's start with Custom ARSession support.

As mentioned earlier, RoomPlan is relying on information from ARKit to detect walls, windows, doors, openings, and other objects during scan. For this, RoomCaptureSession is running with a default ARSession. New in iOS 17, RoomPlan can use a custom ARSession with ARWorldTrackingConfiguration. This allows for some new ways to combine the use of RoomPlan and ARKit within the same workflows. Let's take a look at some examples. One way to use RoomPlan with a custom ARSession is by combining RoomPlan results with ARKit scene geometry and plane detection for more immersive interactions between virtual content and real-world geometry. Moreover, you could collect photographic representations of a space by using high-quality image capture from ARKit to create richer real estate listings with RoomPlan. And if you are using RoomPlan as part of an existing AR experience, you can combine results from RoomPlan without disrupting your existing ARAnchors. These are just a few use cases that a custom ARSession can provide. Let's take a look at some codes to see how you pass a custom ARSession to RoomPlan.

Here is the init and stop function in previous RoomPlan. And here is how you pass over a custom ARSession to init function. Any custom ARSession with ARWorldTrackingConfiguration will be honored inside RoomCaptureSession. Also, the stop function now includes a new option to determine whether you want to pause the underlying ARSession. You can set the Boolean to false in case you want your ARSession to continue running after RoomCaptureSession stops. In the next section, we will see how working with ARSession enables for new workflows, like merging multiple scans into one larger structure Let's turn our attention to MultiRoom support.

What is MultiRoom support? In previous RoomPlan, you could do a single scan and get a 3D model for a single room. Say you have done several scans of different rooms in a house, such as a dining room, kitchen, living room, hallway, and bedroom. If you want to merge them, you'll face some challenges. First, they are all in their own coordinate system, meaning the origin and orientation of the world coordinate is different for each room. Second, even if you stitch them manually, you'll end up having duplicate walls and potentially duplicate objects.

Let's tackle the different coordinate system first. What we want to achieve here is to have all scans taken in the same coordinate system. Two approaches are suggested here to scan multiple rooms. First, use continuous ARSession. Second, use ARSession relocalization. Let's look at how to use a continuous ARSession. In previous RoomPlan, when RoomCaptureSession stops, ARSession is paused. Each scan has a different coordinate system. Our new API now has a new argument in the stop function, which we can set pauseARSession to false. This allows ARSession to keep running for the next scan, even the one after that, until we finally decide to pause ARSession again.

Using this approach, we can ensure the same ARSession running across several scans. This will allow us to have one common world coordinate system for all our scans. Let's look at how this works in codes. Here is an example to run RoomCaptureSession with a continuous ARSession. We start the first scan with RoomCaptureSession.run. Then, here is the key part. With the new API in RoomCaptureSession.stop, we need to set pauseARSession to false. This will keep ARSession running for the next scan. After that, we use the same roomCaptureSession instance for the second scan. Then, we stop the second scan. Finally, we could get the first and second scan results in the same coordinate system. Another approach to take individual room scans within the same coordinate space is to use ARSession relocalization. This approach is most suitable if you're taking individual room scans at different times, such as coming back to same location on the next day or week. Let's take a look at how this works.

Again, this is a single scan to get the 3D model of a single room. Since we stop RoomCaptureSession and pause ARSession, for relocalization to work for future scans, we need to save ARWorldMap to disc. If you want to continue the previous scan when an ARSession is paused, you can now resume scanning by loading an ARWorldMap from the paused ARSession.

Using this ARWorldMap, you can relocalize against a previous scan's environment to ensure that a series of scans all share a common coordinate system. Let's look at the sample codes for the scanning workflow with relocalization. First, we run the first scan. Then, we stop the first scan with pausing ARSession. After that, for relocalization to work, we need to save ARWorldMap when ARSession is paused.

Before running the second scan, we need to restore our previous ARWorldMap. First, we load the ARWorldMap. And we assign the loaded ARWorldMap to ARWorldTrackingConfiguration. initialWorldMap. Then, we run ARSession to trigger relocalization. Once relocalization is completed, the previous ARSession is loaded. Current world coordinate is aligned with previous world coordinate. Then we can run the second scan. Last, we stop the second scan. With all these steps, the first and second scan results are in the same 3D world coordinate. We have seen two approaches to link multiple scans in the same 3D coordinate system. Next, we will take a look at how to merge them into one combined structure by using our new MultiRoom API.

For each scan, we can run RoomBuilder API to generate individual CapturedRoom. As shown previously, with continuous ARSession and ARSession relocalization, all the CapturedRooms are in the same 3D world space. Here is the output from RoomBuilder with three CapturedRooms. Now we provide a new merging API, StructureBuilder, to merge all of them into one large structure, CapturedStructure. Next, let's look at the sample codes for StructureBuilder API.

Here is how we use StructureBuilder API to merge multiple scans. First, we create a StructureBuilder instance with configuration option. Then, we create an Array to load multiple CapturedRooms, which we scanned earlier. After that, we call StructureBuilder API to get capturedStructure, which is the merged result. Finally, we can export the capturedStructure to a USDZ file. And here is the CapturedStructure definition. First, it has a property of rooms. It's an array of CapturedRoom instances. Then, it has properties for the merged walls, doors, windows, openings, and objects. Last, it has a function to export to a USDZ file.

Let's see MultiRoom in action. We provide you a sample.app, which allows you to merge multiple scans with StructureBuilder API and export to a USDZ file. Our USDZ file is ready to preview in both iOS and macOS. You can take this one step further by loading your USDZ file to a Digital Content Creation Tool such as Blender.

After we applied some beautification to the 3D model, the result can look even better.

Lastly, let's talk about some considerations to get the best MultiRoom experience with MultiRoom support. MultiRoom works best for single-floor residential houses, with a typical setup of one to four bedrooms, living room, kitchen, and dining room. For scanning and merging individual rooms, we recommend a maximal total area of 2,000 square feet or around 186 square meters. Moreover, having good lighting of 50 lux or higher is recommended to ensure RoomPlan can scan with a clear video stream and good AR tracking performance. Now, here is Antoine to tell us about even more enhancements for RoomPlan in iOS 17. Antoine: Thanks, Alex. Picking up from here, let's talk about accessibility. When we speak about rendering, most of us think about visual modality, but for low vision people, this modality is far from being the most useful one. This year, RoomPlan added an audio feedback when VoiceOver is enabled, allowing your phone to provide guidelines about scanning and to describe what it sees. VoiceOver: Move device to start. Point camera at bottom edge of wall. A fireplace. A wall. A window. Antoine: Now let's talk about the new information that RoomPlan can collect from a room and how it is rendered. RoomPlan makes it easy to scan a wide range of rooms. However, so far, it was limited to accurately represent a finite set of room situations. RoomPlan now supports an even larger variety of rooms, which includes slanted walls and curved walls, but also recessed kitchen elements such as dishwashers, ovens, or sinks. RoomPlan was also improved to detect configurations of objects of a given category. For example, there are a lot of types of sofas, from single seat to L-shaped, and on to simple squared sofas that RoomPlan will detect in the new version. When we introduced RoomPlan, we talked about the two elements that RoomPlan could scan: surfaces and objects. Now, we're adding new elements to describe areas inside a room. We call them sections. Walls can now be described as polygons, to handle non-uniform walls such as slanted walls or walls with a beam. So far, curved walls and windows were part of data-only API. Now, final result from RoomCaptureView can also render curved walls. Besides the surface categories, floor categories is another addition that can also be described as a polygon. Objects now have attributes to better describe the different configurations inside a category. Surfaces and objects now have a new parent variable. It contains the identifier of their parent. For example, the parent of a window is a wall, the parent of a chair can be a table, and the parent of a dishwasher can be a storage. Let's detail all those improvements with examples. Each section describes the different areas in your room or house. A section has a label among the following: livingRoom, bedroom, bathroom, kitchen, and diningRoom. It has a given position, at a given floor.

Non-uniform walls can be rendered as polygons using the polygonCorner variable. Floor is now represented as a rectangle during scanning and is beautified as a polygon when scanning ends. Parents of dishwashers, ovens, and sinks are now carved in rendering. When we introduced RoomPlan, categories were used to describe an object. However, this representation had limitations. Let's take the example of a chair. Several types of chairs can represent this category, for example a stool, a dining chair, or an office chair. They all have a different purpose. In order to have a better representation of objects, we're now adding attributes. In this example, using attributes, we have a more faithful understanding of what was scanned. In RoomPlan API, attributes are available through a polymorphic array of enums. However, enums are not the best way to understand attributes. In next section, we'll explore new ways to arrive at a more appealing representation. Attributes, together with other new information captured during scanning, can now be included in our exported results. There are two additions to the kind of data that we can now export: a file to find back metadata from a USDZ node and a structure to enrich exported USDZ with models. When exporting a Room as a mesh, we create a USDZ that contains a tree of nodes for surfaces and objects. This year, we added a section group containing section centers. However, when doing so, we're missing a substantial amount of information from our scan, for example dimensions of walls and objects, but also object attributes. A mapping file can now be created when exporting a room. It's an encoded dictionary of String to UUID that creates a bridge between a USDZ unique node name and a CapturedRoom element, uniquely identified by its identifier.

Let's see how it unfolds in RoomPlan API. In previous version of RoomPlan, a room could be exported in two ways: as a USDZ, through the export function, and as a JSON or Plist, encoding the CapturedRoom structure. New in iOS 17, specifying a metadata URL to map a room to a USDZ in the export function, you can relate the two exported information. This way, when rendering your scanned room, you can query additional information about a surface or an object. Along with the new mapping file, we are introducing Model Provider in order to replace objects, that used to be represented as boxes, by models that match the attributes that were scanned. What we want to do here is to associate an object to a 3D model, and more specifically, to its URL. This way, we'll be able to replace the bounding box that used to represent the object, by a more compelling and truthful rendering.

For that purpose, a new structure is available in RoomPlan: ModelProvider. ModelProvider maps categories and sets of attributes to Model URLs. From an object with categories and attributes, you can ask ModelProvider to give you the corresponding model URL. Now generalizing it to an entire room, which is a collection of sections, surfaces, and objects with categories and attributes, a ModelProvider instance specified in the export function can associate 3D Model URLs to each object of the room. Now that we have a structure to handle model association, let's see how we can work with Model Provider. There are many ways to populate ModelProvider. You can, for example, fetch models matching a set of attributes from a database, or annotate an existing catalog. Here, we'll walk you through a very simple example on creating your own small catalog of assets. We'll create and use a catalog in four steps. First, we'll parse categories and attributes that RoomPlan supports. Then, we'll associate a model for each category and attribute set. After that, we'll instantiate a ModelProvider. And finally, we'll use it to export our room.

Let's see how to discover attributes and create a model provider using some models. First, we iterate through all categories that RoomPlan supports. Then, we create a folder for each supported category. We can add models to each folder later. Then for each category, we request the attribute combinations that are supported by RoomPlan. For each set of attributes that is supported, we create a folder. In each of those folders, you can add a 3D model corresponding to the category or set of attributes. Now that the content of our catalog is ready, we need to create an index file for it. Here is a an example structure that will handle a catalog index. It contains an array of elements, each one of them containing a category or a set of attributes. Each element has a reference to the path of a corresponding model. Now we can save our index file as a plist and our catalog as a bundle. Each time we want to export a room with models or each time we want to use a model provider to relate an object to a model, we can now use this catalog bundle to generate a ModelProvider. All we have to do is to iterate through our catalog categories and attributes, find the corresponding model URL, associate the model URL to a category if there's no attribute, or associate the model URL to an attribute set. The last step is to call the export function, specifying the output URL, a model provider instance, and a .model option. Et voila! What we get is a USDZ with 3D models that correspond to your scan. To take it further, you can optionally take the USDZ into DCC tools like Blender to add lights and shadows for an even more realistic result that can be obtained in a matter of minutes. To help you create compelling results, we added a pre-populated catalog in our sample code. Back to you, Alex! Alex: Thanks, Antoine. Let's take a look at what we shared today. First, custom ARSession support enables new use cases such as capturing high-quality images and videos alongside a scan to improve real estate listings. Now, you can also scan multiple rooms and use the new StructureBuilder API to generate a merged 3D model of an entire house. To improve the scanning experience for low vision users, RoomPlan now supports VoiceOver when using RoomCaptureView. RoomPlan's new object attributes allow you to more accurately represent a scanned room. And finally, you can now use our new export API to assign 3D models from a custom catalog to corresponding scanned objects. That is RoomPlan in iOS 17. We can't wait to see what you make with it.