Design considerations for vision and motion

♪ Mellow instrumental hip-hop ♪ ♪ Manda Paul: Hi, my name is Manda. I'm on the Apple Vision Science team Herman Damveld: And I am Herman, and I'm a research scientist here at Apple. Creating experiences in three dimensions makes use of unique modalities such as depth and full field of view. We will introduce you to how a person may experience this new level of immersion through visual cues and motion information from 3D content, and where human limitations may interfere with comfort of people experiencing your apps. Manda: Our guidelines are to help you create experiences that are comfortable and enjoyable. We'll talk through visual depth cues, content parameters, eye effort, motion of virtual objects, head-locked content, motion in windows, and oscillating motion. To begin, it's important that your content provides correct visual depth cues to the viewer. This matters because well-designed 3D content enables the brain to perceive depth. First, a bit of science. We have multiple sensory systems with which we experience the real and virtual world. At the very high level, the visual system is responsible for taking in light from the content you create and interpreting it into what is perceived. That is to say, your design decisions have a significant impact on vision comfort for your viewers. Agreement between the depth intended by you and the depth perceived by the viewer is necessary for vision comfort. As an image moves in depth, the brain enlists the eye muscles to converge the line of sight and fixate the eyes at the image depth. This enables a single image to be perceived. To converge the eyes correctly, the brain needs correct visual depth cues. Vision comfort might be affected when depth cues are missing, conflicting, or misleading. The viewer might experience discomforts such as double vision or eye fatigue. Let's see how we can use image cues to maintain vision comfort. When working with scenes that are very sparse, as in this image of a citrus, many visual cues are missing that might have conveyed depth information. The brain might be unable to reconcile what depth to converge the eyes to, hence causing discomfort. Seeing the color of the citrus enables the familiar size cue. If the slice looks green, the brain uses the prior knowledge of the size of a lime to inform what depth this slice might be floating at and converges the eyes at that correct depth. Using blur is another way to convey depth information. Relative size can also be used to add depth cues to a sparse image. The larger of the two slices from the lime is perceived to be closer to the viewer. Gentle motion can also provide a reference for what depth the line of sight should point to. My colleague Herman, will elaborate on the nuances of using motion in your content. For now, let's add just a few more cues to direct the eyes. An added background, light and shadow, occlusion, and texture density are some more methods to add in the missing visual depth cues. Conflicting cues are also troublesome. As mentioned, the bottom slice appears closer since it looks larger. We expect it to be in front and occlude the smaller looking slice. However here, relative size cue conflicts with occlusion cue, telling the brain the opposite of what we expect. The conflict between the depth cues might lead to vision discomfort. Conflicting cues are not unique to sparse scenes. The more conflict there is, the more challenging for vision comfort. So, ensure all of the depth cues from your content give the brain correct information so to align the viewer's eyes to the depth you intended. One more thought about depth is correcting the visual cues that are misleading, such as repeating patterns. Here, both eyes are pointed to the same object in the repeating pattern. That's great! However here, each eye locks onto a different object from the repeating pattern and when the brain tries to combine the images from each eye into one, it's perceived at a different depth than the repeating pattern is intended to be. To mitigate this conflict and to avoid double vision, use smaller portions of the pattern or break up the pattern with another design. A final note about depth perception. The 3D content, along with your visual depth cues will automatically be rendered on each display per eye appropriately to enable someone to see the depth you intended. If you choose to manually render stereo content for stereo video, for example, it is extremely important to not only have correct depth cues but also have correct disparity for each eye so to avoid extremely conflicting visual cues which could lead to severe vision discomfort. All right! Let's move on to content parameters such as size and contrast. Visually comfortable content have parameters suited for each specific visual experience. Choose parameters that reduce visual effort. Here are some scenarios. Content requiring fixating the eyes for an extended time, such as for reading, is most visually comfortable when placed farther than arm's-length distance. In addition, enable a person to adjust content depth so they may choose a depth that's most comfortable for them in the experience. Reserve content closest to the viewer for experiences demanding only momentary visual effort or for experiences inviting direct interaction which occurs when someone's hand actually intersects an item in 3D space. Otherwise, you may use visual cues, such as transparency or blur, to redirect eyes to a farther depth to increase visual comfort.

Similar to depth, content size and contrast should match the particular visual experience you are designing to allow for vision comfort. For example, utilize higher contrast for reading text and lower contrast, transparency, or blur to redirect visual attention elsewhere. Choose text size and window size and depth for extended reading such that text is legible with comfortable eye scanning, without the need to rotate the head or body. One more parameter to be mindful of is content brightness. When a large portion of the field of view is dark as in this image, slow down the transition to a bright scene to give time to the eye for brightness adaptation and ensure vision comfort. One last thing to be mindful of for vision comfort is the effort required from the eyes. For a more comfortable visual experience, minimize eye effort demanded from your viewers. Here is how. When choosing content location, keep in mind it's most comfortable for people to turn their eyes downwards or left and right. So choose those positions for your content when possible. Upward and diagonal eye rotation requires most eye-muscle effort. Holding the eyes at those positions can be fatiguing. Content requiring extended reading or elongated target fixations should be placed towards the center and slightly below line of sight to provide a comfortable visual experience. If extreme eye-rotation angles are necessary for your experience, design for brief interactions, or else move them to the center of field of view to reduce eye-rotation effort. It is also important to consider that a person might prefer to take a break to rest their eyes. To enable this, design for natural break points in your experience to allow for the person's eyes to rest. That's it for vision comfort! Next, my colleague Herman will walk you through how to design apps for motion comfort. Herman: Thanks, Manda! Next, we are going to have a look at how the motion information from content in your immersive apps can affect a person's experience and where human limitations may interfere with comfort. Let's start with digging a little further into the science. We, human beings, use multiple sensory systems to experience the real and virtual world. We already saw that the visual system is responsible for taking in light and interpreting what is perceived. Part of that perception is your own motion. The vestibular system in your inner ear also measures your own movements. Normally, the visual motion information and the motion that you perceive with your vestibular system are always in agreement. But when the visual motion information is missing or conflicting with the vestibular information, you can experience motion discomfort, such as dizziness or an upset stomach. Well-designed content enables the brain to perceive the world as stationary. This provides a sense of stability and is necessary for motion comfort. Let's go over some scenarios with motion and see how we can maintain motion comfort. The first scenario deals with motion of virtual objects. When one or more virtual objects, like these blue polyhedrons here, cover a large part of the field of view and move, the viewer's brain might interpret the visual motion of the objects as if the viewer themself is moving. This upsets the sense of stability and can cause motion discomfort. As a designer, you can mitigate this by making objects semitransparent when they move, so the passthrough content is clearly visible during the motion. Whenever possible, avoid using head-locked content. That is content that is anchored to the user's head. Head-locked content will be, always, in the same position in the field of view. If head-locked views are necessary, use a smaller window size placed near the center of the line of sight and farther away from the viewer. Whenever possible, use world-locked views to prevent content from feeling stuck. Another alternative is to use a lazy-follow animation. This moves content slowly towards a destination over time. You also want to be mindful of the motion of the content that will be visible in a virtual window. When the content in a window, such as this virtual landscape, moves...

...the viewer's brain might interpret the visual motion as if the viewer themself is moving.

To provide optimal motion comfort, you want to be mindful about camera motions. For starters, keep the horizon of the content in the window aligned with the real horizon. The point where all the pixels appear to be coming from is called the focus of expansion. That's where you're going, and people have a natural tendency to look at that point. You want to move the camera in such a way that the motion of the focus of expansion is slow and predictable, and you want to keep the focus of expansion within the field of view. That means that you want to avoid fast turns or pure rotational motion. The focus of expansion shifts out of the field of view in those cases. As an alternative to fast rotations, you can use instantaneous directional changes during a quick fade-out. To optimize for motion comfort, you also want to avoid that large objects shoot by at close range, as seen here on the right. Best keep objects small and at a larger distance, as on the left. And if possible, use plain textures with low luminance contrast, which makes the motion less perceivable by your eyes. The last type of motion scenario to be mindful of deals with sustained oscillations of objects and animations. To ensure a comfortable experience, try to avoid oscillations in general, and in particular those with frequencies around 0.2 Hz. That is one oscillation per 5 seconds. And if you really cannot avoid showing oscillating motions, try to keep the amplitude of the motion low and make the content semitransparent. This makes the motion much less perceivable by your eyes, as seen on the left. And it's always a good idea to provide an oscillation-free alternative through the Reduce Motion accessibility setting. To wrap it up, here's a summary of the things to keep in mind when designing for motion comfort. Try to provide a sense of stability, avoid head-locked content, be gentle with camera motion, and avoid oscillations if you can. To learn more about principles of spatial design and see many of our vision and motion guidelines in action, I can recommend visiting the sessions listed here. Manda: That's it for us. We can't wait to see what amazing experiences you will create! Herman: Thank you for watching! ♪