Explore game input in visionOS

Hi, I'm Charlyn, from the Technology Evangelism team. Today I'm talking about what goes into great game input in visionOS.

You have a lot of flexibility when it comes to the type of games you can make on visionOS and the frameworks you can use to develop those games.

On Apple Vision Pro, people use their eyes and their hands to navigate apps and games on the system. There is also support for external physical input devices. Game input that's designed around gestures is the easiest way to let players jump right in your game, without the need for any extra equipment.

I'll walk you through the different types of gestures you can choose from.

Then, I'll describe some of the physical input devices that are supported by visionOS, and some things you should keep in mind if you've decided to add support for them.

Input designed around visionOS gestures can be either direct or indirect.

Direct input means you're reaching out and handling a virtual object directly. Just keep in mind, that with the infinite canvas of visionOS, players might need big arm movements to reach all the objects in your game. Indirect input means you're operating on those objects at a distance, by looking at them and tapping to select them. So your hand can be held closer to your body. Indirect input lets players make small hand movements that are amplified into big actions. So if they're reaching far across the viewing area, or making big repetitive actions, indirect input can make longer play sessions a lot more comfortable. For example, this is Loona, a puzzle game on visionOS. You can take hold of each piece from a distance, with indirect input. Just select a piece by looking at it, and move the piece with your fingers. You can place it directly into the puzzle, or you can double tap your fingers to snap it into place.

You use gestures to rotate the puzzle or move it around, so you can see every angle. That way, you can comfortably play either standing or sitting. Indirect input can make for a great sit-back experience when you're playing a game. But sometimes direct input makes sense, maybe because your game is meant to be high energy, and besides, direct interaction with virtual objects can be a lot of fun.

This is Super Fruit Ninja, where you slice flying fruit along with the menu buttons.

The game is designed around direct input.

If this type of input fits your game, make sure to include plenty of feedback, so every contact with an object feels satisfying for the player.

You can use a combination of visual feedback, like the lightning streak behind your hand and the flying pulp. And you can add sounds, like squishing and splashing.

Even without haptics, the player knows right when they've made contact, whether it's with a UI button or a flying pineapple! It's best to support both direct and indirect gestures in your game, and there's a few different ways you can implement them.

Your game can respond to the built-in system gestures, or, you can combine two or more system gestures together which gives you even more fine-grained control over what input your game responds to.

If you need to, you can use the hand tracking in ARKit to create your own custom gestures. Before you decide, keep in mind there are some real advantages to using the built-in system gestures. For one thing, players already know exactly how to use the system gestures, because they're already using them to navigate all the other apps and games on the platform.

You won't need to teach them how to interact with your game.

And there's no extra equipment needed. Your game works right out of the box! Players can pick it up and play it anytime. Plus, no matter where your game lives on visionOS, system gestures are supported. They work in every space, from the Shared Space to full immersion. There are a variety of system gestures available to use.

The simplest one is to look at an object, and tap two fingers of one hand together.

If your game uses a single tap on a 2D screen for input, just substitute the tap gesture in visionOS.

You can also respond to a double tap. There's a pinch and hold gesture, and a pinch and drag, both of those operate on objects directly or indirectly. Using two hands, the player can zoom in and out, or rotate an object. This visionOS game, What the Golf, is a great example of using system gestures to indirectly control objects in the game. I look at the ball and use a pinch and hold gesture to activate it.

I can now move my hand to control the ball's direction and the force it will have when I release it.

Because my hand has taken over the input, I'm free to look around the level to see where I'm aiming.

Once I release the pinch, the ball flies, and I get a hole in one! Responding to system gestures is really easy.

Any entities you include within your RealityView that you want to be tappable need to have an InputTargetComponent and a CollisionComponent. Then you'd attach a gesture to whichever RealityView contains your entities, and then indicate that any entity within that view is tappable.

Once the gesture is detected, you can respond to it in the gesture's handlers. To learn more about responding to system gestures, watch last year's video, "Develop your first immersive app," and check out the documentation on interactivity at developer.apple.com.

And if your game is developed in Unity, you'll use the Unity Input System to detect and respond to gestures. To receive tap events, add an input collider to your game object.

The tap data is made available to you as a World Touch Event.

Check out "Create immersive Unity apps" for more details.

System gestures give you a ton of choices right out of the box. If you need even more flexibility, you can combine the system gestures together to make something unique. You can pair two or more system gestures and detect when they happen simultaneously, or one after the other, sequentially. You can also combine gestures with keyboard modifiers. And you can use SpatialEventGesture for low-level access to gesture events like Tap Begins, Moves, or Ends. This is really useful if your players are using two hands to handle two different objects simultaneously.

Use SpatialEventGesture to keep track of the individual targets for each action.

In this example, you can select the satellite, move it, resize it, and rotate it simultaneously. The system recognizes each gesture as it happens, even if the previous gesture hasn't ended yet.

Here's how you can do that in code.

While someone is using the Drag gesture to move an object, adding simultaneously(with:) indicates that your app should listen for both the Magnify and Rotate gestures if they happen as well.

This means the transitions between multiple gestures will be seamless. To learn more about combining gestures, check out last year's video, "Take SwiftUI to the next dimension," and the documentation.

Because they're familiar to players, system and combination gestures should be your go-to for game input on visionOS.

But if you do need to round out your arsenal of input, you can define your own custom gestures.

Any custom gestures you create for your game need to be easy for players to learn, and easy for them to remember.

Make sure the gestures are a great fit for the action in your game so they'll be intuitive. And when you design your custom gestures, make sure the player knows they're doing it correctly by adding in the right amount of feedback, with both visual effects and sounds. To define your own gestures, you've got access to full hand skeleton tracking through ARKit.

You can combine system with custom gestures within the same game. This is Blackbox, and it's a great example of mixing them together in creative ways.

During their onboarding, Blackbox teaches the player to make this custom gesture that creates a new bubble in the environment.

The gesture is simple and intuitive, which means it's easy to learn and remember.

And it's important, because it's the main way you enter and exit a level in the game.

But don't overdo custom gestures. Use them sparingly. Blackbox uses the system gestures whenever they fit. For example, I can look at the bubbles and tap my fingers to pop them.

Blackbox lives in a Full Space on visionOS, which is important. If your game lives in the Shared Space, alongside other visionOS apps at the same time, then ARKit and hand tracking isn't available to you.

Now, in a Full Space, your game is the only app that appears, whether that's in mixed, progressive, or full space. Your player might see their surroundings, like they can in Blackbox. Or your game might replace the surroundings with a new environment.

For creating custom gestures, ARKit and full hand skeleton tracking lives in Full Space.

Here's how you'd create a custom gesture in the shape of a circle.

Check all of the joints you'll require for hand tracking, and get their positions in world coordinates. Then, for this circle-shaped gesture, check the distance between the tips of both index fingers and both thumbs. If it's smaller than 4 centimeters, the gesture is recognized and your game can respond to it.

If you develop your game in Unity, you can use Unity's hand tracking package to access information about the player's joints. Check out these videos for more information about bringing a Unity game to visionOS.

No matter what type of gesture you use, indirect or direct, system gestures or custom, you'll want to make sure you build in an alternative for players with different abilities.

For example, Synth Riders has input designed around using both of your hands and arms to contact objects. But imagine a player that only has the ability to play with one arm.

In this game, there's an option in the "Comfort and Accessibility" settings where you can choose to play the game in one-handed mode and to choose if it's the right or the left. Now I can play the game with only my right hand.

It's really important to think about these types of accessibility use cases and build in some input fallbacks for your players.

If you're using RealityKit, you have access to the native accessibility frameworks. Check out the video "Create accessible spatial experiences," for more information on designing for different abilities.

If you're a Unity developer, there's an Apple Accessibility plug-in that can get you access to all of the accessibility features. Check out "Plug-in and play" for more details.

With the flexibility of system and custom gestures, you've already got a lot of options when it comes to designing input for your game. But in some cases, you may want to add support for one of the physical input devices available in visionOS.

visionOS supports all of the same game controllers that iOS and macOS support, including the Xbox Series X, and the Sony Playstation DualSense controllers. Supporting game controllers can be the right choice if you need more than two inputs at a time. Or maybe you're bringing a game from another platform, and you've already based your input around controllers.

visionOS also supports Bluetooth keyboards and trackpads, in case that input is the right choice for your game.

It's simple to respond to game input from physical controllers, keyboards and trackpads that are connected with Bluetooth to visionOS. All the same devices work across iOS, iPadOS, macOS, and visionOS. And it's the same game controller framework across all these platforms, too! So if you're bringing your game across multiple types of devices, the same code will work across them all. And physical controllers work across every type of experience in visionOS, from a windowed game to fully immersive space.

The Arcade game Wylde Flowers has great controller support. If you have a game controller paired and connected with your Vision Pro, the game input works automatically with that controller. This is really important to do! Detecting that a controller is connected, and switching your input over to work with that controller, is much better than making the player search for a switch in your game's Settings menu. Another thing Wylde Flowers does well is to make sure you can do everything in the game with the controller. Expecting players to use a system gesture to access part of your UI would be inconvenient and pretty confusing. Design your input so that everything in the game, even the UI, is accessible with the game controller.

It's easy to detect if the player has connected a game controller. Just set up an observer for GCControllerDidConnect, which will alert you to any game controllers that are connected to the system. Be sure to also listen for GCControllerDidDisconnect events, so you can fall back on other forms of input (like system gestures) if there's no game controller present. Once you receive the alert that a game controller is connected, you can poll the state of specific buttons on the controller. Or if you prefer, you can set up change handlers so you get notified when input state updates. Input from Keyboards and Trackpads has the same flow. Listen for their connection, and then either poll them for input, or set up change handlers.

Here's the code for detecting and responding to a game controller. Set up your observer and wait to be notified that a game controller has connected. If the controller was already connected before your game launched, you'll still get a notification here for every device that's connected to visionOS. Once you're notified, either set up your input change handlers, or poll the device for the input you're expecting. And there’s one last step unique to visionOS 2. Add the game controller interaction modifier to the RealityView that contains your game objects. This indicates that game controller input directed to your RealityView will be handled by your game. And it's important to give the player feedback on actions, just as it was with gestures. Use the CoreHaptics framework to trigger the rumble in the controller to match the on-screen action. And don't forget to add sound effects and give players immediate visual feedback when their action is successful. If you're new to the game controller framework, there are some great resources to get you started. And if your game relies on the Unity engine, you can use the game controller plug-in for Unity to add support for game controller input. Check out these videos for more details. Here's a quick look at which spaces support different types of input. Most types of input, including system gestures, traditional game controllers and keyboards, are supported in all types of spaces, from a window or a volume in the Shared Space, to a Full Space experience. If you want to respond to input from custom gestures, you'll need to design your game for a Full Space.

Just to recap, you've got a lot of options in your game to design input that feels just right to players. The built-in system gestures in visionOS are simple and natural for players to use. And if you need more flexibility, you can consider sprinkling in a custom gesture or two. For certain types of games, physical controllers are supported and can expand your input options. And one last thing: if you have an existing game on iOS or iPadOS, find out more about how you can transform it to feel more immersive in visionOS with "Bring your iOS or iPadOS game to visionOS." Thanks for watching!