Build custom experiences with Group Activities

♪ Bass music playing ♪ ♪ Willem Mattelaer: Hi, my name is Willem and I'm an engineer working on Group Activities. I'll be joined by Angus and Adam to talk about how you can create custom experiences with Group Activities. To start, I'll introduce the app we're going to be working on in this session and explain why it's a great candidate to add Group Activities support to. Next, we'll go over the activity creation and session management steps and explain the differences compared to making a media experience. Finally, we'll mention some ways you could polish the experience your users have while using your app. Group Activities allows you to build shared experiences across devices with SharePlay. Although the focus is on creating media experiences, that shouldn't prevent you from letting your creativity go wild and see how your app could be experienced across multiple devices. We will be building on some of the concepts that were introduced in the "Coordinate media experiences with Group Activities" session. I highly recommend checking that one out.

In this session, we'll be working on an app called DrawTogether that allows you to -- you guessed it -- draw together while in FaceTime. It's a pretty simple app where almost the entire screen is the canvas and everyone gets a random color they can use to draw with. But despite being simple, it's still incredibly fun to draw with my friends and admire the great skill of some of them or laugh at their poor drawings. I definitely fall into the second category myself. Let me show you a quick demo of what we're going to work towards.

Hey guys, what's up? Adam, I saw you had a question for me. Adam Syed: Yeah. So Angus and I were talking, and he mentioned that you're a really good artist -- like a grade A Picasso. So you got to show us, man! Angus Burton: I've seen some of Willem's drawings, and they're impressive. Willem: I think, Angus, you're exaggerating just a tiny bit. But I'm happy to show you what I can do. Let's go to the DrawTogether app. Adam: OK. Can we do, like, a pastoral landscape? Willem: Sure, uh... let me start by drawing a house, maybe. Adam: OK. Angus: Willem, this house is looking pretty basic, man. Maybe I'll help you out with a sun. Adam: I'll try a tree. Willem: That tree is way too tall. Adam: No, the sun's too low, man. Willem: Let me add some grass. Adam: I'll draw some more trees. Willem: OK. Um, I think I'm going to call it. We are not artists and we should stop what we're doing right now. Let's -- I'll talk to you guys later, OK? Adam: OK Angus: Bye. Willem: As you saw, one of the core experiences of the Group Activities API is allowing you to do things together while being physically apart. The real-time interactions that you can unlock and the instant reaction your users will get, since they can see and hear everyone else, can lead to some truly magical moments. This is something you should keep in mind when considering how to integrate Group Activities in your app. There are two steps to adopt Group Activities: activity creation and session management. We've covered this in more detail in the "Coordinate media experiences" session. In this session, let's take a look at how these steps change when building a custom experience, starting with activity creation. There are two parts to create a Group Activity. First you configure your activity, and that's followed by activating the activity. Compared to a media Group Activity, only the configure part is different for a custom activity. When configuring the activity, you have to think about the specific experience that you want to share between all the participants. The activity should contain all the information that remains constant throughout that experience. If you've watched the "Coordinate media experiences with Group Activities" session, this should look familiar. We've defined a DrawTogether struct that conforms to the GroupActivity protocol, and we've implemented the metadata property where we construct the associated metadata with a title. Now to make this a custom activity, we'll just need to set the right type on the metadata. By setting that to generic, we configure this activity to be a custom activity. And that's all you need to do when configuring a custom activity compared to a media one. Now let's jump right into Xcode and start creating our custom Group Activity in the DrawTogether app. First, let me quickly walk you through the code that we start with. DrawTogether is a SwiftUI app using the SwiftUI app lifecycle. ContentView is the main view of the app. At the top of the view, there is an indicator that shows the color that will be used when drawing. Below that is the CanvasView. This takes a canvas and will be responsible for drawing all the strokes in the canvas and updating the canvas based on any user input. Finally, at the bottom there is a ControlBar, which contains a couple of controls that could be useful while drawing. Currently we have a single button there to clear the canvas and start from scratch. The canvas itself consists of an array of strokes where each stroke has a color, an identifier, and a list of points. The canvas also has an activeStroke -- which represents a stroke that the local user is currently drawing -- and the stroke color that will be used by the user. Let's now start with configuring our activity. Before we do that, I'll have to add the Group Activities entitlement. I do that by going to the project settings, and in the Signing & Capabilities tab, I'll add a new capability. I'll search for Group Activities and select it. Now that we have the entitlement, let's finally configure our activity. I'll add a new file by going to File > New > File... and selecting Swift File. I'll call it "DrawTogether"... ...and click Create. First, I'll import our framework. Next, I'll define a new struct called "DrawTogether" that conforms to the GroupActivity protocol. The GroupActivity protocol has two properties that should be implemented: activityIdentifier and metadata. For the activityIdentifier, I will rely on the default implementation. The metadata property is, however, still necessary. So let's add it.

In this computed property, I create a GroupActivityMetadata object, and I set the title. I also set the type to be generic. This is crucial for a custom activity. Finally, I return the metadata object.

Now that we've configured the activity, we still need to activate it at the appropriate moment. I'll add a new button to activate it -- and what better place for that then our control bar? I'll add the button at the start of our HStack. For the label of the button, we're using an SF Symbol. And in the action closure, we'll create a new instance of our custom Group Activity and call activate on it. And that's all that is necessary to activate our activity. I just showed how to configure a custom Group Activity and how to activate it. These are the two parts necessary for the activity creation step. Now, I'll hand it off to Angus who will tell you about the session management step. Angus: Thanks, Willem. Next, we are going to talk about how you can send and receive custom data in your application using Group Activities. This is at the heart of creating unique SharePlay experiences with Group Activities. From the previous session titled, "Coordinate media experiences with Group Activities," you should be familiar with the three steps of receiving a session, preparing for playback, and joining a session. Instead of playback synchronization, we need to configure a session for our custom experience. But before we jump into that, let's add the code to receive the Group Session and join in. Let's go back to Xcode, and the first thing we'll do is to navigate to ContentView, and import GroupActivities.

Next, we will create an async task to receive our GroupSession. Now that we have a GroupSession, we need a place to store it. Let's store it on our Canvas object using a new method that we'll call "configureGroupSession". Now, let's navigate over to Canvas and implement the configureGroupSession method.

We'll start by going to the top and making sure to import GroupActivities. Let's go to the bottom of our file and implement our new method next. Here we assigned the groupSession object we receive to a new property on our class. Also note that we reset the canvas before setting the groupSession property. Last but not least, let's add the code to join the groupSession. At this point, we should be able to build and compile our project. Let's test that now by going to Product > Build.

Great. Now that we've set up the code to receive and join our Group Session, let's look at how to configure the session for sending and receiving custom data in our application. For configuring the session, we'll be using GroupSessionMessenger, which provides a simple API to send and receive raw data or structured messages to and from participants within the Group Session. Let's look at how to use GroupSessionMessenger next. To start, we will create a GroupSessionMessenger from our groupSession. The first step in using GroupSessionMessenger is to define what type of data needs to be exchanged between participants in your application. In DrawTogether, the specific data we need to share with other devices are the strokes themselves. We could represent a stroke with three properties: an identifier, a color, and a coordinate point. Note that we make our UpsertStrokeMessage conform to the Codable protocol. This is because GroupSessionMessenger allows us to send and receive structured messages and will automatically handle the serialization and deserialization for us, as long as the messages are codable. The second step for configuring the session is to receive data using the Messages API on GroupSessionMessenger. For DrawTogether, we will need to handle receiving the UpsertStrokeMessages. The messages API shown here is able to take the codable type and returns an async sequence -- which hands us a tuple containing messages of that type -- and the context surrounding the message -- which includes information like which participant sent that message. The third step for configuring the session is to send data using the send API on GroupSessionMessenger. For DrawTogether, we will send an UpsertStrokeMessage to all participants within the group. Note that the send API is an async throwing method. The errors it throws should be handled appropriately by your application. Now, let's go to Xcode and add our GroupSessionMessenger code. We'll start by navigating to the Canvas source file and creating a GroupSessionMessenger from our Group Session.

We'll add the messenger property to the Canvas which will hold the messenger object we just created. Next we need to define the UpsertStrokeMessage, which will be sent and received between participants. Let's create a new file for this. We'll do that by going to File > New > File... selecting Swift File... ...and let's stick it in our Model folder. And let's call it "Messages". Now let's add the code to define the UpsertStrokeMessage. Now that we've defined the message to send and receive with GroupSessionMessenger, let's write the code to send and receive it. We'll navigate back to Canvas to do this.

Let's go to the bottom of our file and add the code to receive the messages.

Here, we create a detached task to receive the UpsertStrokeMessages from the async sequence and call a new method, handle, to process the message. Let's implement that next. In this code, we check to see if we already have a stroke by checking for its identifier, and if so, add the point to it. Otherwise, we create a new stroke, add the point to it, and append the stroke to our array of strokes. Next, we'll write the code to send the messages. We'll go up to the method above, addPointToActiveStroke. Awesome! Now let's build and run our app and see a shared DrawTogether experience in action. So I'll go over to my two devices, and I'm going to start a FaceTime call with myself. I'll go to the Phone app and then call myself. And then I'm going to answer on my other device. Turn off the mics. Now on this device, I'm going to start a shared DrawTogether experience. I'll tap the bottom-left icon. And on the other device, I'll join the Group Session. And now I'm going to play tic-tac-toe with myself. I'll move first.

Looks like I won. Cool. Looks like our code works. Now that we've talked about how to configure the session using GroupSessionMessenger by first defining the messages, then receiving the messages, and finally sending the messages, let's talk about a few other things to consider when using GroupSessionMessenger.

Under the hood, GroupSessionMessenger provides reliable and FIFO-ordered message delivery to all active participants in the group. The messages you send do have constraints; if they are too large, an error will be thrown from the send API. GroupSessionMessenger is meant for smaller payloads and should not be used for streaming large assets like files, images, or videos. Another thing to consider when sending messages is flow control and rate limiting. Sending a burst of messages in quick succession, like a loop, could result in an error being thrown from the send API. Finally, when defining the messages to use with GroupSessionMessenger, consider adding versioning support to your application protocol. This will allow your applications to support inter-op with devices running an older version of your software. Now I'll hand it off to Adam who will talk to you about how you can polish your GroupActivities experience. Adam: Thanks, Angus! So now let's talk about some finishing touches that your app may need for its custom experience. First, let's talk about late joiners. Late joiners are devices that join into an activitySession after the session is started. To ensure a proper experience, late joiners need to be given the latest information so that all the devices are working off of the same data. Accounting for this scenario is critical to ensure a coherent user experience, but it isn't a one-size-fits-all. The data needed for this catch-up process will depend on your app and experience. So let's see how it applies to the DrawTogether app. Let's say that we have two devices in a Group Session. These two devices have the same information: a smiley face drawn on their canvas. You can see that because they were both in the session when the drawing happened, they have the same data throughout the experience. Now, let's add another device. At this point, the new device calls join on their GroupSession, but there's nothing on the canvas. So we draw a cloud, and -- Oh, that's -- that's not good! Because the new device didn't have prior context, our smiley face now has a cloud on it. Now let's back up and try that again. How do we fix this so that the new device has a smiley face show as soon as it joins? Once the new device calls join on the Group Session, every other device joined into the Group Session will see the activeParticipants property on GroupSession fire. Devices that observe that signal will then send its catch-up data -- in this case, the existing drawing canvas -- to the newly joined device. Now when the new device joins, it sees that there was a smiley face already there so we can draw around it! So now, how we do this in code? The first thing we need to do is understand what data our app needs to transmit in our catch-up message. Since we're a DrawTogether experience and our goal is to ensure that the canvas is the same on everyone's device, let's go ahead and make a new message in our Message.swift file called "CanvasMessage". This struct will contain all of the strokes that we have and a variable that we'll call "pointCount", which will be used as a heuristic to calculate which message is the most up to date. Awesome! Now, how do we handle receiving this message? If we go over to our Canvas model in Canvas.swift, we're able to set up a message handler in configureGroupSession using the GroupSessionMessenger like Angus showed us previously. From here, you can see that we call into our handle function, so let's go ahead and implement that. In this code, you can see that we guard against our pointCount heuristic to only accept catch-up messages that are newer than what we currently have. If that passes, then we go ahead and override our canvas' strokes with the catchupMessage's strokes. Now, like we discussed earlier, we need to listen for activeParticipants to change to figure out if there's any new participants that we need to communicate to. So let's go ahead and add that to our configureGroupSession function. In this handler, you can see that we grab the delta between our new activeParticipants and our old activeParticipants. This ensures that we only send our catch-up message to the newly joined participants. Great! Now we just form and send our message. This message will contain our current canvas' state and will send it only to the newParticipants. And that's it! That's catch-up! So, now that we have all the pieces in place for a Group Session to go on for a specific activity, what do we do if we're changing activities completely? This could be something like changing the drawing canvas or changing movies. Our API provides two ways of changing activities: you can either create a new Group Session or update the activity for everyone on your existing Group Session. So, let's talk about the two. The first and preferred way to change content is through calling the same API that started the Group Session: prepareForActivation on GroupActivity. This approach makes it easier to reason about a consistent state between participants, since it provides a clean barrier for entrance and exit of the GroupSession, so you don't need to worry about lingering states or messages that you don't need from the old GroupSession. This is incredibly helpful for when users back out of the activity to find the next one, such as searching for a new note or movie. This also gives the system an indication of a major change, which will be used to notify the user. After this call, in the same fashion as starting a Group Session, you'll receive your new GroupSession through the sessions async sequence on GroupActivity. Now, what if your application has a list of activities that you are going to be transitioning between, such as multiple songs playing after each other? Our GroupSession API provides a simple way for you to trigger an update for everyone by simply setting the activity property on the GroupSession. From there, you then listen to the activity property change. Our API will ensure that devices always converge onto the same activity so you don't need to worry about it. Now that we conceptually understand the two, which one should we use for our DrawTogether app? Because our app wants a clean slate on each new drawing canvas, the New Session API would give us exactly what we want. So now let's hop over to Xcode to see how to implement it. The first step here is to decide how we want to trigger a new session. In our case, let's make it so when a client uses our Reset button, we'll go ahead and create a new GroupSession. If we look at our ControlBar code, we can see that we already have a CapsuleButton that calls into our Canvas model to reset the local state. So let's go ahead and modify that function to tear down the GroupSession and create a new one. In this code, we'll go ahead and cancel any tasks and cancelables that we have for our GroupSession. We'll also check if we have a GroupSession -- and if we do, leave it and call into the activate property on our DrawTogether type. From there, our normal flow for receiving a GroupSession will take place, and we're good to go! We now have a clean way to transition to a new canvas. Now, what if we want to change our UI to indicate to users that they can try our SharePlay experience with their friends? For example, in our drawing app, we want our canvas to change from this to this where when we're eligible for a GroupSession, you'll notice that we now show a button to share the canvas. So how do we do this? With the GroupStateObserver API, we're able to listen to a publisher to tell us when the device is eligible for a Group Session. We can then use this to dynamically show and hide our button. So let's go and implement it! Like we saw, we want a small button in the bottom left of our application. Since our application already has the sharing button in our ControlBar view, let's just change the behavior to show and hide the button based off of the GroupStateObserver. First we go ahead and add our groupStateObserver to our view. Now let's surround our CapsuleButton to only show if we're eligible for a groupSession and we're not in this groupSession already.

And that's it! Our button now dynamically shows only when it's helpful to the user. Now let's go over what we saw in this session. We went through the full process of creating a simple drawing app that we then changed to leverage Group Activities to make it synchronized and connected like never before. But more importantly, we went through all the steps needed for you to fully unleash your creativity and create any custom SharePlay experience with Group Activities. We talked about creating a custom activity with a generic type, configuring and leveraging GroupSession and GroupSessionMessenger for synchronized communication, and edge cases and APIs your app should adopt to make a truly rich user experience. I hope you enjoyed creating this custom experience with us, and we look forward to seeing your creativity go wild with the Group Activities framework! Your next step while learning about GroupActivities should be the "Design for Group Activities" session And if you haven't seen it already, check out the "Build media experiences with Group Activities" session as well. If you have any questions, please find us at the Group Activities lab. And finally, thank you all for tuning in and have a great WWDC. We can't wait to see what you build! ♪