Introducing SiriKit Media Intents

Hi. I'm Danny Mandel. And welcome to Introducing SiriKit Media Intents. We've added media domain support to SiriKit for audio use cases and we're super excited to tell you all about it. So what are we going to cover in this session? The first thing we'll do is introduce the new SiriKit Media Intents and talk about their capabilities.

Then we'll talk about what's required for you to handle SiriKit Media requests on your app. And finally, we'll talk about some best practices you're going to want to follow to provide the best user experience possible when you add SiriKit Media Support to your app. This year, we're allowing you to control audio in a whole new way with SiriKit Media Intents. And we think people are going to love using the Siri Media capabilities you build into your apps. With SiriKit Media Intents, people will be able to do things like play audio, update taste profiles, add to collections, and search. This means that they'll be able to use the rich natural language processing capabilities of Siri to say things like "Play Khalid on my app," to immediately begin playing Khalid in your app.

Let's take a look at the new SiriKit Media Intents and their capabilities.

There are four SiriKit Media Intents. The first intent is INPlayMediaIntent. INPlayMediaIntent allows people to play audio by saying something like "Play Outer Peace in my app." Now, you might remember that we launched Media Playback Support in IOS 12 with Shortcuts. And this year, we're adding SiriKit features to INPlayMediaIntent. The next intent is INAddMediaIntent. INAddMediaIntent allows people to add items to their playlists and libraries. An example of this could be "add this song to my road trip playlist in my app." We have INUpdateMediaAffinityIntent, which allows people to express affinity for media items. People can say this by saying something as simple as "I like this song." And finally, we have INSearchForMediaIntent, which lets people search your app for a particular media item. For example, "Find Billie Eilish in my app." Let's talk about the supported media types in SiriKit Media. SiriKit Media supports a number of different audio types, with the first one being music. And music support is going to let you say things like "Play the song Awesome Song in my app." In addition to songs, we have support for albums, artists, playlists, genres, and many more. So you're going to want to check out the documentation for INMediaSearch to get the full list of supported search terms. And we want you to adopt as many of them as possible to provide the best Siri user experience in your app. Additionally, we have playback controls, like shuffle, repeat, and playback queues. And this lets people say things like "Play Khalid shuffle in my app," or, "Play Outer Peace next in my app." The next supported audio type is Podcasts. People can begin playing podcasts by saying something like "Put on the Stuff You Should Know podcast for my app." Additionally, people can also control the playback order and speeds of podcast episodes. This lets people say something like "Play the newest episode of Stuff You Should Know podcast in my app," or, "Play the Stuff You Should Know podcast in my app at double speed." Moving on, we have audiobook support. This lets people say things like "Play the audiobook Becoming in my app." And like Podcasts, people can begin to playback at a specific speed when asking to play audiobooks. And finally, we have radio support. Radio support allows people to ask for a specific radio station in your Radio Playback app. For example, "Play 89.1 FM in my app." And don't worry if your app doesn't fall into one of the previous media types, you can still adopt the SiriKit Media Intent and get the full power. People will be able to say things like "Play search term in my app," and you'll be able to look up search term in your app and play it. The only thing that will be missing will be support for strongly parsed media types. So say you had a nature sounds app and you said "Play reptile sounds in my app," or, "Play mammal sounds in my app," Siri is not going to know that those are two different types of animal sounds. So you'd get a string of mammal sounds or reptile sounds, and you could look it up and play it. So not quite as structured as the other types but still supported.

Let's look at how we handle these intents in SiriKit.

So the first thing to know about how to request with SiriKit Media is that SiriKit Media Intents are just like any other SiriKit domain. So all the intent-handling happens in your Intents app extension, where you conform to the SiriKit Media Intent handling protocols. The details of SiriKit request handling have been covered really well in previous WWDC talks, so I'd refer you to those talks and to the developer documentation online for more general details about SiriKit Request Processing.

Now let's look at what happens for a typical request in the SiriKit Media domain.

The request processing begins when someone says "Play cool song in my app." And Siri is going to recognize that this is a request for your app, it's going to launch your Intents Extension. Now, there are three steps in SiriKit Request Processing: resolve, confirm, and handle. The first step in request processing is the resolve step. In the media domain, the resolve step is where we take the intents' INMediaSearch object and we run a search against our app catalog. The output of resolve is one or more concrete media item objects to play.

Alternatively, if we didn't find anything that matched or another error occurred, we can return an unsupported result, which will tell Siri to display the appropriate error dialog. The next step in Request Processing is the confirm step.

And typically, we discourage use of the confirm step in the Media domain. In looking at usage in our own apps, we find that using confirm actually lowers the likelihood that people will continue on to play media. So we don't recommend using the confirm step in the Media domain. The final step in Request Processing is the handle step. Now, for INPlayMediaIntent, this ends up being really simple, because we're going to return the response code "handle an app," which is going to do a background app launch. And inside of our background app launch, we're just going to play media like we normally do in our app. The only tricky part here is testing. You're going to want to make sure everything plays because there's not going to be any UI on screen. You're also going to want to make sure that you test in a variety of situations. For example, in CarPlay or when you're wearing headphones. So now that we've seen an overview of SiriKit Request Processing, let's take a look at a simplified resolve media items method. And the first thing to note is that the parameters are going to be slightly different but the same resolve logic is going to be the same for all four media intents. Resolve's job is to search the app catalog. And you're going to want to do it the same way for all four intents. So we'll initialize a result to the unsupported result. And this is going to tell Siri to say the appropriate error dialogue if we don't find anything in our app catalog. INMediaSearch is the intent field that contains the details about what the user asks to play. INMediaSearch represents the universe of all the audio-related queries that Siri supports. Our job in Resolve is to go from that universe of possibilities to a single item to play. And in this example, the first thing we're going to do is read the media name off the INMediaSearch.

Then we're going to retrieve a list of items from our app catalog. And we're going to use the media name property off the media search to compare against the item's name property. And we'll talk a little bit more about this later on, but this isn't really something you're going to want to do in your shipping app. But if we had an exact match on the name, we found the item to play, and we're going to create a success result using that item's properties.

Then we'll call a completion handler and move on to handle. And in this case, as we said before, handle ends up being very short, since all we're going to do is return the handleInApp success response code. And this is going to start the process of background launching our app. Now let's take a look at background app launch.

The method that we implement in our app delegate to support background app launch is application handling intent completion handler.

Again, this is a pretty short implementation. We're going to read the first media item to play out of the intent and then we're going to just play it in our app the way we normally do.

And finally, we'll call the completion handler with our success response code. So now that we've seen the new intents and how they all fit together, let's hand it over to Ryan Klems, who's going to show us how this all works in a real app.

Thanks, Danny.

Adding SiriKit Media Intent handling to your existing media application is easy. Here we have our music application and all we need to do to add Siri support is to add the Siri extension target, add a few methods, and then we'll up and handling Siri requests in no time. To add to the intent's extension, all we have to do is go to file, new, target, select the intents extension and click next. Give it a name and then click finish.

It will go ahead and create our intent handler for us. We'll want to go ahead and add the Siri capability to our application.

And then we'll come over to our control extension and we will add our intents that we support, and in this example, we'll just go ahead and support the INPlayMediaIntent and the INAddMediaIntent.

We'll go ahead and select the music type.

We have a few methods here that we want to add to make sure that we build in our extension.

And we'll want to make sure to turn on our proper code signing.

Now we'll come over to our intent handler and all we need to do is add support for the INPlayMediaIntent handling protocol. And then we'll drop in some stubs for our resolve and handle methods. For this beginning example, we're just going to return the unsupported result from resolve media items, which will cause Siri to speak to the fact that it couldn't find the item. So we'll go ahead and try that out. This is what that would've looked like.

So Siri speaks to the fact that it couldn't find the item due to the fact that we returned the unsupported from the intent handler.

So what we'll do now is we'll go ahead and hook this up to our existing search implementation.

And in this case, the first thing that we're going to want to do is determine what the user is searching for. So for this simple example, we'll just search for an artist. And in the method, we'll go ahead and resolve the media item.

And once we resolve the media item, then we will return to our handle method and we're just going to return the handleInApp method response code, which will cause us to background launch the application. So in order to do that, we'll switch over to our app delegate and we need to add the handle intent method. And this will just extract the INMediaItem that we passed, that we resolved, in the previous step, and pass that to playback. And so we'll go ahead and play what that would look like.

So you can see, we return the INMediaItem and it has resolved that, handed it back to the application, and begun playback. So now that we've done that, why don't we go ahead and add support for the add method. So to do that, we'll just extend this by adding the INAddMediaIntent handling protocol.

And then we'll add our methods for resolving and handling for the add method. So you notice here that the resolve media items for add looks pretty much identical to the resolve method for play.

Additionally, now for ad, we'll also have a resolve media destination. And this is where we're going to determine whether the user's trying to add to the library or to a playlist. And in the case of a playlist, you might want to do something like "Return playlist name not found," if the playlist that the user specified was not present in their library.

And also, what's different in the ad is there's no reason to go back to the application like to begin playback like we do in the Play Media Intent. So in this case what we would do is we would actually handle everything in the extension itself. So we have the resolved media item, and we'll just go ahead and use our applications methods to add to the library or to the playlist. And in this case, we're just going to use the media player utilities to add to it. So let's go ahead and take a look at what that would look like.

So it speaks to the item that is added to the playlist as well as the playlist name, because those are specified in the request. So as you can see, it's relatively easy to add support to your application. And we really look forward to seeing what you do in your application. Thank you. Danny, back to you.

Thanks, Ryan. So what did Ryan show us? First, he showed us how to add our intents extension to our app. Then he showed us how to specify our supported intents and supported media types. And finally, he showed us how to implement resolve and handle for INPlayMediaIntent and INAddMediaIntent so we could immediately begin playing and adding media using Siri.

So let's look at some best practices you're going to want to follow when you adopt SiriKit Media. We have great news if you've already implemented shortcut support for media playback. SiriKit Media uses the same code and handle and for background app launch. While Shortcuts operates on previously donated intents which don't require intent resolution, SiriKit does require a resolve step. So the two things you're going to need to add are your resolve method and your intent handler, and you're going to need to update your intents extension supported media types in Xcode so Siri knows what content types your app supports.

The extensions handle method should be the same between both implementations and the background app launch to the app delegate's handle intent can be the same, as long as you use the same identifiers for your media items. So let's see what this looks like. Here's our Shortcuts implementation from last year. You'll notice that there's no resolve method, but the rest of it's the same. So to go from Shortcuts to SiriKit, we just add in our resolve method and we're good to go. Now let's talk about what you need to do to bring SiriKit Media Support to the Apple Watch. On watchOS, apps launch in the foreground. And the way you do that is by returning INPlayMediaIntent response code continueInApp from your handle method in your intent's extension.

This code is going to do a foreground app launch and forward the intent to your WKExtension delegate in your app. You'll note that the apps handle method looks pretty similar to the one in iOS. The method signature is slightly different, but you're going to want to read the intent off the NSUser activities interaction property. And then, just like on iOS, you read the media item to play and start playback in your app. One word of caution. You're going to want to use the on-device cache in your resolve method on watchOS. Only go over the network if it's absolutely necessary. So we know that when someone says "Play Awesome Song in my app," the first step in Request Processing is to resolve the media items to play. And we looked at a previous implementation where we checked the value of the media item's name against the intent's media name property.

And it was an exact match. So what are some edge cases that we didn't cover in that implementation? The first place that our previous method won't do the right thing is if we have a mismatch on either case or punctuation. So in this example, "Play hello in my app," we can see a few cases where exact string comparison will fail.

The exact song title is uppercase HELLO with an exclamation point. But it's possible that the Siri speech engine could give us lowercase hello. Or maybe it gives us uppercase HELLO without the exclamation point. So it's really important that we ignore case and punctuation in our resolve method.

Similarly, a lot of music entities have things in the title that people either won't say or they're going to say in a way that doesn't exactly match with the item's title. For instance, a lot of albums come in a deluxe edition variant. And people aren't going to say this when they ask to play the album.

They aren't going to say "Play the album Outer Peace deluxe edition in my app," they'll say "Play the album Outer Peace in my app." And soundtracks are another example. People aren't going to say -- they'll say "Play the Rocket Man soundtrack in my app." And they aren't going to say "Music from the motion picture." And finally, a lot of hip-hop songs have this featuring abbreviation in their title. So people either won't say it or they'll say the word "featuring." So exact match isn't going to work here either.

And podcasts also have some cases where there is a mismatch between what people say and what entity titles are. So some podcasts have the word "podcast" in their title. So if someone said "Play the Stuff You Should Know podcast in my app," Siri could parse it as stuff you should know in a media item type of podcast. So an exact match isn't going to work here either. And additionally, some podcasts come in audio or video variants, and that audio or video word appears in the title. But SiriKit Media implies the audio variety, so people aren't going to say that either. Finally, remember that you're working with a speech recognizer, and the speech recognizer can come with word formatting variations. So if someone asks to play the song 81st, you can get the number 81 st or you can get the hyphenated eighty-first.

Or if someone asks to play the song I Love You Son, you could get sun or son. Now, Siri is going to do the best job it can to give you the entity titles for the things that it knows about. But it's better for you to be flexible in your resolve method.

When you implement your SiriKit Media Support, you control what Siri says by the INMediaItem objects you return from your resolve method. As you can see here, the user asks to play the song Maybe Sometime by Special Disaster team in Control Audio. And Siri said, "Here's Maybe Sometime by Special Disaster Team from Control Audio." In this case the returned INMediaItem had a title property of Maybe Sometime and an artist property of Special Disaster team. So make sure you always populate title, artist, and type in the returned media items, as these can all influence what Siri says. And one thing to note, if you return more than one item in your resolve method, Siri is going to speak to the first item in the list.

It's super important that you handle error cases appropriately in SiriKit Media. When you're interacting with an intelligent assistant like Siri, it can be unclear why something happened when it happened. And handling error cases appropriately is going to allow you to give the user the best idea of what happened when something goes wrong. So the most common case you're going to run into is when you don't find something in your app catalog that someone's asked for. And you handle this case by just returning the unsupported resolution result from resolve media items. But there's a lot of other errors that could happen. Maybe someone asks to play something that requires cellular data but they have the cellular data switch turned off for your app. Or maybe they ask to play something that requires a subscription and they don't have one.

The full list is in INPlayMediaMediaItem UnsupportedReason. And there's generally similar naming for all four intents, so make sure you adopt them for all the intents you support. Now let's talk about some of the variety of things that people say to Siri and how you can do a good job handling of them in your SiriKit Media implementation. So one of the most popular things that people say to Siri is "Play my app." They don't tell you exactly what it is that they want to play, and it's your job as a SiriKit Media developer to choose the right thing to do for your app. Now, this might be something as simple as resuming an existing queue. If you're not an audiobooks or podcasting app, this is probably the most reasonable behavior to implement.

But you can make the behavior as dynamic as you like. Maybe you want to direct them to a recommended playlist or some hot new trending music. The choice is yours. And how do you know that someone said, "Play my app," well, there will be no search criteria specified in the INMediaSearch object. One thing that might seem like a good idea is to ask someone what they want to play. But for the same reason that we don't recommend using confirm, we don't recommend this approach. Putting dialogue prompts in front of people is one of the most common ways that they'll quit the SiriKit Media experience. People can ask to initiate playback with additional controls, and some of the supported options are repeat, shuffle, resume, and playback queue location. And if you support these in your app, make sure you support them in your INPlayMediaIntent implementation as well.

And people can also ask to play content with a variety of search options. And one of the most useful search options is the sort parameter. You can say things like "Play the new Stuff You Should Know podcast in my app," and you'll get INMediaSortOrder newest. Or you can ask for the best album by an artist, and you'll get INMediaSortOrder best. Or you can ask your app for a recommendation, and you'll get INMediaSortOrder recommended. Check out the full list in INMediaSortOrder. And another powerful search option is the reference property, which can have INMediaReferenceCurrently Playing.

This is really useful for INAddMediaIntent or INUpdateMediaIntent, because people can easily add the currently playing item to a library or a playlist, or they can tell your app that they love or they hate their currently playing item. And one thing to note too is if you've populated the external content identifier in MPNowPlayingInfo center, that identifier is going to be in INMediaSearch's identifier property, so you know exactly what item to find. Now, telling Siri more about how your customer uses your app is going to help Siri provide a wonderful SiriKit Media experience.

So when you give user vocabulary to Siri, this helps Siri understand the parts of your catalog that are interesting to your customer. It's important to note it's not the entire contents of your app catalog, it's only the pieces of content that your customer is specifically interested in.

It's also important to note that the vocabulary is ordered, so make sure that you include the most important items at the beginning of the collection.

And depending on the type of media your app supports, you're going to send different types to Siri.

Music apps should use playlist title and music artist name. Audiobook apps should use audiobook title and audiobook author. And podcasting apps should use show title.

And for those things that are applicable to everybody that uses your app, check out the global vocabulary support in SiriKit. So in conclusion, we're launching new SiriKit Media Intents to allow you to use Siri to control your audio apps. You'll be able to play, add, update taste profiles, and search for media using the new intents.

It's very important that you provide the best experience possible. And you can do this by embracing search flexibility. Because people don't say exactly what they want to play. You can handle errors appropriately, so people know what's happened when something goes wrong. And you're going to you want to make sure that you construct your INMediaItem objects appropriately so Siri can speak the best dialogue possible. And finally, make sure you give Siri the important contextual information possible so Siri can make the best choices possible on your customer's behalf. So come see us at the labs and check out the documentation online. We think you're going to love building apps with the new SiriKit Media Intents and we can't wait to see what you'll build. Thank you.

[ Applause ]