Transition media gaplessly with HLS

♪ Bass music playing ♪ ♪ Simon Goldrei: Hello. This session is all about how to transition media gaplessly with HLS. I'm Simon and I'm a member of the Streaming Media team here at Apple! Do you utilize AVQueuePlayer, or would like to offer your customers a seamless, Up Next-style playback of HLS items in your application? Do you want to present your media with the continuity that the producer intended? Can we support our customers by providing an experience to enjoy our media for longer? If yes, then you've tuned in to the best session to learn how you can now transition HLS items gaplessly. Here's our agenda. We'll survey some of the engaging media experiences you can deliver with this new feature, I'll detail the media authoring requirements, I'll step through some playlist examples highlighting best practices, and we'll end with a demo. The benefits of transitioning between HLS-backed AVPlayerItems in an AVQueuePlayer are straightforward. Until now, the point of transition could be a little jarring; anything from an audible glitch to a momentary visual hiccup or even an indication of indeterminate buffering. Gapless item transitions maintain a bond that your application provides between the customer and the content. Perhaps you recall the phrase "don't touch that dial." You can build an episodic treatment that seamlessly transitions from the end of one episode into the start of another. Gapless transitions permit the ability to replicate the seamless track-to-track album experiences that we've long enjoyed. When doing this, we're delivering a more correct reproduction of what may have been live or intentionally authored. We can deliver an experience that is unbroken and uninterrupted. We can replicate linear programming. But we can also provide a dynamic and perhaps even interactive experience that takes the viewer from one task or scene and directly segues into the next. Consider a transition between a set of workouts or other instructional media. With gapless transitions we can now, programmatically, stitch scenes of an audiovisual sequence while maintaining accurate continuity. You can add reps... ...or add a cooldown, to the workout. We can achieve all of this while still delivering media with all the benefits of adaptive bitrate media streaming over HTTP. Of course, we use this, too! New features of Apple Music deliver content via HLS and achieve a gapless transition between each song. We're very excited to bring this to our customers, and we'd like to offer this to your customers, too. Let's now take detailed look at the authoring requirements you need to incorporate to affect the same item transitions in your HLS content. By providing -- in your HLS manifests -- variants that offer audio-format continuity between sequential items, you're enabling a gapless transition. Your content should provide variants that match the following audio media traits: the use of FairPlay Streaming, the audio codec -- specified by the codecs attribute of the Stream tag -- and the channel count, as specified by the Channels attribute of the Media tag. Note that if you decide to offer audio renditions, at different sample rates and bit depth, you may inhibit the gapless transition. The same advice to provide a match and maintain continuity applies to these traits, too. Which is to say, to achieve a gapless transition, be sure to offer equivalent audio media in each variant for consecutively enqueued items. Finally, when authoring your media, please adhere to CMAF authoring guidance. CMAF stands for "common media application format." In this MPEG specification, there is a specific recommendation to use an edit list to signal the presence of priming and remainder frames. Let's now look at a simple example by inspecting the master variant playlists of two candidate items. In this HLS manifest I've offered four audiovisual variants: a pair of 720p variants and a pair of 1080p variants. Within each pair, I offer a variant encoded with the High-Efficiency Advanced Audio Codec, HE_AAC, and another with AAC Low Complexity, or AAC_LC. In this example, consider that playback conditions support the selection of the 720p video paired with HE_AAC. As playback nears the end, the AVQueuePlayer will, at least initially, select the similarly encoded variant offered in the subsequent item. This is true even if network conditions now support a higher-quality tier. Servicing the gapless transition is a higher priority than choosing a high quality when transitioning gaplessly. As playback proceeds, the adaptive bitrate heuristics are evaluated and, if the conditions persist, the player will seamlessly switch to that higher quality. Fantastic! Now suppose we didn't offer HE_AAC in item two. In that case, the AVQueuePlayer is unable to maintain audio continuity and the gapless transition but is free to select the most appropriate initial variant for the playback conditions. Let's say that we have authored our content to support a gapless transition. In that case, we should remind ourselves how to use an AVQueuePlayer. Here we have a short code snippet. We create two AVPlayerItems and enqueue them, in order, ahead of playback. Note that our items source from two different URLs. After creating our AVQueuePlayer, we can use the "insert after" method to inform the queue player of the intended sequence. All that's left is to message "play".

An advanced technique I'd like to share with you permits the use of a single asset. From this asset, you can instantiate multiple AVPlayerItems. We can define different in and out points on each distinct AVPlayerItem, prior to enqueuing in our AVQueuePlayer. Using the AVPlayerItem seekToTime method, we can define our in point. Then we can define our out point with forwardPlaybackEndTime. Here's an illustration of this concept. Item one, two, and three are all sourced from the same AVAsset, consisting of the same three scenes. They are distinct in that I have cued each with different in points, using seekToTime, and defined distinct out points, using forwardPlaybackEndTime. Now I can enqueue these three otherwise identical items into my AVQueuePlayer, and what I achieve is a customized, dynamic sequence. Now let's see and hear gapless transitions in action with a demo. In this demo, we will interactively stitch together three workout video streams. I've left the AVKit controls visible in each so that you can really see these are distinct stream resources. I'll start by showing a seamless playback of all three and then we'll edit the sequence interactively. In your software, you can imagine offering the same -- or perhaps perform the reordering -- programmatically. Right! Let's do a warm-up. ♪ Energetic electronic music ♪ ♪ Ready or not? Let's do a hills run. ♪ Lively hip-hop music ♪ ♪ That's it! We're done. It's time for a cooldown. ♪ Calm electronic music ♪ ♪ I can break a sweat just watching that. I'm tempted to skip straight to the "That's it, we're done," and skip the hills run. There's a lot of that here in San Francisco. To do this, let's edit the order of our exercise sets. I'll move the hills exercise to the top. This allows me to just do the warm-up then immediately and gaplessly segue into the cooldown. Then I'll start by selecting the warm-up, now in the second position, as my first workout. Right! Let's do a warm-up. ♪ Energetic electronic music ♪ ♪ That's it! We're done. It's time for a cooldown. ♪ Calm electronic music ♪ ♪ Let's summarize the key points that we've learned in this session. We've learned that to achieve gapless transitions using HLS, we must provide variants in each item that have matching audio formats. Then we saw how easy it is to instruct an AVQueuePlayer to perform the gapless transition just by enqueuing items in sequence. We have one related session, where we introduce new APIs for variant discovery. You may find that session useful if your software or service cannot guarantee the items of matching audio format. I hope you've enjoyed this session and enjoy the rest of this WWDC 2021. Thank you. ♪