Explore WKWebView additions

♪ Bass music playing ♪ ♪ Megan Gardner: Hello! My name is Megan Gardner, an engineer on the WebKit team, and today we're going to explore WKWebView additions together. Before we start our journey, let's make sure we know what we're exploring. Let me tell you what WKWebView is and why you might want to use it in your app. When a user clicks a URL, they are expecting a web page to load and be able to interact with that content. Loading the URL, rendering the content, and running any JavaScript on that content is a very complicated task! So we have encapsulated most of that work for you inside of WKWebView. And not just for you! WKWebView is the view that is used to show all web content on iOS and much of MacOS. This includes Safari. So if you need to show a web page in your app, then you might assume you need to add a WKWebView. But wait! What are you doing with that content? If you need a way to view a web page, but you don't need much customized interaction with that content, you probably just need an SFSafariViewController. This is like a tiny Safari inside your app. Using one of these views is incredibly simple, but there is a trade-off. There is little you can do to interact with the web content. But this year, we have made it easier to do just a bit more with Safari view controller. Your users have always been able to run app extensions through the share sheet on your Safari view controller's content, but for special functions built only for your app, that has been a little cumbersome and hard for users to discover. So in iOS 15, we have added a new API to bring one of your app extensions to a customized button on Safari view controller. You can map this button to one of your app's share extensions, and you can even set an image for that button that will best represent the extension that will run, allowing users to run your app extensions directly from the toolbar, including running JavaScript on the page. But this is still a very limited amount of interaction. And that's where WKWebView comes in. WKWebView allows you to do many things with the content that it is showing. We have many APIs for loading, manipulating, and interacting with the content. So, if you need to show web content, and you need to have complex interactions with it, then WKWebView is for you. And since WKWebView is about interacting with web content, we are always striving to make that easier for you to do. We take on more of the workload by making easy APIs for you to adopt that will help you access and manipulate the content of your web view. This year we have been hard at work, adding new and helpful features to WKWebView, and today, I'd love to tell you about some of them. First, we'll cover several APIs that will allow you to interact with and manipulate your web content without using having to inject JavaScript. Then we will cover several APIs that will give you access to functionality that previously has only been available in Safari, allowing for a deeper browser-like experience in your app. Before we go over these APIs that will allow you to opt-out of using JavaScript, let's review why you might want to avoid JavaScript. We love JavaScript! It is the language of the web! But injecting JavaScript is complicated. The interface between native and web that you need to cross via injection can be difficult and cumbersome to navigate. It can also easily have unintended side effects or be difficult to manage when dealing with content from multiple web sources. It's best to avoid that headache if you can. There are also some features that are just incompatible with injected JavaScript. Last year, we introduced a new feature called app-bound domains. This allows you to specify which domains you would like to allow deep interaction with in your app. App-bound domains helps increase the security and privacy of your app for your users. But to get this benefit, you cannot inject JavaScript into your web view. Doing so disables the feature. There are also other high-value features, like Apple Pay, that you do not have access to in your web view if you choose to inject JavaScript. Therefore, we've added several new APIs to allow you to easily interact with the content in your web view without having to deal with injecting JavaScript. These new APIs fall into three categories. First, we have simple ways for you to access the pages' theme color and related colors for a website. Next, a way to disable text interaction, as well as being able to control media playback in your web view. First, let's explore theme color. You might remember last year we told you about one of our mission-critical internal apps, Browser Pets, which was written by Brady Eidson to combine the feeds from our WebKittens and Pups on Safari websites. Well, with all the new APIs available, Brady and I decided we should update this important tool, to get the most out of our many morale-raising cats and dogs. One of the engineers on the WebKittens website has decided to add a dynamic theme color that changes with the season, adding a little flair to our important pet photos. I wanted this to be reflected in the app as well, but I did not want to duplicate the code. Fortunately, we have a new API that makes this very easy. As you can see here, I can set the color of the header view to the theme color of the website using this new API, thus leveraging the work that my fellow engineer put into the WebKittens website. The website engineer can even add new themes in the future, and they will be reflected in the app without any additional work on my part. If themeColor isn't set, in a future seed we have an alternate calculated background color exposed as underPageBackgroundColor, which you can use similarly. You might have noticed how Safari is using this to create a look that blends the app with the web page. You can even write to this value to customize the color that is used to fill the background when scrolling past the end of web content. Next, let's talk about text interaction. We have recently added videos to the WebKittens and Pups on Safari websites, and Browser Pets users have been complaining about accidentally activating the text selection controls when trying to play those videos. Well, we don't want that! Fortunately, there is a new API to quickly and easily disable text interaction on a web view. It's simple! When setting up your web view, just make sure to set textInteractionEnabled to false in the preferences used to create the web view. This will turn off all text interactions in that web view. Now text selection controls won't get in the way of the media controls, and we can play those kitty videos unhindered. Speaking of media controls, we have some great updates for media playback controls! Previously, if you wanted to pause or suspend media that was playing in your web view, you'd need to inject JavaScript. You'd also need to find the specific element in the DOM to be able to control it. But now, we have a simple API that makes it easy to control the state of media in your web view. You can now easily pause all media, close all media windows, get the state of the media on the page, or set media suspended, which will pause media and disable user controls until setAllMediaPlaybackSuspended is set to false. To help you get more familiar with these new APIs and how easy they are use, I'm going to turn it over to Brady who will show you how we utilized these new media control APIs in Browser Pets! Brady Eidson: Thanks, Megan. Hello, I'm Brady Eidson, an engineer on the WebKit team. As Megan alluded, both the WebKittens and Pups on Safari websites recently added the ability to post videos instead of just still images, which means our Browser Pets app has videos now, too. Here's how the app looks right now. These buttons are native UI buttons for various features I'm adding. That banner at the top of the page is web content, and the website has set a theme color, which I have adopted using the web view themeColor property Megan mentioned. Once videos were added, some folks expressed their preferences to not have videos autoplaying. This Pause button is to accommodate them. Let's see what it does.

Just the kittens video paused, not the dogs. And pressing Play, nothing happens. Let's take a look at the code.

This JavaScript was my only option to try to pause these videos before, and it's been problematic. I have to understand the structure of web pages from two different websites, and that structure is constantly changing. For example, this JavaScript from the Pups website used to work but doesn't anymore. Fortunately now there's a better way. First, I adopted calling pauseAllMediaPlayback on the web view.

That's equivalent to calling the JavaScript function pause on every media element on a page.

And it works nicely. But notice what happens when I refresh the page. The videos play again. New page load. These videos have never been paused. If I want something more sticky, I'd use setAllMediaPlaybackSuspended. Let's try that now.

Notice how it pauses the videos, and they remain paused, even after refreshing the page. That's because this setting is a property of the web view itself and not of any content inside of it. It also allows me to resume. Thanks to these new APIs, you now have the flexibility to add a better media experience to your app. Back to you, Megan. Megan: Thanks, Brady! Those pets look great, and even better with the new media controls. So those are our new APIs for WKWebView that let you avoid using JavaScript. We hope they make writing your apps and interacting with your web content even easier than before. Next, we'll review several browser-level APIs that will give you access to functionality that previously has only been available in Safari. We have three areas of API to cover. First is a simple HTTPS override flag. Next is media capture or -- as it known on the web -- getUserMedia, which is fully available in your web view now. And finally, a new API to manage downloads which will work in connection with WKWebView to allow and manage downloads in your app. So let's start with HTTPS. Security and privacy on the web has always been important, and we are always searching for new ways to make security and privacy easier for you and your users. There has been a broad industry push to move traffic to HTTPS, as this is a safer and more secure way to browse the web. Many top sites support HTTPS, and some even automatically upgrade connections from HTTP to HTTPS. Beginning in iOS 15 and macOS Monterey, we are taking HTTP requests to sites that we know support HTTPS and upgrading them for you. In order to get this added security, you don't need to do anything at all! But, if you do need to turn it off for some local debugging, there's an easy flag to set on configuration. Hopefully you will not need this, but it's here if you do, and it probably shouldn't be done in production code. Next, let's review what's been happening with media capture. Staring in iOS 14.3, we enabled getUserMedia for WKWebView, which allowed WebRTC functions to work inside your app. Now, we've made it even better! When you load your web content from a custom scheme handler, the user request prompt will show your app as the origin of the request, rather than show a request from the website URL. This should make the experience more intuitive and seamless for your users. If you want the prompt to remain as a request from the URL, just load without the custom scheme handler, and the prompt will be shown as it is today. We also have a new API to allow you to decide when and how to prompt the user for camera and microphone permissions when working with web content. Once you have obtained user permissions for the camera and microphone via normal entitlements and prompts, you can decide if the permission prompts should be shown. You can use this to implement your own prompt or remember the users' previous response for web content you control and trust. By creating and setting the UI delegate on your WKWebView, you can now decide when the user should be prompted for permission to use their camera or microphone. If you do not implement this delegate, users will continue to get the prompt as they do today. But if you want to skip the prompt because you have saved the user preferences from previous interactions with a website, this delegate will allow you to do that. How did we leverage these new APIs for Browser Pets? Well, due to circumstances beyond our control, the team had to cancel our monthly Pups in the Park meetup. To fulfill that need to share our dear doggies with each other, Brady and I decided to add a virtual dog park to Browser Pets, so we could still get our social doggy goodness going on. This involves using getUserMedia to set up WebRTC functionality so we could see and chat with each other and our doggie friends real time! When setting up the UI delegate, we know that requests from our server are ones that our users would give permission to if they already gave permission to the app to use the camera and microphone. So I can skip that prompt by setting this delegate on my WKWebView. But that's not all I can do! Instead of having to execute JavaScript to start and stop this user media, we can get and set the state, similar to how we can interact with media controls without JavaScript. This work for camera as well as for microphone. To help you get even more familiar with these new APIs and how easy they are use, I'm going to turn it back over to Brady who will show you how we added that virtual dog park to Browser Pets! Brady: Thanks, Megan. As Megan mentioned, we had to get a virtual dog park up and running because of the times we're living in. Let's take another look at the Browser Pets app. Some of you may have noticed a Call button. Let's tap that now. We're instantly connected to our virtual dog park hosted at our friend's house. Let's join the fun by tapping Connect. Now, because this is the first time we've ever used the camera in this app, we get the system-wide prompt about Browser Pets. Let's definitely choose Allow. And the second prompt -- specific to the website we are currently viewing -- is the default behavior today in the system. Let's allow that also. And there we are. getUserMedia in action. But let's see if we can make a change to our WKUIDelegate, like Megan mentioned, to make the experience even better.

In our delegate, we'll just check for the host providing our virtual dog park content and allow media capture access. Let's see how that changes Browser Pets.

Great. No prompt. And the way we used the delegate, we also keep our users safe and private if future content comes from any other host and tries to access the camera. Now that we're in a video chat, let's try the Mute button. As you can see, my camera feed dropped. You'll also notice that the recording indicator in the status bar has disappeared and unmuting, I can bring back my camera feed and the recording indicator returns. This was easy; no JavaScript necessary. All I had to do was hook it up to webView.setCameraCaptureState as Megan had mentioned. This was just a taste of how these APIs enable you to easily customize the media capture experience in your WKWebView app. Back to you, Megan. Megan: Thanks, Brady! I'm so glad we could continue to meet with all our fabulous pets. Lastly, let's talk about downloads. Browser Pets users have been telling me they want to be able to share these cute animal pictures with their friends and family, but only Apple employees have been allowed to use the Browser Pets app. Fortunately, we've added an API that lets you allow and manage downloads from your web view, making sharing files easier than ever! There are three ways to initiate a download from the web. The web content can initiate a download, the server can initiate a download, and the app can initiate a download. Web content initiates a download with JavaScript similar to this. When this JavaScript is executed, it initiates a download navigation action, which calls the navigation delegate with shouldPerformDownload set to true. You should use this value to determine -- along with other logic -- if the download should be allowed. The server can initiate a download in HTTP like this after calling loadRequest on your web view. When that happens, your WKNavigationResponse will have a Content-Disposition header field with a value containing "attachment", and when it does, you should return WKNavigationActionPolicyDownload from your decision handler to start the download. Finally, your app can decide it wants to download something in the context of the current page using an NSURLRequest like this. Whatever method you use, when you get the WKDownload object, you need to set the delegate property on that object to be able to tell it where to write the bytes to disk. If you do not, the download will automatically be cancelled. If a download fails, the data to resume the download will be handed to you if you implement the method on the delegate. You can then use that data to restart the download off of a web view. With this new API, I can now offer an easy way to download the kitties to Files, and not to save them to Photos. So those are our browser-level APIs. We think they will offer you and your users many more options for having a rich web experience in your app. Wow, what a journey! Today we have explored so many things together. We've talked about several APIs that you can use instead of injecting JavaScript, and we've covered some more powerful browser-level features. It's so much easier to use our new theme color, text interaction, and media playback APIs. And we've made it easier and safer to have a richer experience in your app with automatic HTTPS connections, easy and safe control of getUserMedia prompts, and the addition of WKDownload, so you can easily manage downloads in your app. You can learn more about some of the new web APIs we are supporting this year, in the "Develop advanced web content" session. You can also discover new ways to debug and improve your web content in "Discover Web Inspector improvements"; two great sessions that will help you with making great web content for your app or for the web. The web is still growing, and as it does, so will we. Many of the APIs we implemented this year were based on developer feedback. So if you have an API you are wanting to see, we want to hear from you! In addition to feedback with Apple, webkit.org has multiple ways to reach out, from our Slack workspaces to our mailing lists. WebKit is open source, so this is also where you can learn to check out and build WebKit yourself, as well as file bugs. Thank you joining me on this exploration of our new APIs! I hope you have a great rest of WWDC21! ♪