Exploring New Data Representations in HealthKit

Good afternoon.

My name is Luke Spicer and today my teammate Divya and I are very excited to show you how HealthKit has expanded our data representations in iOS 13. As many of you already know, HealthKit provides a centralized data store for health and fitness data while also providing interoperability for health and fitness apps and experiences. If you're not already excited about using HealthKit, let me remind you that there are over 70,000 apps in the health and fitness category on the App Store and all of them can benefit from the interoperability and functionality provided by HealthKit.

Today, we're going to show you a few things related to data representations in HealthKit, starting with a review of the HealthKit data model. Then I'll go into some specific APIs focused on quantity data.

Later, Divya will come out to show you beat-to-beat measurements, heart rate events, and the brand new API's focused on hearing health. Let's jump right in and start with the HealthKit data model. In HealthKit, we represent our data-- in HealthKit we represent our data with samples, which are measurements taken at a particular time that span some time interval.

Most of these measurements are simple measurements, like the heart rates measured with Apple Watch or body mass measurements taken with a connected Bluetooth scale. Other measurements are more rich and complex, like workouts or the clinical health records that can be downloaded directly to iPhone.

All of these rich complex data representations share a common structure; a top-level sample that's backed by some specialized data representation. For example, a blood pressure measurement can be represented with a correlation sample, an allergy resource by clinical record, and a workout route by a series sample. All of these are examples of specialized sample types that all have their own unique backing data format. For example, the blood pressure correlation is a set of blood pressure measurements. The allergy resource is backed by a file resource, and the workout route is an array of CL location data. And just as HealthKit provides these specialized data representations for these special types of measurements, we've also expanded our data representations for our most common type of measurement, which are for quantity data. So, let's continue by talking about quantity data.

To quickly review, quantity data in HealthKit applies to some very commonly-measured concepts, like walking distance, body mass, and heart rate. All of these concepts we refer to as quantity types and they have measurements that look something like this, where each of these measurements consists of a quantity that is a value and a unit as well as a date interval that tells us the interval over which the measurement was taken.

We call each of these measurements a quantity sample.

Throughout this presentation, we're going to talk about quantity types and quantity samples specifically focused on how we can efficiently represent large numbers of quantities, but if you'd like more of an introduction to these concepts, you can always refer back to our 2014 presentation, Introducing HealthKit.

To forward our discussion of quantity data, I'd like to describe a scenario. If you would, please imagine that together we're working on a brand-new app. This app connects to a never-before-seen heart rate sensor that's been built right into a video game controller. Our task is to take measurements from this heart sensor and save them to HealthKit so that our users can see what their heart rate was when they play any particular game. The heart rates that we're receiving from the sensor might look something like this; a sequence of measurements that are coming into our app over time while the user plays. And we have to decide how we want to represent this data so that we can save it to HealthKit.

One approach that we could take would be to use a single quantity sample and that might look something like this.

This single quantity sample spans the entire measurement interval, that is the entire duration of the game, and represents all of the measurements we received from our sensor with a single quantity. Maybe an average.

This representation gives us an object to represent the measurement we took, which was the heart rate during this game, but if later we want to see changes in heart rate over the game, we won't have the resolution to do that with this representation.

Another representation we could take is to use multiple quantity samples and that might look something like this, where every measurement is represented by a different quantity sample. This representation will allow us to keep the full resolution measured by our sensor, but it is not an efficient representation of this data because we have redundancy. All of these quantity samples have identical metadata and device information and we no longer have that single convenient object that represents the measurement that we actually took, which was the heart rate during the game.

Thankfully, we have a third approach that we can take that we call quantity series. The quantity series representation looks something like this. Again, a single quantity sample that spans our entire measurement interval, but in this case, instead of holding a single quantity, it can be backed by multiple quantities.

This representation gives us the best of both of the previous approaches; a single object to represent our measurement that keeps the full resolution measure-- taken by our sensor.

And notice that we've moved those redundant pieces of information that were previously on the individual quantity samples, so we only store a single copy at the quantity series sample level. We all want to be respectful of our users' device storage and device performance and this representation allows us to do that. Because a quantity series sample is a quantity sample, we need a way to represent the sequence of quantities with a single value and in HealthKit we do that through a technique called aggregation.

We have two primary aggregation styles; cumulative and discrete. The cumulative aggregation style applies to some very common quantity types, like distances, calories, and steps. All of these quantity types are constantly being accumulated by our users who are taking more steps and burning more calories and moving more distance and a natural way to accumulate-- to aggregate multiple quantities for these types is with a sum.

On the other hand, we have some other common quantity types for which a sum doesn't make sense. For example, heart rate, body mass, and height won't mean anything when summed together. If you take multiple heart rates over a day and add them up, you'll get a nonsense value. It's much more natural to aggregate quantity types like this with an average and maybe some other aggregating statistics, like minimum, maximum, and most recent value. Again, we can see that a sequence of cumulative quantities can be represented and aggregated by a sum and a series of discrete quantities like heart rates are going to be aggregated to produce some rising statistics, like minimum, maximum, average, and most recent value. In HealthKit, we use quantity aggregation style to tell us the aggregation style for a particular quantity type. And in iOS 13, we've decided to deprecate the discrete aggregation style in favor of a new discrete arithmetic aggregation style. We've done this to make clear that the average calculated for this aggregation style is the simple arithmetic mean. We've also introduced a couple of new aggregation styles, starting with discreteTemporallyWeighted, a special aggregation style that uses a time-weighted average that we apply when aggregating heart rate quantities.

We've also added discreteEquivalent ContinuousLevel, a special aggregation style that gets applied to audio exposure quantities. Divya's going to go into more detail about audio exposure later on. Now that we know how to aggregate multiple quantities to produce some rising statistics, we need somewhere to store this information on our quantity sample. And we've done this in iOS 13 by introducing two new quantity sample subclasses, starting with cumulative quantity sample, which has a sum property, and discrete quantity sample, which has a average, minimum, maximum, and most recent quantity properties.

We've also made quantity sample an abstract base class, which means that from now on all instances of quantity sample that you interact with will be one of these two quantity sample subclasses, depending on the quantity type's aggregation style.

And also I want to note that all quantity samples can be thought of as quantity series, some of which just happen to have a count of 1. We've also introduced corresponding predicate keypaths for both of these new sample types so that you can query for corresponding properties of both of these samples.

Going back to our quantity series sample example, we can see that quantity series sample can be summarized with our aggregating statistics, average, min, max, and most recent value.

Now I'd like to show you how we go about building this quantity series. We're going to start our quantity series at the time that our measurement starts. In this scenario, it's at the start of a game.

Then, we'll receive measurements from our sensor which we can insert into this quantity series and we can continue to take measurements from our sensor and insert them into the quantity series as long as the game is being played.

Finally, when we're done measuring, we can end the quantity series and get back the summarizing quantity series sample.

Now let's see what this looks like in code. Our first step, as with all HealthKit interaction is to request authorization for the data types that we plan to read and write. In this case, we're going to request authorization for heart rate.

Once we've requested authorization, we can create our quantity series sample builder with our healthStore, the data type, the start date of measurement, and an optional HK device.

Then, as long as we're receiving measurements from our sensor, we can insert those into our quantity series sample builder. And, finally, at the end of the game, we can finish our quantity series sample builder with optional metadata and the end date for our measurement. Now that we've thought about how we can efficiently represent quantity data and save it to HealthKit as a quantity series, we can think about the experiences we can bring to our app with the data once it's saved to HealthKit.

Maybe we want to give our app the ability to record calorie information in addition to heart rate so that we can show our users the total calories that they burned in some time interval or just present some amazing graphs and charts that highlight and help them visualize their data. Or we want to point out averages that have been recorded over a time period. Or help users see the minimum and maximum value that they hit during a particular game.

Or we just want to keep our UI live and up to date as new data is being recorded as it's received from our sensor. We can do all of this and more with a single HealthKit query.

HKStatisticsCollectionQuery.

Many of you are already familiar with StatisticsCollectionQuery, but it's pretty amazing, so I just wanted to reiterate what it's capable of. StatisticsCollectionQuery can help you calculate multiple statistics that can be separated by the source of data and you can receive updates to those statistics as new data is saved to HealthKit.

And if that wasn't enough, we've also updated StatisticsCollectionQuery to support all of our new aggregation styles and to automatically include all of the backing quantity data that is stored in quantity series samples.

If you want more information on StatisticsCollectionQuery and HKStatistics, you can always refer back to Introducing HealthKit. Even though StatisticsCollectionQuery should be our go-to tool to efficiently interact with quantity data that's been saved to HealthKit, sometimes we really do want to enumerate every single quantity that are stored in quantity series. For example, if we want to do something like plot every single heart rate that was measured by our application while a user played the game.

We can do this with QuantitySeriesSampleQuery.

In iOS 13, we've enhanced QuantitySeriesSampleQuery by replacing the quantity sample argument with a quantity type in NSPredicate.

Now, instead of simply enumerating the quantities for a single quantity sample, you can enumerate multiple quantity samples and all of their backing quantity data.

We've also updated the date parameter with a date interval, so you get the full date interval for every single quantity while it's being enumerated. And we also have an optional quantity sample parameter which can give you the quantity series sample that is currently being enumerated. This query is best explained visually, so I'd like to show you this. Here we have two quantity series samples. The first quantity series sample consists of quantities that were measured with the heart sensor built into our video game controller.

And the second series consists of quantities measured on Apple Watch. Because our user happened to be using both devices simultaneously, these series overlap each other in time. By default, the QuantitySeriesSampleQuery is going to enumerate all of the quantities from both of these quantity series in the quantity's start date order. Like this. And note that at the end of the quantity series, the done parameter is set to true. If you need access to properties on the quantity series samples themselves, like device information, source, revision, or metadata, you can set include sample to true on the query before executing it. And notice that, in this case, while enumerating these quantity series you can get the corresponding quantity series sample in the quantities handler.

Now that we've seen how we can efficiently interact with quantity data using quantity series, I'd like to show you how easy it is to add support for this to the game that we've been talking about in this presentation. Follow me over to Xcode and I'll show you a brief demo.

What I have here is our game in its current form. I don't actually have a heart rate sensor that's been built into a video game controller, but if any of you want to make one, come find me after. I do have another external heart sensor that I can use to get information into this app. And remember the quantity series can be applied to any quantity type, not just heart rate, and the data can come from an external sensor, an external database, a file on disk, or just be directly entered by your app. In this game, we have two tabs.

First, the Play tab brings up the PlayViewController where we can play our game. And the second tab, the Last Game tab, brings up the ResultsViewController, which will show us the heart rates that were measured during the most recently played game.

I'm going to go ahead and switch back to the PlayViewController to show you what a round of the game looks like. In this game, I'm trying to click these hot dogs and not hit the broccoli. Might seem a bit backwards for a health app, but this is what I have.

Every hot dog that I get scores another point and my heart rate has a tendency to increase while I play. If I switch back to the Last Game tab, I can see we don't yet have any heart rate data being measured while I played that last game. We'd like to add that support to our app now.

Our first step is going to have-- be to have our app request authorization, just like we saw in the presentation. And because our app is all about writing and reading heart rate data while users play, I want to do this request for authorization as soon as my app finishes launching, so I'm going to add this in application didFinishLaunchingWithOptions in my app delegate.

Here I'm requesting-- I'm creating a set of quantity types specifically for heart rate so that I can request authorization within HK healthStore for these sample types. After we request authorization, we can get measurements from our sensor and save that data to a quantity series. I've already created a class that encapsulates the connection to my external sensor and, if you want to see how that works, you can always check out the sample project associated with this session. I'm going to switch over to the PlayViewController now because we want to add the connection to record heart rate while the user plays a game. I'm going to open the project navigator and switch over to our PlayViewController.

In the PlayViewController, I have a function, startNewGame, that's called whenever that Start Game button is pressed in our app. This is a great place for us to create a connection to our external heart sensor and start our quantity series-- our quantity series sample builder. First, I'm going to create the connection to my sensor.

This HeartSensorSession is the class that I mentioned earlier that I wrote that encapsulates the connection to my sensor and I'm setting the PlayViewController itself as the delegate of the HeartSensorSession.

Now I can start my quantity series sample builder for heart rate.

Here I'm creating a quantity series sample builder with my healthStore for the heart rate type where the start date of measurement is the current date and I'm getting my HK device information from the HeartSensorSession.

The HeartSensorSession communicates to the PlayViewController through its delegate protocol, HeartSensorSessionDelegate, which has these two methods; sessionDidReceiveHeartRate and sessionDidEnd.

With sessionDidEnd-- DidReceiveHeartRate, HeartSensorSession provides heart rate measurements to its delegate and this is a great place for us to take these measurements and insert them into our quantity series sample builder.

Here I'm inserting the heart rates and date intervals that I've received from the HeartSensorSession into my quantity series sample builder.

And, finally, sessionDidEnd is called by the HeartSensorSession when a HeartSensorSession ends at the end of a game. This is the perfect place for us to end our quantity series sample builder to persist the heart rate quantities that we've inserted to HealthKit. And we can also associate the quantity series that we've created with metadata. In this case, I'd like to use metadata to associate these heart rates with the game session that we just played, so that in the ResultsViewController we can query for the heart data associated with the most recent game.

Here I'm creating a metadata dictionary using the MetadataKeyExternalUUID, which I am setting to the heart sensor's UUID string. And finally, I am finishing my quantity series with that metadata and the end date of measurement that we receive from the HeartSensorSession. Now that we've saved our heart rate data to HealthKit, we want to-- we want to display that data to the users in the ResultsViewController. So, let's move over to the ResultsViewController now.

In the ResultsViewController viewDidLoad function, I have a utility function, loadHeartRateQuanitites, that's going to query for the heart rate data associated with the most recent game session and generate a string representation of each of these quantities that can be displayed in a table view.

Let's go ahead and start this by adding a quantity series sample query for heart rate associated with our most recent game, if we have one.

Here I'm guarding against the case that no game has yet been played.

If we have a game identifier, we can go ahead and create a query predicate for the MetadataExternalKeyUUID using that game identifier. Remember, we use this metadata so that we would later be able to query for the heart rate data associated with our most recent game.

Then, I can create my quantity series sample query for the heart rate type using the predicate we created above.

Inside of the query's handler, I'm going to be enumerating the quantities and date intervals associated with this-- with this latest game and I'm going to create a string representation for each of them. Let's go ahead and create an array of strings we can use to store these result strings.

And then we can use this in our query handler. First, I'm going to guard against errors that might have been returned during enumeration. And if I don't have any errors, I can go ahead and create a heartRateDetailsStrings using the enumerated quantities and date intervals and I'll append these detail strings to my heartRateStrings array.

Finally, when enumeration is complete, the done parameter will be set to true and this is the perfect place for me to dispatch back to the main queue to reload our ResultsViewController to display these strings.

Now I'd like to re-run the app to show you what this looks like now that we've added support for writing and reading heart rate data.

Remember, the first thing we're going to see when our app launches is that request for authorization. I'm going to go ahead and turn on authorization to read and write heart rate data.

And now I'll play another round of our game.

Remember, I'm trying to hit the hot dogs and miss the broccoli. Oh, scored zero. Let me play another round.

Oh, scored two, but we can still see what my heart rate looked like. There we go. We have some heart rates that were measured during this most recent game.

What we just saw was how easy it is to efficiently save quantity data with a quantity series sample builder and how we can query for that data using QuantitySeriesSampleQuery. Now I'd like to invite up my teammate Divya, who is going to show you how HealthKit has expanded our representations in the area of heart and how we've moved into the brand-new health domain of hearing health. Thank you. So, my colleague Luke just described to you the new efficient way to store HK quantities now in iOS 13 and HealthKit has become a repository for more and more of our users' daily health data. And in addition to storing more and more data, we're also storing more kinds of data. This year, we're adding on to our existing support for heart health and adding new support for hearing health. Our users have been interacting with heart-related features since iOS 8, where they've been able to get heart rate from Apple Watch, or from sensors connected to apps created by developers like you, and view that data all together in the Health app.

HealthKit has always had heart rate support, so if you had a heart rate reading like the one here, you could determine that over a five-second period that the average heart rate was 68 beats per minute. And in HealthKit, you would save that as an HK quantity sample. In iOS 11, we introduced heart rate variability SDNN. Heart rate variability is a measurement of the variation in the time intervals between heartbeats and SDNN stands for standard deviation normal to normal. So, Apple Watch would take the same heart rate reading and take the time intervals between each beat to calculate a standard deviation and save that as a quantity sample to HealthKit.

Heart rate and heart rate variability are both important metrics for cardiovascular health and our users have loved looking at these at a glance throughout the day or in more detail, like when they do a workout.

So, let's take a look at this same heart rate reading one more time. Thus far, I've described to you ways in which we can summarize this heart rate data, but sometimes you want the actual underlying data itself. So, let's say that I want the time-- to know the times at which each heartbeat occurred. And returning to our scenario, let's say that our game controller has a sensor capable of telling us any time a new beat comes in while the user is playing a game. So here, the first heartbeat occurs at 0.5 seconds from the start point of data collection, the second at 1.49 seconds, and we can get-- we can continue to get the rest of the times that these beats occurred since the start of data collection. So, you'll notice that each beat happens at a certain point of time and put together they form a series of heartbeats. To save this data to HealthKit, we have an HKHeartbeatSeriesSample that stores a series of heartbeats by the time stamps at which they occurred.

Now, you might notice that this feels similar to the series API that Luke showed to you before, but it's important to note that heartbeat series samples encapsulate a type of data that deviates from our other sample types in HealthKit. There are no values or units like the HK quantities that are the underlying data behind a quantity series, and therefore we can more efficiently just store a series of timestamps to represent a heartbeat series. But because we're still storing a series of data that could get potentially large, we have in-- we have designed the API to be familiar to the quantity series, so we've equipped this HeartbeatSeriesSample with its own builder and custom query. Like quantity series samples, heartbeat series samples get created with a builder and finished when you're done saving data. So, let's build one in code.

Our first step, as always, is to request the proper authorizations. For this, you'll need to request the new heartbeat series type as well as the quantity type heartRateVariabilitySDNN introduced back in iOS 11. You'll need to request authorization for both of these types since heart rate variability is a metric that can be directly derived from heartbeat series and this way your users have a clear understanding of exactly the kind of data they're sharing with you. Once we've requested authorization, we can initialize a heartbeat series builder with a healthStore, gameDevice, and the gameStartDate that will indicate when we're starting data collection. And while the game is ongoing and our user is playing the game, we'll add heartbeats with the time interval since the series start date to our builder. But you might enter a case like this where your sensor goes down and here we have a gap in data collection between seconds two and three. Now, it might look like there is a 1.99 second gap between beats two and three that could lead to erroneous interpretation of this user's heart data.

In order to account for this, we'll set precededByGap to YES, which you can set for each beat that you add if you're aware that there was a gap in data collection from a sensor going down. Now I can add metadata to my builder like I would to any other HK sample. And when I'm done saving data, I'll finish the series, which will save the Heartbeat series sample to HealthKit. Now we've added support for beat-to-beat measurements to our game and we're ready to start querying for the underlying beat measurements. Like what Luke described to you before, we can interact with our normal HK queries to fetch the high-level samples and then use the custom queries to interact with the finer-grain data.

So, my first step will be to run a normal HK sample query to fetch my heartbeat series sample of interest. And once I've done that, I'll initialize a heartbeat series query with that sample, which will let me enumerate over the times for each beat. And finally, I'll execute my query. Now, heartbeat series are a powerful addition to HealthKit, but that's not all we have related to heart features. Back in iOS 12, Apple Watch started notifying users with heart alerts.

A low heart rate alert when Apple Watch detected a heart rate below a given BPM threshold, a high heart rate alert for when heart rate rose above a given BPM threshold, and an irregular heart rate alert for when Apple Watch detected a rhythm that could be indicative of AFib.

Well, in HealthKit, these alerts now come in the form of three new category types that will get saved to HealthKit any time Apple Watch detects an alert. Now, in addition to all this great heart support, there's also this new area that we're exposing in iOS 13 and I'm excited to share with you that hearing health has found its place in HealthKit. At some point in your life, you might remember getting a hearing test where you put on a pair of headphones and listened to a series of sounds and would raise your hand as soon as a single sound was loud enough for you to hear it. Well, this was an example of a pure tone hearing test, where a pure tone is a sound with a single frequency. Pure tone testing helps identify the quietest sound that you can hear at a set of different frequencies and it can provide assessment on the kind of hearing impairment or loss that you might have.

The results of pure tone testing most commonly get displayed in graphs called audiograms. This here is an example of an audiogram for someone with mild hearing impairment and let's just zoom into this graph to get a better idea of the kind of data that is stored in an audiogram. So, here you'll see two lines, one corresponding to the pure tone hearing test results from the left ear and one for the right ear. And let's just take a look at the first two data points at the 125-hertz line.

This shows that for this user to hear a sound at a 125-hertz frequency, they need around 11 decibel hearing level units in their left ear and 31 in their right. The decibel hearing level unit measures the intensity of a sound relative to the quietest sound that a young, healthy individual would be able to hear.

So, we can get the rest of the data points associated with this audiogram and in order to store this data in HealthKit, we're introducing an HKAudiogramSample that stores an array of hearing sensitivity points associated with a hearing test. So, let's create an audiogram sample and code. Our first step is to create an HKAudiogramSensitivityPoint that encapsulates the intensity of a sound required for both ears to hear a given frequency. So, I'll create a frequency quantity with my new HK unit hertz unit and a left-ear and right-ear sensitivity quantity with my new decibel hearing level unit.

Now I'm ready to create an audiogram sensitivity point. Once you've created an array of audiogram sensitivity points, you can store that in an audiogramSample. Now, you'll need to make sure that the array of sensitivity points are all unique and in order, since that's how you should expect to interact with this data later on in analysis or charting.

And finally, I'm ready to store that data to HealthKit. And that's how easy it is for you to start creating audiogram samples and to start building hearing health apps with HealthKit. But an audiogram sample only represents the health of your ears at a given point of time when you're taking a hearing test. Most of the day, we're exposed to sounds through our headphones or while we're walking down the street in the environment and all of that can impact our hearing health for life. To keep track of the audio exposure that you're exposed to through your headphones, we have a read-write quantity type, headphoneAudioExposure. And for the rest of the day, when we're walking down the street exposed to construction work sounds or traffic, Apple Watch is capable of capturing that environmental audio exposure data and saving it to HealthKit, and for that we have an analogous read-write quantity type, environmentalAudioExposure. And for times when environmental audio exposure gets too high, Apple Watch will generate an audio exposure alert to make sure that you're aware of the possible impact that can have on the health of your ears. And it will save this as a category sample using the new audioExposureEvent category type identifier. So, we've covered a lot about the new data representations now available with HealthKit and iOS 13, from new efficient series representations to the new support with hearing health. Now you can officially store large numbers of HK quantities, the most abundant kind of data stored in HealthKit. And you have the opportunity to represent even more rich data representations related to heart and hearing health.

For more information as well as our sample code projects you saw here, you can visit our session link listed or come talk to us right after this session at the Health and Fitness Technologies Lab. Thank you and have a great rest of your WWDC. [ Cheering and Applause ]