Beyond counting steps

Hello and welcome to WWDC.

Hello folks! My name is Edith and I'm a Motion Scientist on Apple's Software Engineering team. Today, I'm joined by Dan Trietsch, a Motion Engineer, to introduce a new set of metrics - now available on iOS and watchOS - that enable customers to measure their movement in groundbreaking ways. In this session we'll introduce these metrics, and provide guidance for app developers who want to go beyond simple step-counting to create a rich understanding of a customer's mobility. Movement is a key indicator of health. The way someone moves can tell us about their short-term health, like recovery from a musculoskeletal injury, and their long term health, like their ability to age with independence. Movement is affected by a variety of health conditions like neurological disease, and cardiopulmonary fitness. One of the primary ways that we measure movement is observing how someone walks. Walking is something many people do every day and might take for granted, but it's actually pretty amazing how many systems have to come together to do it.

Walking requires coordination, strength, and fitness. Our muscles, brain, and heart are constantly working when we do it. Measuring walking is a great way to see changes in health, but measuring how someone walks is surprisingly complicated.

It requires trips to in-clinic appointments and extensive evaluation with specialized equipment. This is a hallway, and it's a setting for one of the most used clinical walking tests - the six minute walk test. Patients are asked to a back and forth, to go as far as they can in six minutes. The faster and more continuously they walk, the higher the number. The total distance covered is a great indicator in a wide range of health scenarios.

But to do this test properly, it requires 30 meters of clear hallway to make sure the person isn't spending more time turning around than they are walking. How many of us have this kind of space at home to check in on our progress between clinic visits? Questionnaires are less time and resource-intensive tool to evaluate walking. Asking a person to report their level of difficulty or pain when walking is helpful, but questionnaires rely on memory and perception. There ARE systems for measuring movement much more precisely. Motion-capture technology is a very precise method to detect small changes in a person's movement, but it requires expensive equipment. Specialized cameras and markers like you can see here - and trained technicians to gather, process and analyze the data. Overall, even the best of these approaches is only a rare snapshot. It's a substitute for the question that really matters: how are you moving in your everyday life? iPhone and Apple Watch, devices people carry every day, can now bring the motion lab into the real world, filling the gap between research and life.

New to iOS and watchOS, Apple is introducing a category of metrics to HealthKit that capture the complex and important elements of human movement, all while a customer walks around with their iPhone in their pocket (or their Apple Watch on their wrist). Customers can view these metrics in the Health App, and Developers can use them in their apps through the HealthKit API.

Four of the new Mobility Metrics are calculated on iPhone. The first two metrics are: walking speed, how fast a person travels over ground, AND...step length, is a person taking short, cautious steps, or long confident strides? These metrics are sensitive to slow walking and can detect small changes - tasks that traditional pedometers struggle with. Two other iPhone metrics measure walking quality: doubles support time, which is the percentage of time during a walk when both feet are in contact with the ground, and...walking asymmetry, which is the percentage of time that steps with one foot are faster or slower than steps with the other foot. While step counters and pedometers measure ALL walking in a day, these new metrics intelligently gather the most valuable bouts of walking during consistent conditions, over flat ground in a continuous segment, resulting in precise measures that resemble standardized walking tests. The other four Mobility Metrics are calculated only for Apple Watch wearers. In addition to counting flights of stairs, Apple Watch can now record how fast a person goes up or down a flight of stairs. Two features synthesize together multiple data sources to make predictions of what a person could do at their highest effort. First, remember that standardized walking test I mentioned? The six minute walk distance? Apple Watch now uses activity clues that happen naturally over a week to estimate what a person would have scored on that test. And finally, VO2Max is not new but now it's more accurate and it's better for people with lower levels of fitness. Instead of requiring running level effort, Apple Watch can now predict VO2Max using slower walking speeds. These metrics build on APIs that measure movement with iPhone and Apple Watch, like CMPedometer which is used to track steps, cadence, distance, pace, and flights of stairs, and HealthKit which aggregates these metrics to add calories, exercise minutes, and workouts from Apple Watch.

You can find more resources on these technologies in WWDC talks from 2017 and 2018.

That's a lot of new information and there are so many possible applications for this new collection of Mobility Metrics but today, we'll focus in on one scenario where these data could be helpful - recovery from a musculoskeletal injury.

We'll show you how to access and use these data types to create a rich user experience. Imagine you have a customer named Jamie, who just injured their ankle while training for a 5K. While Jamie is still able to walk around and complete daily tasks, they want to track progress during rehab and recovery...and how far there is to go. Recovering from an injury is often a slow process, and it can be hard to recognize and celebrate small increments of progress to stay motivated in between visits to the doctor and physical therapist.

What can you do to help fill in the gaps? In the next few slides we'll use the mobility metrics to show Jamie their progress and help them put their recovery in context. For details about sinking data with HealthKit or making apps in CareKit, check out these past WWDC sessions. We're going to break today's work up into three steps. First, how to measure walking by querying one of the new mobility data types. Second, how to aggregate that data to see changes over time. And finally, how to bring in other data types to build and interpret a holistic picture of walking. First, what should you measure? You'll start by showing Jamie their walking speed using the new HealthKit quantity type. There are a lot of pedometers out there already, including one on iPhone and Apple Watch, that measure walking quantity.

They're optimized to count as much walking as possible. But this measure is designed to capture small changes in how they're doing that walking, so it works a little differently. It doesn't need calibration against GPS, making it consistent and easy to compare values over time. It's accurate at slow walking speeds, so it works for people with limited mobility and can capture even those early days of recovery. It's sensitive to even small changes.

By design it doesn't count all the walking. Instead, you'll see a start and stop time for a samples of continuous walking on flat ground, which makes walking bouts easier to compare. And because it's passively collected on the motion coprocessor all day, any time Jamie carries their iPhone in a pocket or on their hip, they can have measurements dating back to before the injury happened, which allows pre and post injury comparison. Now, Dan will walk through how to use this metric.

First thing's first, since walking speed is a HealthKit quantity type, we will need to ask Jamie for permission to read their data. For our app, we are not planning on writing new data back into HealthKit, so we only request read access. Next, we will construct a query for walking speed samples around the time of Jamie's recent injury. Let's assume that they have already shared with us the date of their injury. Let's set our query bounds beginning thirty days before the injury so we can establish a baseline speed, and ending sixty days later, so we can track the recovery progress. Then, we specify the type of data we are interested in, walking speed, and how we want it to be sorted.

As soon as we "execute" this query, our callback will receive all of the matching HKSamples. Be aware, depending on Jamie's activity level, this query could return hundreds of walking samples. Now that we have Jimmie's query results, we'll need a way to visualize the data. Fortunately for us, Apple has a pair of frameworks to chart and graph our results, CareKit and ResearchKit. In today's example, we'll be using ResearchKit's GraphChartViews. Specifically, ResearchKit's LineGraphChartView and DiscreteGraphChartView will provide an excellent way for us to visualize Jamie's query results over time. For each sample, we append our results to an array of ValueRange objects. Then we call a function we added, update PlotPoints, to our ChartViewDataSource object. Finally, we have our GraphChartView reload our data in the UI on the main queue. In this graph, we're presenting walking speed results in ResearchKit's DiscreteGraphChartView. That's quite a bit of data! Jamies injury date is at "day zero" on the x axis. Notice that because we're plotting Jamie's full range of walking speeds by day, it's difficult for us to know where most of the data is distributed within any particular day. However, just from this plot, we are starting to see the impact of injury on Jamie's walking speed. Edith, what's next? Alright, Dan got us started with a straightforward visualization of the raw values for walking speed. However, you could probably see that with a passively measured, all day metric like this, there's a lot of data to make sense of. The next step toward a simple but insightful experience is to aggregate the raw data into something more meaningful for the recovery process. In the first plot of all the daily data, the injury was striking, but it was harder to see the more subtle recovery. When working with "all- day-every-day" data, not all walking is going to have the same meaning. Think about the many things that could impact a person's walking speed other than their health. Maybe they're rushing to a meeting. Maybe they're walking with a dog that likes to stop and smell everything. Maybe they spend some time walking with a partner who is slower than they are. As a developer, you, and maybe some friendly domain experts, like doctors, physical therapists, or biomechanists, we'll need to find the right way for your scenario to aggregate and summarize the samples to cut through that noise. Here's a look at the data underlying the plot Dan just showed of Jamie's walking speed over time. There are many samples of walking during the day, covering a wide range of speeds with the occasional outlier. You've consulted the domain experts or your data science team, and they've identified that faster walking speeds are the most meaningful way to understand if someone is on track for tasks like hustling to make it across the street before the light turns red. So, you decide to use the ninety fifth percentile of the metric to track maximum walking speed - without being too sensitive to outliers. And since this recovery process is going to be a matter of weeks, not days, your product team doesn't want Jamie to be too concerned about the small day to day fluctuations, so you decide to average over seven days. But, of course, you don't want to blend the pre and post injury values together...so you reset the calculation after the date of injury. Now, Dan will walk through how to do the same calculation directly in your app. Thanks, Edith. As you recall, we had previously queried all of Jamie's walking speeds from thirty days before the injury up to sixty days after. As Edith demonstrated, in order to measure maximum walking speed, we will want to calculate and display the top ninety-fifth percentile speed per day. Let's pick midnight as the starting point and ending point for each day. We will call "enumerateDates" to iterate over each day and find all the walking speed samples contained within.

Notice that my output array is made up of "optional" values. We cannot assume that Jamie will have a walking speed for every single day in the range of our query. Perhaps they left their phone on their bedside table, or maybe they just weren't very active that day. But, for the days when they do have sufficient data, we can calculate their ninety-fifth percentile walking speed. First, we extract all the walking speed values from HKQuantitySamples. Since we have been using miles per hour in our plots, let's select that for the values we are calculating. And just to finish it up, let's sort the list and find the ninety-fifth percentile walking speed. Wow! Thanks to the analysis of our scientists, and our code implementation, Jaime can really see the impact their injury had and how they came back stronger and faster than before! Edith, back to you! Now that you've aggregated walking speed to create a meaningful chart, let's talk about how to help Jamie interpret the big picture. Filtering all that data down to something more digestible will reveal daily and weekly progress. But recovery from an injury can take a long time, and there will be other factors that impact daily walking speeds. They could be health-related like: decreased pain, or dropping off from physical therapy, or taking on a new exercise program. And, non-health related, like: a brisk walking commute that has become a morning stroll while working from home, or a family vacation that means spending several days walking with an older relative, or maybe it's just too hot out to walk very fast. Luckily, Apple's Mobility Metrics go further to capture aspects of walking quality. By combining multiple Mobility Metrics together, you can create the whole picture of recovery. We have a good understanding of how Jamie's walking speed has changed before and after their injury. Now, let's take a look at their DoubleSupportTime.

This is the percentage of time during a walk when both feet are in contact with the ground, and a lower value is actually better because it indicates that the walker is spending less time using both feet to stabilize themselves. You can show Jamie that, even though their speed slowed down during a heat wave, when they didn't feel like going on their brisk afternoon walk... they continued to make steady improvements in lowering their double support time back to pre-injury levels. And it doesn't stop with this rich record of daily walking. Recall the occasional six minute walk tests that are done in a physical therapy clinic to provide snapshots of the recovery process.

They might not be done very often in a busy clinic, and they can't tell us what Jamie's baseline was before their injury. But because Jamie wears an Apple Watch, the movement signals from Apple Watch and iPhone are combined into a predicted six minute walk distance each week. These values give insight into what Jamie's movement was like before the injury and the recovery process in between physical therapy appointments. Dan, what do developers need to do to bring in these added dimensions - like Double Support Time and Six-Minute Walk test to the app? Let's add Double Support Time in Six-Minute Walk Distance to the list of types we'd like to request. We query Double Support Time and Six-Minute Walk and in much the same way as we do Walking Speed. Now is a good time to also filter the data according to how it was collected. As you know, data stored in HealthKit can come from a variety of different sources. For example, Six-Minute walk data could be predicted values calculated on Apple Watch or user-entered values in the Health App. For Six-Minute Walk graph, let's aggregate both the predicted values and the value Jamie had manually entered after completing a Six Minute Walk test in clinic. We can accomplish this by checking the SourceRevision and Metadata of the HKSample. First, we check the source's bundle identifier to see if the data was recorded by Apple. Then, we check the metadata key for the "WasUserEntered" key. If this key is both present and set to "true", then we will know that it was a sample entered by Jamie into the Health App. Otherwise, it is a predicted value from Apple Watch. With these newly added dimensions, Jamie gets more than just the walking speed plot we made earlier. They can see how their percentage of double support increased after their injury and steadily decreased during recovery. Typically, this type of data would have required specialized motion lab. Now, these metrics are collected right in their iPhone. Let's graph two sets of data points for Jamie's Six-Minute Walk distance. First, in red, we plot values recorded from a test at a physical therapy clinic and manually entered in the Health app. Second, in green, we plot predicted Six-Minute Walk values, generated by iPhone and Apple Watch. The automatic, in-system values give Jamie a pre-injury baseline for comparison and paints a fuller picture of recovery. Back to Edith! Thanks, Dan! Developers, you've helped Jamie own recovery from that injury by measuring, aggregating, and interpreting changes in mobility over time. In today's scenario, we've gone deep on the powerful new Mobility Metrics available for iPhone and Apple Watch. There are already some great examples of groups exploring Mobility Metrics for clinical use cases. The Society of Vascular Surgery will use these mobility metrics to monitor patients with peripheral vascular disease as they work through a home exercise program.

And Zimmer Biomet a leader in orthopedic devices and care will use them in mymobility, an app that helps patients and their doctors prepare for and manage knee replacement surgery. I want to close with a few final pieces of advice. Apple devices offer rich opportunities for motion sensing, in a variety of APIs, that round out the ways to measure movement. You can access inertial sensing signals directly on the iPhone and Apple Watch using CMMotionManager. You can track changes in altitude using CMAltimeter. And, new this year, you can now get those same useful inertial sensor signals, in a new location, Air Pod Pros. with the headphone motion API. Second, Mobility Metrics have useful metadata, like the start and end times for each walking bout, or the calibration status of a predicted six minute walking distance sample. So check those out when considering how to aggregate the data. Next, remember that the mobility metrics require particular conditions from measurements to be recorded. The iPhone values are recorded when the user carries their phone near their waist - such as in a pocket - and walks on flat ground. Six minute walk distance is recorded with a week of Apple Watch wear.

And finally, work with domain experts like physical therapists, doctors, and biomechanics to develop the strategies for aggregating these metrics to find the insights and meaning that suit your use case.

There are so many health scenarios beyond acute injury when movement will change. We're excited for all the creative applications you find for Mobility Metrics, and we can't wait to see what you come up with. Thanks for being a part of WWDC 2020.