Track down hangs with Xcode and on-device detection

♪ Mellow instrumental hip-hop music ♪ ♪ Hi everyone, my name is John Crowson, and I'm a software engineer on the Performance Tools team here at Apple. In this talk, I'm excited to introduce you to several new tools to track down hangs in your app with Xcode and on-device hang detection. I'll be your guide as we visit different phases of your app development process, and consider which tools are best suited to help during each phase. This talk is broken into four sections: First, I will cover, What are hangs? Then, I will present tooling to help you discover and diagnose hangs while developing your app, while beta testing your app, and after releasing your app. Let's get started! I want to tell you about a new app my team is developing: Food Truck, which will help manage food trucks that specifically sell donuts. Let me introduce you to some of the donut types I've created.

Huh, that took a really long time to scroll through my list of donuts. The app was laggy and would not respond to any of my touches. At Apple, we call this period of unresponsiveness a "hang." An app's main thread is responsible for processing user interactions and updating the view content. A hang is reported when the main thread is busy doing work or waiting on another thread or system resource, causing a delay in updating the view content by at least 250 milliseconds. The main thread is also unavailable to process new user interactions until the hang is resolved. To the user, it appears the app is completely stuck. Creating a responsive app is critical to providing a positive user experience. Consistently unresponsive apps may result in users force quitting the app, switching to a different app, and in some cases, even deleting your app and writing a negative review. Because of this, tracking down hangs in your app is critical to gaining and maintaining your user base. Providing a responsive experience ensures people will enjoy using your app.

For more information on hangs and what causes them, as well as strategies to eliminate them from your code, checkout the "Understand and eliminate hangs from your app" talk from 2021. The app-development process can be broken down into three phases. First, developing the latest app version at desk using Xcode. Then, testing the app and collecting feedback in a beta environment without Xcode. For example, you may have an app version that's development-signed on your device or distributed through TestFlight. Finally, releasing the latest app version on the App Store. Even for the most proactive developers, new hang issues can arise at any phase, so it's important that you know the tools to resolve them during each one. Before iOS 16 and Xcode 14, we offered two tools that assisted with discovering and diagnosing the hangs in your app. MetricKit is a framework that supports collecting nonaggregated hang rate metrics and diagnostic reports from individual users on your beta or public release app. The Xcode Organizer provides an aggregated hang rate metric from users on your public release app. There are gaps here, specifically when developing your app or when trying to understand what source code has caused the public release hang-rate metric to rise. In iOS 16 and Xcode 14, we've been busy introducing several new tools to help. Let's introduce each of them briefly before we cover them in more detail. The Thread Performance Checker in Xcode alerts you to hang-causing threading issues while debugging your app without actively tracing it. Instruments in Xcode now detects and labels hangs while tracing your app. On-device hang detection provides hang detection without using Xcode or tracing, providing real-time hang notifications and supporting diagnostics while using your development-signed or TestFlight app. And finally, the organizer in Xcode now supports hang reports, which provides aggregated hang diagnostics from users in the field. Now that you know what hangs are and the different phases they can arise, I will cover how to track down hangs while developing an app with Xcode. In Xcode 14, the new Thread Performance Checker tool notifies you in the Xcode Issue Navigator when it detects priority inversions and non-UI work on the main thread of your app, both of which are common causes of hangs. I have now returned to Xcode to diagnose the hang I observed earlier in the Food Truck app when I was scrolling through the donuts I've created. When I built and ran the app, and repeated the user interaction, the Thread Performance Checker tool alerted me to a hang risk caused by a priority inversion. This means a higher priority thread was attempting to be synchronized with a lower priority thread. This may indicate the hang we are noticing is caused by the main thread waiting on different lower-priority threads. To detect priority inversions and non-UI work on the main thread of your app, enable the Thread Performance Checker tool from the Diagnostics section of the appropriate scheme. The Thread Performance Checker alert has helped me discover the potential culprit of my hang, but in order to triage further, I will want to know what the other thread was doing during the hang duration. Let's use another tool to dive deeper. The Time Profiler instrument gives you the ability to know what each thread in your app was doing over time by providing call stacks. New in Xcode 14, the Time Profiler also detects hangs and labels them directly in the corresponding process track. In the Food Truck app, I'll use the Time Profiler to confirm a hang is occurring when scrolling through my donuts, that it was caused by a priority inversion on the main thread, and figure out what the lower priority threads were busy doing that caused the main thread to wait. I start from the Product > Profile menu in Xcode. This builds the app for release and launches Instruments already setup to target the app. I launch the Time Profiler template and begin recording a trace of the problematic user interaction in the Food Truck app.

I see there is a hang being detected and labeled in the timeline. The hang duration is also specified to help evaluate the severity of the issue. Next, I can triple-click the hang interval which creates a time filter for the duration of the hang and filters the information in the detail views at the bottom to only events occurring in this selected time interval. It also makes it easier to see what was happening during this time period in other tracks. The first thing I notice is that the main thread has barely any CPU usage during the hang interval, meaning the main thread was unresponsive because it was waiting on another thread, not because it was doing too much work itself. This aligns with the Thread Performance Checker's priority inversion alert from earlier. Next, I see a worker thread that has lots of CPU usage during the hang. This is likely the thread that the main thread is waiting for. The next step would be to examine what the worker thread was doing during the hang and resolve the priority inversion. Hang detection and labeling in Instruments is a great way to surface any hangs that are encountered while profiling your app. It is available by default in the Time Profiler and CPU Profiler instruments. There is also a new standalone hang tracing instrument that you can add to any trace documents to surface hangs in combination with other instruments. In addition to hang detection and labeling, it allows you to configure a hang duration threshold to find specific periods of unresponsiveness. You have now learned how to use Xcode to discover and diagnose hangs at desk. Even with great testing coverage during development, beta and public release environments are likely to uncover hangs in code paths you hadn't considered. Next, I will introduce how to track down hangs once the app is deployed in a beta environment. I've now deployed a build of the Food Truck app to TestFlight through App Store Connect and it is downloaded on my personal device. I'll test the app when selling donuts around town, including in places where I don't always have a strong network connection. But how do I discover and diagnose hangs if my device is not connected to Xcode? To continue to monitor for hangs under these conditions, iOS 16 introduces on-device hang detection in Developer settings, which provides real-time hang notifications and supporting diagnostics. This is available for development-signed or TestFlight apps. It's time to start selling orders. When I attempt to open my current orders, I receive an on-device hang detection notification that my app is hanging, this time for over three seconds. I wonder why I didn't notice this hang when I was developing with Xcode? I will need to use the diagnostic information provided by the on-device hang detection tool to learn more. Once your app is setup for development, this feature can be enabled by opening Settings > Developer > Hang Detection, and toggling the switch. The Hang Threshold setting allows you to configure the minimum duration of hangs to detect. The shortest hang threshold is 250 milliseconds and can be bumped to 500 milliseconds or higher. Long hangs tend to have a higher user-impact, but even shorter ones can be disruptive to the experience, depending on the context, especially if they happen consecutively. After installing your app, it will appear in the list of monitored apps. The final section shows a chronological list of available logs for the hangs you were alerted to. Note that these diagnostics are best-effort and processed in the background at a low priority to minimize performance overhead. This means the processing can take longer, especially if the system is busy. Fortunately, a passive notification is displayed when new diagnostics become available. Let's examine the diagnostics for the hang that was detected when I was opening the orders in the app while selling donuts around town. I am given both a text-based hang log and a tailspin for the detected hang. A text-based hang log has less information, but can give us an understanding of the hang at a glance. For a deeper investigation, open the tailspin in Instruments for viewing the thread interaction within your process or identifying the usage of system resources, for example. To start, I'll use the Share button to send the text-based hang log to a Mac, where I can symbolicate it and view it on a larger screen. From viewing an excerpt of the text-based hang log I transferred and symbolicated, I see that during the hang, I am calling a method on the main thread that I know performs synchronous requests to the network. When I am testing the application at my desk with Xcode and a strong network connection, there may not be any delay when requesting data from the network. However, when testing the app in places with a limited network connection, the request takes longer and results in a hang. It is important to test the beta version of your app under these different, real-world conditions, and on-device hang detection allows you to monitor for hangs using just your device. At this point, I have discovered and diagnosed hangs using the available tooling in the development and beta phases, and I am ready to make the Food Truck app available to customers in the App Store. I'll now present how to track down hangs once your app is in the hands of your customers, on various OS versions, devices, and in other real-world conditions you may not have been able to replicate in your prior testing. New in Xcode 14, the Xcode Organizer supports hang reports to deliver aggregated hang diagnostics from customer devices. The collected data is from customers which have consented to share app analytics, and they contain information about the main thread stack trace that led to the hangs. Hang reports are available from the left-side navigation of the Xcode Organizer. When similar stack traces are collected, they are grouped together to form a single signature. In the list, the signatures are shown sorted based on the user impact. For each signature, you can find a few sample hang logs. Each hang log contains the main thread stack trace containing the code responsible for the hang, the hang duration, and the device and OS version from which the log originated. Each signature also provides aggregate statistics about how many hang logs the signature was responsible for and a breakdown of those logs by OS Version and device. To identify the hangs most affecting your customers, pay close attention to your top signatures. In this case, the top signature is responsible for 21 percent of the hang time in this release. Since I've submitted the app to the App Store with symbol information, the hangs report shows me all of the functions named just as they are in the source code. By examining the functions in this main thread's call stack, I can infer that this hang was caused by synchronously reading a file from disk on the main thread, which is known to block for significant periods of time. It is important to tackle the performance problems that are most affecting your customers, and the Organizer is a great tool to identify them. Check this data after each app release to validate the previous hangs have been resolved, and to address new hangs that may appear.

You can also retrieve the same hang report data through the App Store Connect REST APIs. This can help you integrate performance data with your own systems, or run additional analysis. I highly recommend you check out the "Identify trends with the Power and Performance API" video to learn more about using the Power and Performance APIs. Added in Xcode 13.2, you can now receive notifications when monitoring power and performance metrics in your app. I recommend you enable notifications by clicking the Notifications button in the top right of the Xcode Organizer's Regressions view. This will alert you to sudden rises in your app's hang rate. Find out more about performance regressions in the "Diagnose Power and Performance Regressions in your app" talk from 2021. To improve your experience in the Xcode Organizer, I strongly recommend you build and submit your app to the App Store with symbol information. This symbol information is used to add function names from your app to reports in the Xcode Organizer. This makes stack traces significantly easier to understand. It also enables one-click navigation from a function name in a stack trace to the function definition in the Xcode source editor. The information extracted is limited to function and methods, names and paths of source code files, and line number information. It is important to note that the limited symbol information will be securely stored and will never be shared. Fantastic! You now know how to discover and diagnose hangs at each phase in the development process. Going forward, discover, diagnose, and fix hangs in the earliest possible phase of the development process. Use the tools available to help, including proactively profiling new features using Instruments. Be sure to enable the Thread Performance Checker and on-device hang detection. After each release, use the Xcode Organizer to tackle hangs that are most affecting your customers and to validate that hangs from previous app versions have been resolved. Enable regression notifications to be proactively alerted to regressed performance metrics, which can be the first sign of power and performance problems. And finally, build and submit your app to the App Store with symbol information to improve the usefulness of the Xcode Organizer. By following these steps, your apps will have even better performance to provide the best possible user experience. Thanks for hanging out at WWDC! ♪