Get your test results faster

♪ Voiceover: Hello, and welcome to WWDC.

Sean Olszewski: Hi there, and welcome to my session: Get Your Test Results Faster.

My name is Sean, and I'm an engineer in the Developer Technologies Group at Apple, working on the XC Test Framework and its Xcode integrations.

This session is grounded in a concept called the Testing Feedback Loop, and if case this concept is new to you, I wanted to briefly take you through it.

After this, we'll be going over some techniques and features within Xcode that you can use to speed up getting results from your tests.

If you've ever written an automated test before, there's a good chance that you have an intuition of what the testing feedback loop is, and this is because writing a test is the beginning of the loop.

The loop then continues into you running those tests, usually alongside some other tests, and then ends up with you interpreting the results and making some decisions based on what's in your test report.

Depending on your report, you may decide to write more tests because there are cases or features you're interested in covering.

Or you may decide that you've written enough tests, and the results give you confidence your code behaves as you expect.

This confidence lets you move on to other tasks.

Having short feedback loops is important because that means you get results from your tests faster.

If your tests are faster, than you can get confidence in your code is faster, which that means you can ship features to your users faster.

Now, having gone over the Test Feedback Loop, I wanted to let you know what will be going on for today.

We're first going to discuss some features in Xcode 12 that will ensure your tests always complete.

We'll also talk about how to use some diagnostics these features surface to figure out what may be breaking your feedback loops.

Then we're going to talk about how you can get even faster results from your tests by using the Test Paralyzation Options available to you in Xcode 12.

Now let's ground this discussion in a real world example.

Imagine you're at work looking at the results from a CI job that you kicked off on Friday before leaving the office.

With a hot beverage in hand, tou realize that your long-running test suite never finished.

You'd probably feel like how this person looks: frustrated and a little upset that you need to start the week understanding why your tests aren't working.

If we don't investigate this problem, our tests will continue to take longer and will intermittently not finish, which is going to ruin our confidence in our tests and application code, and that's going to hurt our ability to quickly deliver new features to our users.

This is a very unfortunate situation to be in.

But lucky for us, there are some features in Xcode 12 to help us out.

So let's look into it.

We're going to start with fixing the hang in our test suite so we can always get feedback from our tests.

Right now, the feedback loop is broken.

Because our tests hung and never finished running, we never got our results, and so we can't interpret them.

We're left to cancel the tests and forfeit getting a complete understanding of our codebase's codebase's quality.

This image is from a result bundle from that CI job, which never finished.

It has an error message that says testing was canceled because we had to cancel the CI job, which isn't exactly actionable.

I'm left wondering what exactly went wrong in the first place.

Without much in terms of diagnostics to understand why our tests are hung, we can try thinking of a few causes offhand.

A classic example is a deadlock, where two sections of code are waiting for the other to make forward progress and therefore neither does.

Though even if our tests aren't stalled, their rate of progress may be so low that they're effectively stuck.

Alternatively, this could just be due to poorly chosen timeout values in some application code.

Or it could be due to large amounts of CPU work that we're doing on the main thread of our app or framework for testing.

Available in Xcode 12 though, is a solution to our problem of hung tests.

It's a new test plan option called Execution Time Allowance.

Execution Time Allowance is a customizable feature that you can opt into when running your tests.

When enabled, Xcode enforces a limit on the amount of time each individual test can take.

When a test exceeds this limit, Xcode will first capture spindump, then kill the test that hung, then, restart the test runner, so that the rest of the suite can execute.

We know it's not a great experience to try to guess at what could be causing our tests to hang.

After all, our code base could be large and complex, and that doesn't lend itself to being easily reasoned about.

Instead, we'd benefit from having some better diagnostics given to us, so we can understand the cause a bit better.

A spin dump can help us out here a lot.

And this is why Execution Time Allowance attaches them to your test report.

A spin dump shows you which functions each thread is spending the most time in.

If our tests are stalled, a spin dump would help us see what functions the issue may lie in.

It's also possible to manually capture spin dump from Terminal using the spin dump command or from within Activity Monitor if you prefer a GUI.

By default, each and every test will get 10 minutes.

If a test successfully finishes before that 10 minutes elapses, the timer will get reset for the next test.

If you need more time for all tests, you can customize the default allowance in your test plan's configuration.

And if you need more time for a specific test or test class, you can use the executionTimeAllowance API to special case a particular test or subclass.

Execution time allowance is represented as a time interval property on XCTestCase.

It's important to note that time allowance values will be rounded to the nearest minute.

For values under 60 seconds, they'll be rounded up to 60 seconds, and for a value like 100 seconds, it would be rounded up to 120 since that's the nearest whole minute.

Having gone over the new Execution Time Allowance feature let's go through a quick demo of turning it on and using the spin dump it attaches to fix our hung test.

I have here in Xcode 12 the test that was hanging in CI.

It's a test called TestUpdatingSmoothiesFromServer and it's a test of a method called FetchSynchronouslyFromServer.

I'm going to try to reproduce the issue at my desk here, and I'm going to do that by pressing the play button in the Source Editor Gutter.

Now when I do this, I see the activity indicator in the test navigator spinning.

If this test was working and not hung, it would execute immediately.

I'm going to stop the tests because there's no use in waiting for it to stall.

Now, I want to turn on the Execution Time Allowance feature to get a spin dump, so let's do that.

I can do that by opening the test plan menu, clicking the edit plan item, selecting configurations, turning test time-outs on, and then rerunning my test by pressing the Play button again in the Test Navigator.

This will generate a new report for me that I can use, and I can view that report in the Navigator.

If I go ahead, and I open up the report that it just generated, I'm going to see the same test is failing, but its failure reason is different.

It's going to say that it exceeded the test execution time allowance of 10 minutes, which is the default.

And it's also going to attach a spin dump.

You can open that spin dump by double-clicking it, and it will open inside an editor tab.

Spin dumps are generally broken up into two sections: a preamble, which contains metadata, and then a series of stack traces for each thread within the process that was sampled.

Since we're sampling our test runner process, I know that my test name should be somewhere in that spin dump If I do a quick find, I can rapidly find my test within the stack trace and see that it's calling the method under test as well as a private helper method.

After that, I can see that it's acquiring a lock, and then it's waiting.

This suggests to me that the issue is in that helper method that we have.

I'm going to look at the code, and I'm going to navigate to the code by opening up the Smoothy.Swift file.

Upon looking at this code, I can see that in this second method, it's acquiring the same lock as our method under test: FetchSynchronouslyFromServer.

It seems questionable to me that this helper method, which is just for performing a get request, is acquiring a lock, and so I'm going to try deleting this lock acquisition code from here to see if that fixes our deadlock.

I'm going to reopen the test navigator by selecting the test navigator icon and clicking the Play button for our tests.

We see that the test immediately executes, indicating that we fixed the hang.

Having demoed turning on time allowances for our project, let's talk about some ways you can customize them.

There are two ways you can customize the default time allowance.

The first is using the Test plan setting, which is available in Xcode 12.

And the second is using the Xcodebuild option.

Once you've enabled Time Allowances, there's a precedence order the configurations follow.

This is so that you can set course grain defaults and finer grain values for special cases such as CI jobs or long-running tests suites.

The TimeAllowance API has the highest precedence.

While Xcodebuilds TimeAllowance option has the second-highest precedence.

A Test Plan setting has the third-highest precedence.

And the the system default of 10 minutes, has the lowest precedence, and will be overridden by any of the other three options.

With all of these ways to set a time allowance, a question emerges which is: What happens if a test requests unlimited time? There's a way to prevent this from happening, and that's by enforcing a maximum allowance.

Your test is guaranteed not to exceed this value, regardless of the configuration you set and test plans or through API calls.

You can enforce a maximum allowance either via a setting in the Test Plan or through an Xcodebuild option.

Having gone over how to use the new Time Allowance features, we wanted to offer a couple of recommendations for how to get the most out of them.

For starters, use time allowances specifically to guard against test hangs and ensure you get diagnostics when they do.

If you're concerned about keeping your tests fast, we recommend using XCTest's performance APIs to automate testing for regressions in the performance of your code.

And if you need to identify what parts of your code are slow we recommend using Instruments to profile and understand your app's performance.

Instruments provides a rich set of tools that will give you a lot of info that can help you figure out where to begin adding perf tests to your app code.

If you're interested in learning how to use Instruments, check out this talk from WWDC 2019 entitled Getting Started with Instruments.

Now, having adopted Time Allowances, our feedback loop has gone from being broken during the running tests phase to being complete, and what's more is that we will now always get results if our tests unexpectedly hang or stall.

We now have the ability to turn our attention to the fact that our tests take a lot of time.

So let's dig into how we can speed up for this test suite.

Xcode 12 can help us shorten the loop even more by letting us run tests on multiple devices.

This is a test report from Fruita.

We see the results of about a dozen tests that took between a few hundred milliseconds to several minutes to run.

Overall, our tests are just about 13 minutes to run with many of these tests taking time on the order of minutes to complete running.

This is a clue that we would benefit from parallel testing.

Right now Fruita is using non-distributed testing.

That means each and every testcase defined is executed serially on a run destination, and that will always take the most amount of time.

You've likely experienced this if you have ever pressed command-U in Xcode with paralyzation disabled.

A solution to speed this up is to use a feature we call Parallel Distributed Testing.

In the case of Parallel Distributed Testing, Xcode build will distribute tests to each run destination by class.

Each device will then run a single test class at a time.

Once a run destination has finished running a class, Xcode build continues to give it a new one, until there aren't any left.

It's very important to note that the allocation of test classes to run destinations is non-deterministic.

If you're testing logic that is device or OS specific, this can lead to unexpected failures or skipped tests.

When we first added support for Parallel Distributed Testing to Xcode 10, this supported configuration matrix looked like unit tests some MacOS and unit and UI tests on iOS and tvOS simulators.

Starting with Xcode 12, the matrix now looks like this.

You have the ability to run tests in parallel on physical iOS and tvOS devices via Xcodebuild.

To enable parallel distributed testing, set the parallel-testing-enabled flag to Yes.

Then set the parallelize-tests among-destinations flag.

This makes Xcodebuild divide your tests over the destinations you specify.

With just two devices, XCTest own test suites achieved a speed up of 30 percent.

Just imagine what this would do for the Fruita app or your app with more devices.

By adopting distributed testing, we've been able to take our long feedback loops on XCTest and shorten them, enabling us to write, run, and analyze our tests faster.

Now we wanted to offer you a few recommendations for how you can leverage distributed testing in your own tests.

Since test allocation is non-deterministic, it's ideal to use a device pool of identical devices and OS versions.

This is so you can avoid difficult-to-reproduce test failures that may have been driven out due to the particular destination allocation Xcodebuild made.

If you're using a device pool of different devices and OS versions, then we recommend you prefer distributing tests that are agnostic to the devices and OSs they would be running on.

For example, tests for a framework of pure business logic are less likely to encounter issues since they wouldn't be running code that depended on destination-specific details.

Lastly, if you're interested in testing your code against more Oss and devices, for example, to prove your app works with both iOS 13 and 14, then we recommend you use Parallel Destination testing.

Destination testing runs the entirety of a test suite on a given destination and does not distribute the individual tests across destinations.

If you're interested in learning more about destination testing and how tests are allocated, check out What's New in Testing. from WWDC 2018.

Now as a result of focusing on our test report and using Xcode 12, our tests will no longer hang or stall; they'll give us more diagnostic should a test take an unexpected amount of time, and they're faster.

In conclusion, we recommend you use Execution Time Allowances to ensure your tests always complete in the event they hang.

Use spin dumps for diagnosing application stalls and hangs both for when your tests and app-stall.

Use Parallel Distributed Testing to speed up your tests for running portions of your suite on different run destinations.

And use Parallel Destination Testing to simultaneously run your tests on more OS versions and devices.

Do this to get faster feedback on whether your code is behaving as expected between different OSs and devices.

Thank you all so very much for joining me during the session.

We hope your test suites are fast, that their feedback is actionable, and that you enjoy the rest of WWDC 2020.

♪