Running your tests with different environments, localizations and sanitizers is a good way to catch more bugs. You can use schemes to create the different configurations but that quickly becomes unmanageable.

Apple added test plans to Xcode 11 to make it easier to rerun tests with varying test configurations. Here’s my notes on how to get started with test plans.

What is a Test Plan?

An Xcode test plan provides a way to run a selections of tests with various test configurations. The test plan is a JSON file with the .xctestplan extension that you add to your Xcode project and reference from a scheme. It has three main elements:

• Test targets: One or more test targets (unit or UI). You don’t have to run all tests in the test target. For each test plan, you select the tests the plan will run and whether the tests can run in parallel.

• Default options: A set of default (shared) settings that can be overriden by specific configurations. These are the settings you would normally find in the scheme editor: launch arguments, localization settings, screenshot settings, text execution (alphabetic or random), runtime sanitizers, thread checker and malloc guards.

• Configurations: One or more configurations which override the default options. A test plan runs the selected tests multiple times, once for each of the configurations.

What Are They Good For?

Some possible uses for test plans:

• Running tests with different sanitizers/diagnostics: You can’t test with both the address and thread sanitizers enabled at the same time. Using a test plan we can create two configurations, one with the address sanitizer enabled and a second with the thread sanitizer. Running the test plan then runs our tests twice, once for each of the sanitizers. You can easily add another configuration to run with malloc diagnostics.

• Testing multiple localizations: Create a configuration for each language you support. You set the application language and region as part of the configuration. This is also a good way to generate screenshots for localizers.

• Running selected tests with a specific configuration: For example, you don’t want performance tests to run in parallel.

• Different test scopes: I like to have a quick set of tests I run by default when I hit command-U. I also want a full set of unit, UI and performance tests running against several configurations. Test plans make it much easier to set that up.

If you’re already doing some of the above by creating different schemes you may find you can consolidate into a single scheme with one or more test plans.

Converting From Schemes

You create a new test plan from the Xcode menu Product > Test Plan > New Test Plan.... If you have an existing scheme you can convert it to use test plans. From the scheme editor, select the Test phase and use the Convert to use Test Plans... button below the tests:

Alternatively, from the Xcode menu: Product > Scheme > Convert Scheme to use Test Plans. Either way you get to choose from three options:

Choosing the first option creates a new test plan populated with the tests and configuration from the scheme. Save the test plan file to the project directory (and check it into version control). I like to save the test plans in a top-level project directory named TestPlans:

If you look at the scheme now you’ll see it lists the test plan where the tests used to be:

You can add multiple test plans to the scheme, each with a different selection of tests and configurations.

Test Plan Examples

Let’s look at some examples of how we might use test plans. Here’s the three test plans I’m going to create:

1. Run just the unit tests with a single configuration (a quick set of default tests).
2. A full run of all my unit and UI tests with multiple configurations covering both the address and thread sanitizers.
3. Just the performance tests.

I’ll run all the tests in parallel (and random order) except the performance tests. See Random and parallel tests for a recap of these test options.

Unit Test Plan

I’m starting from the scheme editor with a scheme with no test plans. Select the test phase and use the “+” button at the bottom to create an empty test plan:

I named the test plan UnitTest and saved it to the TestPlans directory in my project. Xcode lists the new test plan in the scheme editor as the default:

You can close the scheme editor at this point and click on the test plan in the Xcode navigator. The test plan is empty so the first step is to add one or more test targets. My project has both a unit test and UI test target. I only want to run unit tests but I’m going to add both targets. Use the “+” button at the bottom of the test plan window to add the targets:

Why add both targets? If I don’t have the UI test target added to the current test plan Xcode doesn’t allow me to manually run any of the UI tests from the source code editor:

Adding all my test targets to the default test plan doesn’t mean that I need to have of the tests enabled. I can disable all the UI tests for this test plan (this doesn’t stop me from running them manually from the source editor):

I want to run the unit tests in parallel on the simulator. That’s an option you can change for each test target in the test plan:

With the tests selected, click on the Configurations tab. A new test plan has shared (default) settings and a single configuration named Configuration 1. Start with the shared settings. Settings which differ from the defaults show in bold. I changed the execution order to random and enabled the address and undefined behavior sanitizers:

We only need one configuration for this test plan which will have the default settings. I changed the name to make it clear it uses the address sanitizer:

Full Run Test Plan

Following the same steps I added a second test plan named FullTests to the scheme. This time after adding both test targets I deselected the performance tests from the UI test target:

The default configuration settings are the same as for the unit test plan. I’ve added a second configuration where I disabled the address sanitizer and enabled the thread sanitizer (I enabled the undefined behavior sanitizer in both configurations):

The selected tests will run with both the ASAN and TSAN configurations.

Performance Test Plan

Finally in my performance test plan I added the UI test target and selected the performance tests (it’s sometimes easier to create performance tests in their own test target). This time I haven’t changed the option to run the tests in parallel which could affect the performance:

I have a single configuration which has all of the sanitizer and diagnostic tools turned off:

Running Test Plans

There are a few ways to run your test plans. From the Xcode Product > Test menu or more simply command-U will run the current test plan. You can see the results or change the current test plan in the Test navigator:

One thing to be aware of if you are working with a test plan with multiple configurations. If you manually run tests from the source code editor Xcode will, by default run the test(s) for all configurations in the test plan. You can option click on the test to choose a single configuration if you prefer.

The report navigator shows the results for each configuration organised by test target and test:

From The Command line

My preference is to have a default test plan with all test targets that runs a selection of tests with a single configuration. I then either switch test plans or use the command line to run more extensive or specific test plans.

To list all test plans in a scheme:

$ xcodebuild -scheme ToDo -showTestPlans
Test plans associated with the scheme "ToDo":
        UnitTest
        FullTests
        PerfTests

To run the default test plan on the iPhone 8 (iOS 13.3) simulator:

$ xcodebuild test -scheme ToDo 
  -destination 'platform=iOS Simulator,OS=13.3,name=iPhone 8'

To run a different test plan:

$ xcodebuild test -scheme ToDo 
  -destination 'platform=iOS Simulator,OS=13.3,name=iPhone 8'
  -testPlan PerfTests

Read More

• WWDC 2019 Session 413 Testing in Xcode
• Xcode 10 Random and Parallel Tests