Customize your advanced Xcode Cloud workflows

♪ Bass music playing ♪ ♪ Itai Rom: Welcome to "Customize your advanced Xcode Cloud workflows." My name is Itai, and I'll be joined later by my colleague, Yan. This year we're introducing Xcode Cloud. Xcode Cloud is a continuous integration and delivery service built into Xcode and designed exclusively for Apple developers. In other sessions, we've shown how you can set up end-to-end workflows to continuously build, test, and distribute your apps. In this session, we're going to talk about some more advanced features that you can use to customize Xcode Cloud to better fit your team's needs. Xcode Cloud is designed to integrate with Apple developer tools and services -- like the developer website, TestFlight, and App Store Connect -- right out of the box. It also integrates with your essential, day-to-day development tools, like all major Git-based source control providers, and even Slack for messaging. However, you may have in-house or proprietary tools and other external services that are a key part of your pipeline. In this session, we'll demonstrate how you can customize Xcode Cloud to integrate well with these tools and services. We're going to cover four customization topics. First, we'll discuss using environment variables to pass extra bits of information into your builds. We'll learn how to use scripts to run custom commands in the actions you run in your build. And also how to add other source repositories that you need during a build. And finally, how you can use webhooks to integrate Xcode Cloud with other systems your team uses. There's a lot to cover, so let's get straight into it with environment variables. As you plan and configure workflows for your project, there may be times when you want the behavior of your build to be slightly different depending on which workflow is running. For example, if your app depends on an API service, you may want your tests to use a staging environment rather than production. In this case, you'd want to pass a different URL for the API service into your tests. Environment variables allow you to do just that. They're simple key-value pairs that allow you to define some information that you can use to further control the behavior of your build. You can configure any environment variables you need right in the Environment section of your workflow. Since they're part of your workflow configuration, you don't have to commit any additional information to your source repository. Every time your workflow runs, the variables you've defined will be set in the environments that are running the actions. For sensitive information, like API keys or access tokens, you can configure a secret environment variable. Secret environment variables are handled securely. They're encrypted and stored securely at all times, and their decrypted values are only available in the temporary environments that are used to run your actions. The values are also redacted from logs, and you can't view them in the workflow editor like you can for a nonsecret variable. Making an environment variable secret is easy. Just check the "Secret" checkbox in the environment variables table. The value of your environment variable will then be hidden from view and once you save your change, will be stored securely and can no longer be viewed in the workflow editor. Environment variables provide a handy mechanism to customize behavior and pass additional information into your workflows. They're even more powerful in conjunction with our next advanced feature: custom scripts. Between Xcode Cloud's workflows and Xcode's schemes, you have a lot of flexibility in how you can set up the actions that you want to run in your workflows. But sometimes you'll need to run custom logic or additional commands during an action, and custom scripts provide a powerful and flexible way to achieve this. A custom script is a shell script that you write and include in your source repository. Your custom scripts are run in each of the actions in your workflow. There are three types of scripts available: the post-clone script, the pre-Xcodebuild script, and the post-Xcodebuild script. Each time Xcode Cloud runs an action it performs a series of steps and each of the custom scripts, as hinted at by its name, is run as a step at a particular point in an action. First, Xcode Cloud sets up a temporary environment and clones source code from your primary repository. Afterwards, Xcode Cloud runs the post-clone script. After resolving all other source dependencies, Xcode cloud runs the pre-Xcodebuild script. Next, Xcode Cloud runs the action's corresponding Xcodebuild command. When the Xcodebuild step finishes, Xcode Cloud runs the post-Xcodebuild script and saves any artifacts it generated earlier. If your workflow includes several actions -- for example, multiple build actions -- or build, test, analyze, and archive actions -- Xcode Cloud runs your custom scripts at the appropriate times for each action. Adding a custom script to Xcode Cloud is easy. You simply add a shell script with the appropriate name into a folder called "ci_scripts", and place this folder at the same level as the project file or workspace you're using in your workflow. Since custom scripts are part of your source code, you can test out script changes in pull requests or even customize their behavior on different branches. When Xcode Cloud runs your actions, it looks for the existence of each of the scripts at the appropriate time and runs them if they're present. You don't need to configure your workflow to run your custom scripts. If the scripts are there, they'll be run. Do note that the name of the ci_scripts folder and the scripts inside must exactly match this naming convention in order for Xcode Cloud to find and run your scripts. Environment variables that you've configured in your workflow are available to use in your custom scripts, including secret environment variables. In addition, Xcode Cloud also provides a variety of other useful environment variables. You can use these variables to add flow control to your scripts, so that you can ensure the commands you want to run are run at the correct points during your workflow. For example, if you want to check if your action is running for iOS, macOS, tvOS, or watchOS, you can use the value of the CI_PRODUCT_PLATFORM variable. In a more targeted scenario -- perhaps there's a command you only want to run during the archive action of a specific workflow -- you can check that the CI_XCODEBUILD_ACTION and CI_WORKFLOW variables match the archive action and the specific workflow before running the command. Let's see custom scripts in action. My team works on a smoothie ordering app called Fruta. In the "Explore Xcode Cloud workflows" session, you learned how to set up workflows to build, test, and distribute apps, like Fruta. We use Xcode Cloud to build and test every time we create a pull request. We also distribute builds from our pull requests to members of our team via TestFlight, allowing them to verify and sign off on changes before the code is merged. Today, I'd like to make it easier for team members to tell if the build installed on their device is from a pull request. We can do that by using a different app icon when we're building from a pull request. As you might've guessed, using a custom script makes setting this up a breeze. Let's see how.

Over in Xcode, I already have the Fruta project open. Before I can add a custom script, I first need to add a ci_scripts folder to my project. I can do this by selecting my project in the project navigator and clicking the plus button at the bottom and then selecting New Group. I'll type in the folder name, ci_scripts. Next, I'll add the beta app icon set that our designer created to the ci_scripts folder, so that my custom script can swap it into place during the build. To do that, I'll drag it in to my ci_scripts folder from Finder. In the sheet, I'll uncheck any selected targets and click Finish.

Finally, let's add a pre-Xcodebuild script. This script will run prior to the Xcodebuild command, and I'll use it to swap Fruta's default app icon set with the beta app icon set when appropriate. I've already created a script to use, so all I need to do is add it into the ci_scripts folder.

Again, I'll uncheck any targets in the sheet and click Finish.

Now that my script is in place, let's take a closer look at what it does. First, I want to make sure that the app icon is only swapped out when the build is coming from a pull request. I can use one of the environment variables that Xcode Cloud provides to check at run time if the build is a pull request or not. There are a variety of environment variables related to pull requests that I could use for this, but in this case I'm going to check that the CI_PULL_REQUEST_NUMBER environment variable is set.

Also, I only want the beta app icon to be used for builds that are distributed to TestFlight. Whenever Xcode Cloud distributes a build to TestFlight, it always builds an archive of the project first. So a good way to check this is to verify if the CI_XCODEBUILD_ACTION environment variable's value is "archive".

If both of these environment variables have the expected values, then I remove the existing app icon set and replace it with the beta app icon set using the rm and mv commands. Also, take note that I'm using the CI_WORKSPACE environment variable to construct the right path for both the default and the beta app icon sets. All that's left to do is to open up a pull request with these changes and wait for Xcode Cloud to build and distribute Fruta to TestFlight. Rather than walk through that process now, I've prepared a build ahead of time. Here, Xcode Cloud has built and distributed Fruta from my pull request branch. In the TestFlight app on my phone, I can verify that my build is using the new beta app icon I just added. Now I can merge my pull request and everyone on the team will start seeing the same beta app icon in builds of their own pull requests. Now that we've seen how to use custom scripts in Xcode Cloud, there are a few important things to note about them. The standard output and standard error from your custom scripts are included in the logs for the action they ran in, and they can also be downloaded from the Artifacts tab. If your script doesn't appear to be running when you expect it to, double-check that you've named it correctly and placed it in a ci_scripts folder alongside your project or workspace. Make sure to add helpful logging and resiliency to aid you in troubleshooting any failures in your custom scripts. For example, if your script makes network requests to external services, you may want to include logic to retry those requests with verbose logging enabled. Additionally, Xcode Cloud respects the exit codes of your scripts. So if your script exits with a nonzero value, Xcode Cloud will consider this a failure and will fail the overall action. You can take advantage of this to ensure that the commands you need to run in your script are successful before Xcode Cloud continues on with the rest of the action. Lastly, it should be noted that in a test action, multiple environments are used to build and run your tests. Only the environment that is used for building your tests will have your source code cloned into it by default. The environments that run your tests won't have source code cloned into them. They'll only have the ci_scripts folder made available on them. As a result, the post-clone script won't run in these environments and your custom scripts and any of their dependencies, such as other shell scripts and small tools, must be entirely contained within the ci_scripts folder. With custom scripts and environment variables, you have two powerful tools to customize the behavior of your workflows. Next, my colleague Yan will show you how to use additional repositories with Xcode Cloud so that you can make use of Swift packages and other dependencies in your workflows. Yan Huang: Thanks, Itai. Many projects are built using tools, libraries, and frameworks. Those dependencies are often hosted in Git repositories shared across projects, and they need to be retrieved for your project to build successfully. Xcode Cloud automatically helps adding these extra repositories. For example, we want to add a new feature to allow users to invite their friends and share a drink to the Fruta app. Another team has implemented a similar feature. So we are going to reuse their package called "InvitationsKit", which is hosted on a private Git repository shared with my team. Let's see how to add this package. Here am I in Xcode with the Fruta project. I can add a package from the File menu and select Add Package. I already have the Nature Labs shared packages collection, which contains a list of packages within our organization. I'm going to select InvitationsKit and click on Add Package. Now that the dependency is added, I'm going to commit this new dependency from the Source Control menu and push my changes to my branch.

We setup a workflow in Xcode Cloud that will start a new build when it receives a new commit from any branch. So this new commit should initiate a new build. Since this is the first time that we're adding this dependency, I expect that the build is going to fail because Xcode Cloud does not have access to the InvitationsKit repository. But Xcode Cloud provides an easy UI to address this issue. Let's head to Xcode Cloud in App Store Connect and take a look at this new build. As expected, the build has failed. Xcode Cloud shows a warning banner indicating that it has an issue with accessing the repository and provides me with the option to grant access to it. I go ahead and click on the Manage Repositories button.

It takes me to the settings page, where I can hover over the invitationsKit repository link and then click on Grant.

Depending on your source control service, you might be directed to approve Xcode Cloud to access the repository. I went ahead and provided access to InvitationsKit in Github.

Now, when I look back in Xcode Cloud, it shows Access Granted. I can now rerun the build.

We expect this build to pass this time. You can go back to the Additional Repositories section under Settings to find out all the repositories that have been connected. You can also reject access from there if they are no longer being used. Xcode Cloud will detect newly referenced repositories during a build. When we add a dependency that Xcode Cloud does not have access to, the UI provides a quick and easy way to fix this issue. This is valid for any Git operations including cloning a repository inside a custom script or referencing a Git submodule. This also applies to all the other dependency management tools. In this demo, we used the new Swift Packages Collection feature to include an additional package. If you want to learn more about it, check out "Discover and curate Swift Packages using Collections." So far we have demonstrated to you how to customize your build within Xcode Cloud. But sometimes you and your team may also want to collaborate through external services. For example, you may want to notify the beta testers when a new build is ready. This is where webhooks can come to help. Webhooks provide a way for Xcode Cloud to communicate with your services. With its rich payload that is sent at different stages of a build lifecycle, you will be able to further automate your workflow and improve your teams' collaborations. Let's dive in to find out more about it. You can set up webhooks in Xcode Cloud to receive your real-time updates at three different stages of your build. First, when the build is created, either because you pushed some code or manually started a build; then, when the build is starting; finally, when the build has completed, no matter if the build failed or succeeded. Let's see how to add a new webhook to our project Adding webhooks is easy in Xcode Cloud. Click on Settings tab on the left column. Click on Webhooks and then the plus button. Then it will ask you to enter the names of your webhook and the URL to an app or services that's capable of receiving and handling HTTP requests. Then, click on Save. Xcode Cloud allows you to create up to five webhooks per product. Make sure to provide a unique name for each of your webhooks to ensure you can easily identify them. The payload that is sent to your endpoint is a JSON blob with information about your build and product, such as your App Store Connect application, your workflow, your product, your build, and more. You need to set up an app or a service to receive and handle the payload that is sent over via HttpRequest. Let's take a look at a sample code on how to achieve that using Swift on AWS Lambda. First, we receive the request. Then decode its payload to a JSON object. Then we check the name of the workflow and the build state. If the workflow is a release workflow and the build state has succeeded, we'll post a message to Twitter to let testers or beta users know it's available for testing. Finally, we will return a 200 status code to acknowledge that the webhook request has been successfully handled. If you wish to learn more about running Swift serverless functions, check out this session in WWDC 2020. If your endpoint doesn't return a successful status code, Xcode Cloud will try to send the request again. Xcode Cloud also makes it easy to inspect the content of the webhooks that were delivered to your endpoint. You can go to Xcode Cloud in App Store Connect; click on your settings and the webhooks; select the webhook you wish to inspect. Then it shows you a list of deliveries at different time stamps, and select the one that interests you. It shows you the request that was sent to your service and the response that was received. There are so many things you can do with Xcode Cloud webhooks. Here are a few additional examples. You can automatically create or resolve issues from your bug-checking system; send notifications to a paging system when a build fails; initiate downstream builds as part of a complex release workflow. When it comes to extending your workflow using the exhaustive content of the webhook, the sky is the limit. Let's recap on what we have learned in this session. First, we learned about how to pass in environment variables for your build. Then we demonstrated how to set up scripts to customize your build process and how to work with additional repositories in your project. Finally, we learned how to set up webhooks to receive callbacks from Xcode Cloud at different stages of build life cycle. We hope these features can enhance your team's day-to-day workflow. Thank you for watching. ♪