Scalable Continuous Integration for iOS

Originally published on the Just Eat Takeaway Engineering Blog.

How Just Eat Takeaway.com leverage AWS, Packer, Terraform and GitHub Actions to manage a CI stack of macOS runners.

Problem

At Just Eat Takeaway.com (JET), our journey through continuous integration (CI) reflects a landscape of innovation and adaptation. Historically, JET’s multiple iOS teams operated independently, each employing their distinct CI solutions.

The original Just Eat iOS and Android teams had pioneered an in-house CI solution anchored in Jenkins. This setup, detailed in our 2021 article, served as the backbone of our CI practices up until 2020. It was during this period that the iOS team initiated a pivotal migration: moving from in-house Mac Pros and Mac Minis to AWS EC2 macOS instances.

Fast forward to 2023, a significant transition occurred within our Continuous Delivery Engineering (CDE) Platform Engineering team. The decision to adopt GitHub Actions company-wide has marked the end of our reliance on Jenkins while other teams are in the process of migrating away from solutions such as CircleCI and GitLab CI. This transition represented a fundamental shift in our CI philosophy. By moving away from Jenkins, we eliminated the need to maintain an instance for the Jenkins server and the complexities of managing how agents connected to it. Our focus then shifted to transforming our Jenkins pipelines into GitHub Actions workflows.

This transformation extended beyond mere tool adoption. Our primary goal was to ensure that our macOS instances were not only scalable but also configured in code. We therefore enhanced our global CI practices and set standards across the entire company.

Desired state of CI

As we embarked on our journey to refine and elevate our CI process, we envisioned a state-of-the-art CI system. Our goals were ambitious yet clear, focusing on scalability, automation, and efficiency. At the time of implementing the system, no other player in the industry seemed to have implemented the complete solution we envisioned.

Below is a summary of our desired CI state:

• Instance setup in code: One primary objective was to enable the definition of the setup of the instances entirely in code. This includes specifying macOS version, Xcode version, Ruby version, and other crucial configurations. For this purpose, the HashiCorp tool Packer, emerged once again as an ideal solution, offering the flexibility and precision we required.
• IaC (Infrastructure as Code) for macOS instances: To define the infrastructure of our fleet of macOS instances, we leaned towards Terraform, another HashiCorp tool. Terraform provided us with the capability to not only deploy but also to scale and migrate our infrastructure seamlessly, crucially maintaining its state.
• Auto and Manual Scaling: We wanted the ability to dynamically create CI runners based on demand, ensuring that resources were optimally utilized and available when needed. To optimize resource utilization, especially during off-peak hours, we desired an autoscaling feature. Scaling down our CI runners on weekends when developer activity is minimal was critical to be cost-effective.
• Automated Connection to GitHub Actions: We aimed for the instances to automatically connect to GitHub Actions as runners upon deployment. This automation was crucial in eliminating manual interventions via SSH or VNC.
• Multi-Team Use: Our vision included CI runners that could be easily used by multiple teams across different time zones. This would not only maximize the utility of our infrastructure but also encourage reuse and standardization.
• Centralized Management via GitHub Actions: To further streamline our CI processes, we intended to run all tasks through GitHub Actions workflows. This approach would allow the teams to self-serve and alleviate the need for developers to use Docker or maintain local environments.

Getting to the desired state was a journey that presented multiple challenges and constant adjustments to make sure we could migrate smoothly to a new system.

Instance setup in code

We implemented the desired configuration with Packer leveraging a number of Shell Provisioners and variables to configure the instance. Here are some of the configuration steps:

• Set user password (to allow remote desktop access)
• Resize the partition to use all the space available on the EBS volume
• Start the Apple Remote Desktop agent and enable remote desktop access
• Update Brew & Install Brew packages
• Install CloudWatch agent
• Install rbenv/Ruby/bundler
• Install Xcode versions
• Install GitHub Actions actions-runner
• Copy scripts to connect to GitHub Actions as a runner
• Copy daemon to start the GitHub Actions self-hosted runner as a service
• Set macos-init modules to perform provisioning of the first launch

While the steps above are naturally configuration steps to perform when creating the AMI, the macos-init modules include steps to perform once the instance becomes available.

The create_ami workflow accepts inputs that are eventually passed to Packer to generate the AMI.

packer build \
  --var ami_name_prefix=${{ env.AMI_NAME_PREFIX }} \
  --var region=${{ env.REGION }} \
  --var subnet_id=${{ env.SUBNET_ID }} \
  --var vpc_id=${{ env.VPC_ID }} \
  --var root_volume_size_gb=${{ env.ROOT_VOLUME_SIZE_GB }} \
  --var macos_version=${{ inputs.macos-version}} \
  --var ruby_version=${{ inputs.ruby-version }} \
  --var xcode_versions='${{ steps.parse-xcode-versions.outputs.list }}' \
  --var gha_version=${{ inputs.gha-version}} \
  bare-metal-runner.pkr.hcl

Different teams often use different versions of software, like Xcode. To accommodate this, we permit multiple versions to be installed on the same instance. The choice of which version to use is then determined within the GitHub Actions workflows.

The seamless generation of AMIs has proven to be a significant enabler. For example, when Xcode 15.1 was released, we executed this workflow the same evening. In just over two hours, we had an AMI ready to deploy all the runners (it usually takes 70–100 minutes for a macOS AMI with 400GB of EBS volume to become ready after creation). This efficiency enabled our teams to use the new Xcode version just a few hours after its release.

IaC (Infrastructure as Code) for macOS instances

Initially, we used distinct Terraform modules for each instance to facilitate the deployment and decommissioning of each one. Given the high cost of EC2 Mac instances, we managed this process with caution, carefully balancing host usage while also being mindful of the 24-hour minimum allocation time.

We ultimately ended up using Terraform to define a single infrastructure (i.e. a single Terraform module) defining resources such as:

• aws_key_pair, aws_instance, aws_ami
• aws_security_group, aws_security_group_rule
• aws_secretsmanager_secret
• aws_vpc, aws_subnet
• aws_cloudwatch_metric_alarm
• aws_sns_topic, aws_sns_topic_subscription
• aws_iam_role, aws_iam_policy, aws_iam_role_policy_attachment, aws_iam_instance_profile

A crucial part was to use count in aws_instance, setting the value of a variable passed in from deploy_infra workflow. Terraform performs the necessary scaling upon changing the value.

We have implemented a workflow to perform Terraform apply and destroy commands for the infrastructure. Only the AMI and the number of instances are required as inputs.

terraform ${{ inputs.command }} \
  --var ami_name=${{ inputs.ami-name }} \
  --var fleet_size=${{ inputs.fleet-size }} \
  --auto-approve

Using the name of the AMI instead of the ID allows us to use the most recent one that was generated, useful in case of name clashes.

variable "ami_name" {
  type = string
}

variable "fleet_size" {
  type = number
}

data "aws_ami" "bare_metal_gha_runner" {
  most_recent = true

  filter {
    name   = "name"
    values = ["${var.ami_name}"]
  }
  
  ...
}

resource "aws_instance" "bare_metal" {
  count         = var.fleet_size
  ami           = data.aws_ami.bare_metal_gha_runner.id
  instance_type = "mac2.metal"
  tenancy       = "host"
  key_name      = aws_key_pair.bare_metal.key_name
  ...
}

Instead of maintaining multiple CI instances with varying software configurations, we concluded that it’s simpler and more efficient to have a single, standardised setup. While teams still have the option to create and deploy their unique setups, a smaller, unified system allows for easier support by a single global configuration.

Auto and Manual Scaling

The deploy_infra workflow allows us to scale on demand but it doesn’t release the underlying dedicated hosts which are the resources that are ultimately billed.

The autoscaling solution provided by AWS is great for VMs but gets sensibly more complex when actioned on dedicated hosts. Auto Scaling groups on macOS instances would require a Custom Managed License, a Host Resource Group and, of course, a Launch Template. Using only AWS services appears to be a lot of work to pull things together and the result wouldn’t allow for automatic release of the dedicated hosts.

AirBnb mention in their Flexible Continuous Integration for iOS article that an internal scaling service was implemented:

An internal scaling service manages the desired capacity of each environment’s Auto Scaling group.

Some articles explain how to set up Auto Scaling groups for mac instances (see 1 and 2) but after careful consideration, we agreed that implementing a simple scaling service via GitHub Actions (GHA) was the easiest and most maintainable solution.

We implemented 2 GHA workflows to fully automate the weekend autoscaling:

• Upscaling workflow to n, triggered at a specific time at the beginning of the working week
• Downscaling workflow to 1, triggered at a specific time at the beginning of the weekend

We retain the capability for manual scaling, which is essential for situations where we need to scale down, such as on bank holidays, or scale up, like on release cut days, when activity typically exceeds the usual levels.

Additionally, we have implemented a workflow that runs multiple times a day and tries to release all available hosts without an instance attached. This step lifts us from the burden of having to remember to release the hosts. Dedicated hosts take up to 110 minutes to move from the Pending to the Available state due to the scrubbing workflow performed by AWS.

Manual scaling can be executed between the times the autoscaling workflows are triggered and they must be resilient to unexpected statuses of the infrastructure, which thankfully Terraform takes care of.

Both down and upscaling are covered in the following flowchart:

The autoscaling values are defined as configuration variables in the repo settings:

It usually takes ~8 minutes for an EC2 mac2.metal instance to become reachable after creation, meaning that we can redeploy the entire infrastructure very quickly.

Automated Connection to GitHub Actions

We provide some user data when deploying the instances.

resource "aws_instance" "bare_metal" {
  ami       = data.aws_ami.bare_metal_gha_runner.id
  count     = var.fleet_size
  ...
  user_data = <<EOF
{
    "github_enterprise": "<GHE_ENTERPRISE_NAME>",
    "github_pat_secret_manager_arn": ${data.aws_secretsmanager_secret_version.ghe_pat.arn},
    "github_url": "<GHE_ENTERPRISE_URL>",
    "runner_group": "CI-MobileTeams",
    "runner_name": "bare-metal-runner-${count.index + 1}"
}
  EOF

The user data is stored in a specific folder by macos-init and we implement a module to copy the content to ~/actions-runner-config.json.

### Group 10 ###
[[Module]]
    Name = "Create actions-runner-config.json from userdata"
    PriorityGroup = 10
    RunPerInstance = true
    FatalOnError = false
    [Module.Command]
        Cmd = ["/bin/zsh", "-c", 'instanceId="$(curl http://169.254.169.254/latest/meta-data/instance-id)"; if [[ ! -z $instanceId ]]; then cp /usr/local/aws/ec2-macos-init/instances/$instanceId/userdata ~/actions-runner-config.json; fi']
        RunAsUser = "ec2-user"

which is in turn used by the configure_runner.sh script to configure the GitHub Actions runner.

#!/bin/bash

GITHUB_ENTERPRISE=$(cat $HOME/actions-runner-config.json | jq -r .github_enterprise)
GITHUB_PAT_SECRET_MANAGER_ARN=$(cat $HOME/actions-runner-config.json | jq -r .github_pat_secret_manager_arn)
GITHUB_PAT=$(aws secretsmanager get-secret-value --secret-id $GITHUB_PAT_SECRET_MANAGER_ARN | jq -r .SecretString)
GITHUB_URL=$(cat $HOME/actions-runner-config.json | jq -r .github_url)
RUNNER_GROUP=$(cat $HOME/actions-runner-config.json | jq -r .runner_group)
RUNNER_NAME=$(cat $HOME/actions-runner-config.json | jq -r .runner_name)

RUNNER_JOIN_TOKEN=` curl -L \
  -X POST \
  -H "Accept: application/vnd.github+json" \
  -H "Authorization: Bearer $GITHUB_PAT"\
  $GITHUB_URL/api/v3/enterprises/$GITHUB_ENTERPRISE/actions/runners/registration-token | jq -r '.token'`

MACOS_VERSION=`sw_vers -productVersion`

XCODE_VERSIONS=`find /Applications -type d -name "Xcode-*" -maxdepth 1 \
  -exec basename {} \; \
  | tr '\n' ',' \
  | sed 's/,$/\n/' \
  | sed 's/.app//g'`

$HOME/actions-runner/config.sh \
  --unattended \
  --url $GITHUB_URL/enterprises/$GITHUB_ENTERPRISE \
  --token $RUNNER_JOIN_TOKEN \
  --runnergroup $RUNNER_GROUP \
  --labels ec2,bare-metal,$RUNNER_NAME,macOS-$MACOS_VERSION,$XCODE_VERSIONS \
  --name $RUNNER_NAME \
  --replace

The above script is run by a macos-init module.

### Group 11 ###
[[Module]]
    Name = "Configure the GHA runner"
    PriorityGroup = 11
    RunPerInstance = true
    FatalOnError = false
    [Module.Command]
        Cmd = ["/bin/zsh", "-c", "/Users/ec2-user/configure_runner.sh"]
        RunAsUser = "ec2-user"

The GitHub documentation states that it’s possible to create a customized service starting from a provided template. It took some research and various attempts to find the right configuration that allows the connection without having to log in in the UI (over VNC) which would represent a blocker for a complete automation of the deployment. We believe that the single person who managed to get this right is Sébastien Stormacq who provided the correct solution.

The connection to GHA can be completed with 2 more modules that install the runner as a service and load the custom daemon.

### Group 12 ###
[[Module]]
    Name = "Run the self-hosted runner application as a service"
    PriorityGroup = 12
    RunPerInstance = true
    FatalOnError = false
    [Module.Command]
        Cmd = ["/bin/zsh", "-c", "cd /Users/ec2-user/actions-runner && ./svc.sh install"]
        RunAsUser = "ec2-user"

### Group 13 ###
[[Module]]
    Name = "Launch actions runner daemon"
    PriorityGroup = 13
    RunPerInstance = true
    FatalOnError = false
    [Module.Command]
        Cmd = ["sudo", "/bin/launchctl", "load", "/Library/LaunchDaemons/com.justeattakeaway.actions-runner-service.plist"]
        RunAsUser = "ec2-user"

Using a daemon instead of an agent (see Creating Launch Daemons and Agents), doesn’t require us to set up any auto-login which on macOS is a bit of a tricky procedure and is best avoided also for security reasons. The following is the content of the daemon for completeness.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>Label</key>
    <string>com.justeattakeaway.actions-runner-service</string>
    <key>ProgramArguments</key>
    <array>
      <string>/Users/ec2-user/actions-runner/runsvc.sh</string>
    </array>
    <key>UserName</key>
    <string>ec2-user</string>
    <key>GroupName</key>
    <string>staff</string>
    <key>WorkingDirectory</key>
    <string>/Users/ec2-user/actions-runner</string>
    <key>RunAtLoad</key>
    <true/>    
    <key>StandardOutPath</key>
    <string>/Users/ec2-user/Library/Logs/com.justeattakeaway.actions-runner-service/stdout.log</string>
    <key>StandardErrorPath</key>
    <string>/Users/ec2-user/Library/Logs/com.justeattakeaway.actions-runner-service/stderr.log</string>
    <key>EnvironmentVariables</key>
    <dict> 
      <key>ACTIONS_RUNNER_SVC</key>
      <string>1</string>
    </dict>
    <key>ProcessType</key>
    <string>Interactive</string>
    <key>SessionCreate</key>
    <true/>
  </dict>
</plist>

Not long after the deployment, all the steps above are executed and we can appreciate the runners appearing as connected.

Multi-Team Use

We start the downscaling at 11:59 PM on Fridays and start the upscaling at 6:00 AM on Mondays. These times have been chosen in a way that guarantees a level of service to teams in the UK, the Netherlands (GMT+1) and Canada (Winnipeg is on GMT-6) accounting for BST (British Summer Time) and DST (Daylight Saving Time) too. Times are defined in UTC in the GHA workflow triggers and the local time of the runner is not taken into account.

Since the instances are used to build multiple projects and tools owned by different teams, one problem we faced was that instances could get compromised if workflows included unsafe steps (e.g. modifications to global configurations).

GitHub Actions has a documentation page about Hardening self-hosted runners specifically stating:

Self-hosted runners for GitHub do not have guarantees around running in ephemeral clean virtual machines, and can be persistently compromised by untrusted code in a workflow.

We try to combat such potential problems by educating people on how to craft workflows and rely on the quick redeployment of the stack should the instances break.

We also run scripts before and after each job to ensure that instances can be reused as much as possible. This includes actions like deleting the simulators’ content, derived data, caches and archives.

Centralized Management via GitHub Actions

The macOS runners stack is defined in a dedicated macOS-runners repository. We implemented GHA workflows to cover the use cases that allow teams to self-serve:

• create macOS AMI
• deploy CI
• downscale for the weekend*
• upscale for the working week*
• release unused hosts*

* run without inputs and on a scheduled trigger

The runners running the jobs in this repo are small t2.micro Linux instances and come with the AWSCLI installed. An IAM instance role with the