Graceful Cancellation for Cloud Mac CI Jobs

In remote CI queues, failure is not the hardest condition to handle—cancellation is. When a developer stops a job from the interface, the runner will typically send TERM to the script first. If the script does not handle that signal, a running xcodebuild process, test process, or log collector may continue occupying the workspace. The next job then encounters lock files, a simulator that is still in use, or an incomplete result bundle. What appears to be a random failure is actually the consequence of the previous job not shutting down cleanly.

Define What Completed Cancellation Means

A job disappearing from the queue does not mean its resources have been reclaimed. An actionable definition of completion should include at least four conditions:

1. The main build process receives the termination signal and exits.
2. The script records the actual reason for exiting instead of reporting every cancellation as a build failure.
3. Generated logs and xcresult data are retained.
4. No background processes or temporary mounts belonging to the job remain in the workspace.

Cancellation is a normal pipeline path. If cleanup logic is tested only during exceptional exits, you will usually discover that it does not work when the system is busiest.

Start by confirming which signal the runner actually sends. A test job can include a probe that records signals without capturing sensitive environment variables. Do not rely on assumptions about default behavior, and do not treat an immediate KILL as a cancellation strategy. It gives processes no opportunity to close databases, flush logs, or finalize result bundles.

Trap TERM and Track the Main Process

The build script should explicitly retain the PID of xcodebuild. After receiving TERM or INT, terminate only processes created by the current job; do not use an unscoped killall. The following Bash skeleton writes logs and results to a dedicated run directory:

#!/bin/bash
set -u

run_id="${CI_RUN_ID:-manual-$(date +%s)}"
run_dir="$PWD/.ci-runs/$run_id"
result_path="$run_dir/TestResults.xcresult"
log_path="$run_dir/xcodebuild.log"
status_path="$run_dir/status.txt"
child_pid=""
cancelled=0

mkdir -p "$run_dir"

on_cancel() {
  cancelled=1
  if [[ -n "$child_pid" ]] && kill -0 "$child_pid" 2>/dev/null; then
    kill -TERM "$child_pid" 2>/dev/null || true
  fi
}

trap on_cancel TERM INT

xcodebuild \
  -workspace Example.xcworkspace \
  -scheme Example \
  -destination 'platform=iOS Simulator,name=iPhone 16' \
  -resultBundlePath "$result_path" \
  test >"$log_path" 2>&1 &

child_pid=$!
wait "$child_pid"
build_status=$?

if [[ "$cancelled" -eq 1 ]]; then
  printf 'cancelled\n' >"$status_path"
  exit 130
fi

printf 'finished:%s\n' "$build_status" >"$status_path"
exit "$build_status"

The trap does not delete the directory directly. It only forwards the signal, while the main flow still executes wait. This makes it possible to distinguish an ordinary failure from a manual cancellation. Exit code 130 is commonly used to indicate an interrupted process, but first verify how your CI system maps exit codes to cancellation states.

Give Processes a Bounded Shutdown Period

Some tests may not respond to TERM. A production script can check the PID once per second after sending the signal and wait, for example, 20 seconds before escalating to KILL. The grace period must be bounded, or a cancellation could occupy the runner indefinitely. At the same time, do not reduce the wait to one or two seconds, because xcodebuild may still be writing the result bundle.

Separate Evidence Retention from Cache Cleanup

A common shutdown-cleanup mistake is to run rm -rf "$run_dir" directly from trap cleanup EXIT. Although this leaves a clean directory, it also removes the logs needed to determine what happened. A safer approach is to divide the contents into two categories:

Log archiving must also tolerate files that have not yet been created. Check with [[ -e "$result_path" ]] rather than turning a missing result bundle into a second error. If uploading may take time, give the upload step its own timeout and ensure that an upload failure does not overwrite the original build exit code.

Place temporary directories under a fixed parent directory and include the run ID in their paths. Before deleting anything, validate both the prefix and the resolved target so that an empty variable cannot broaden the deletion scope:

safe_remove_run_dir() {
  local target="$1"
  local root="$PWD/.ci-tmp"

  [[ -n "$target" ]] || return 1
  [[ "$target" == "$root/"* ]] || return 1
  [[ -d "$target" ]] || return 0

  rm -rf -- "$target"
}

Check for Orphaned Processes Instead of Clearing the Entire Host

After the job exits, start by recording a process snapshot:

ps -axo pid,ppid,state,etime,command >"$run_dir/processes-after.txt"
pgrep -P "$$" >"$run_dir/direct-children.txt" 2>/dev/null || true

pgrep -P reports only direct child processes and cannot cover every descendant. A more reliable approach is to have the script launch every helper process and record each PID. Log forwarders, test agents, and port-forwarding processes can each be added to an array and checked individually during cleanup. Do not kill every process with a matching name across the host. Even a dedicated physical Mac mini may also be running long-lived processes intentionally started by a developer.

If the pipeline allows concurrent jobs, the workspace, result directory, simulator device set, and temporary directory should all include the job ID. Graceful cancellation can manage process lifecycles, but it cannot prevent overwrites caused by multiple jobs sharing the same mutable directory.

Validate the Cancellation Path with Three Types of Drills

Run at least three drills before deployment. First, cancel during dependency resolution and verify that the package manager and logging processes exit. Second, cancel during compilation and confirm that the build database is not mistakenly reused by the next job. Third, cancel while tests are running and verify that the result bundle remains readable and that simulator-related processes do not continue occupying the job’s device set.

Check the following after every drill:

• The CI interface marks the result as cancelled rather than as an ordinary test failure.
• The main process exits within the grace period, and any timeout escalation is recorded.
• The end of the log identifies the stage in which cancellation occurred.
• When a result bundle exists, it can be read with xcrun xcresulttool.
• The next job uses a new run directory and starts successfully.
• Running the cleanup script repeatedly produces no errors and does not delete data belonging to other jobs.

When running long-lived CI workloads on RentMini cloud Macs, include cancellation drills in the acceptance checklist after runner updates. For node and configuration selection, confirm the currently available options in the console; the script itself should not depend on a city, machine name, or fixed workspace path. The goal is not to make the directory look as clean as possible. It is to give every shutdown clear boundaries and retained evidence so that the next build can start from a known state.

Frequently asked questions