Thursday, May 29, 2026, 3:03 PM. On a twelve-person iOS team, someone in Slack #ios-ci dropped a Run link: "Local Xcode passed in twelve minutes — PR still Queued." Half an hour of silence. Messages like that are background noise. Forty-eight minutes later the same person added: "Queued 47m, run 12m. Don't expect a merge tonight." A release engineer stepped out of a meeting, glanced at their phone: "ios-release Archive is still in line — PR lane's blocked again."
Screenshots below come from materials a customer shared while we helped triage — redacted, numbers untouched. Cross-check against GitHub's job execution time and queue_wait_seconds docs if you want to reproduce the math yourself.
Two apps, one monorepo, all CI on GitHub-hosted macOS runners. Tolerable in 2025; by spring 2026 the feedback loop had stretched past patience. GitHub Actions wasn't "broken" — for iOS teams it felt more like a shared shuttle that always jams at rush hour.
That Thursday: open the Job page — yellow bar four times longer than green
They sent a screenshot of Run #18472910356's Job page — not a story, a timestamped record. Yellow bar (Queued): 47m 12s. Green bar (Run): 12m 4s. Footer fields: queue_wait_seconds: 2832, run duration: 724. The math is blunt: wait time >> run time.
ios-pr.yml · macos-latest · 2026-05-29 15:04 CSTThe team was still in a meeting debating -jobs parallelism and DerivedData cache keys — nobody had pulled queued vs. run columns for the last ten macOS jobs. Release had it worse: ios-release.yml packed xcodebuild archive, notarization, TestFlight upload, and PR unit tests onto one lane, one macos-latest label. Archive runs 25–40 minutes, chewing scarce concurrency slots — fast-feedback PR jobs get dragged into the same queue.
If you only want to triage Queued right now, start with our macOS runner queue runbook. Their turning point began after the room admitted the problem was supply — and numbers, not gut feel, won the argument.
Evidence: a hand-copied sheet of ten failed jobs
Before standup on May 30, someone manually copied ten failed macOS jobs from the Actions run list (below). No fancy BI — just a table with two columns that matter: queued_s and run_s. Median queued 2650s, run 724s; ratio peaked at 4.5×. The room went quiet for a few seconds: nobody brought up "maybe it's Xcode 17.4" again.
| Often blamed on | Signal in Fig. 3 | More likely cause |
|---|---|---|
| GitHub outage | 10/10 rows queued_s > run_s | macOS concurrency cap, heavy jobs hogging slots |
| Project got bigger | run_s stable at 690–811s | Compile isn't the bottleneck — waiting is |
| Teammates spamming pushes | Multiple runs on one PR (see Fig. 2) | Agent / bot loop |
| Expired certs | Signing failures scattered across runs | Cold keychain, parallel contention |
Fig. 2: one PR triggered six workflow runs
Queue pain was only act one. In March the team wired up a coding agent: CI red → read logs → auto-commit → trigger Actions again. PR #847's Checks page tells the story — six workflow runs, four actors bots, two human. Still called CI; behaviorally a retry loop. See what happens when CI gets looped.
ios-pr.yml triggers (4× bot · 2× human) · 2026-05-29Someone on the team joked later: "CI didn't die — we turned it into a perpetual motion machine." The automation wasn't wrong; every bot fix was competing for the public queue with no dedicated runner pool.
Turning point: one Cloud Mac, and the pipeline split on May 31
After the spreadsheet, they broke logs down by job type: roughly 70% PR validation and integration builds, 20% Simulator tests, 10% Archive and upload — strikingly close to another team's measured mix at 500 iOS CI runs a day. Proposals flew: buy eight Mac minis, migrate to Xcode Cloud. Two things actually shipped:
May 31: rent one Apple Silicon Cloud Mac, register a self-hosted runner (labels self-hosted, macOS, cloud-mac). The A/B test was plain: same commit, fifty builds on hosted macos-latest vs. Cloud Mac — queued variance collapsed from "sometimes 40 minutes, sometimes 5" to "mostly single-digit seconds."
That night, split the pipelines. Move Archive, export, and upload out of ios-pr.yml into ios-release.yml; PR lane keeps build plus unit tests only. Warm-up cron too: nightly xcodebuild build on the Cloud Mac, DerivedData on disk — steadier than cold-pulling from actions/cache every run.
Fig. 6 · Same team — median wall time (PR build, 50 runs per two-week window)
Source: customer A/B logs (2026-05-31 — 06-08); cross-checkable with single runs in Figs. 1 and 4.
June 9: Fig. 4 — a Job page that finally looks sane
Two weeks after the split, Monday morning Slack: someone posted Run #18510488201 — queued 41s, run 9m 17s, runner self-hosted, cloud-mac. No applause in the channel; one emoji reaction. On an iOS team, that's high praise.
self-hosted, cloud-mac · 2026-06-09 09:11 CSTFence in the agent: four rules on the Wiki
Splitting pipelines fixed Archive blocking PRs; the bot loop needed its own fix. The team gave bots a separate branch and a low-priority runner pool — humans review before merge. Four hard rules went on the Wiki:
- PR workflow must not run Archive / store upload
- macOS jobs must set
concurrency— no stacked runs on one branch - Self-hosted runners must be label-scoped; no mixing with experimental workflows
- Agent auto-push goes to a dedicated branch — never direct to protected branches
queue_wait_seconds and run duration (like Fig. 3). Count workflow runs per PR (like Fig. 2). Confirm Archive isn't still inside the PR workflow. If median queued > run, fix runner supply before tuning compile flags.
The new playbook: don't ditch GitHub — change the supply
This team's story boils down to four moves iOS shops actually shipped in 2026: Cloud Mac self-hosted runner pool, PR / Release split, warm-up and disk cache, agent guardrails. Can Xcode Cloud replace it? Great for Apple-native loops; teams mixing GitHub PR checks and Android lines mostly keep Actions and move Archive to dedicated macOS capacity.
| If your team looks like… | Typical landing pattern |
|---|---|
| 1–3 people, weekly releases | Hosted macOS + strong cache; manual Archive on release |
| 5–15 people, daily merges | 1–2 Cloud Mac self-hosted + PR/Release split |
| 15+ people, multiple apps | Runner pool per app via labels + warm-up cron |
| Heavy agent-driven fixes | Separate bot pool + no Archive in PR workflow |
Apple's xcodebuild docs stress scheme and destination consistency; if CI cold-starts every run, "passes locally" stops meaning anything. A customer put it roughly this way: staring at a 47-minute yellow bar every day hurts more than cert rotation or a major Xcode bump.
Start with one Cloud Mac self-hosted runner
If your Job page looks like Fig. 1 — yellow swallowing green — the best next dollar is usually one dedicated Apple Silicon Cloud Mac: register it as a GitHub self-hosted runner and turn PR builds from "queue lottery" into the predictable nine minutes in Fig. 4.
Run a two-week A/B: same branch, fifty builds on hosted macOS vs. Cloud Mac self-hosted, copy a sheet like Fig. 3. The data will tell you whether Xcode is slow or supply is slowing iOS development.
Start with one predictable Cloud Mac and put Archive back on the Release lane. See VPSSpark Cloud Mac plans and register your first iOS self-hosted runner.