Skip to content

ci: rerun jobs that GitHub cancelled for lack of a macOS runner - #545

Merged
Mat4m0 merged 2 commits into
mainfrom
ci/retry-runner-shortage
Nov 4, 2026
Merged

Mat4m0 merged 2 commits into
mainfrom
ci/retry-runner-shortage

Conversation

@Mat4m0

@Mat4m0 Mat4m0 commented Nov 4, 2026

Copy link
Copy Markdown
Owner

What changed

  • New workflow Retry runner shortages (.github/workflows/retry-runner-shortage.yml). When an Application verification, client canary or client recertification run fails, it checks why. If every failed or cancelled job was either cancelled with "not acquired by Runner" or is the verify / verify gate that fails as a result, it reruns the failed jobs. It stops after three attempts in all.
  • A short note under "GitHub enforcement" in docs/development-workflow.md.

Why

GitHub's macOS arm64 pool is short of machines. A job that waits too long is cancelled, and verify / verify turns red although no test ran. Since Nov 1 this has hit #539 (three manual reruns), #544, #537, #541, both main runs (d6d046f, 70cb19f) and the client recertification. Every case needed a manual gh run rerun --failed.

Safety

  • It never reruns a real failure. A failed test, a failed audit, a failed install, or a cancellation for another reason (such as a newer push) stops the retry.
  • It reads job results and annotations only. It does not check out or run PR code, so fork PRs are safe. Its permissions are actions: write and checks: read.
  • Versioned release is not covered; a release rerun stays a manual decision.

Verification

  • actionlint passes. pnpm test:policy passes (188 tests).
  • The run script was replayed against real runs from today, with the rerun call replaced by an echo:
Run Situation Result
41663036291 (#544) 3 jobs not acquired rerun
41622716010 (recertification) derive not acquired rerun
41634472759 (#537) 3 jobs not acquired rerun
41631535647 (#537) audit + Electron shard failed no rerun
41612162024 (#537) cancelled by a newer push no rerun
41643011855 (Dependabot) install failed no rerun
41635280475 (#540) audit failed + 1 not acquired no rerun
invalid run ID API 404 step fails
  • Not verifiable before merge: a workflow_run workflow runs only from the default branch. The first real shortage after merge is the live test. If GitHub does not trigger workflow_run for a rerun started by the workflow token, the limit is effectively two attempts instead of three.

🤖 Generated with Claude Code

Mat4m0 and others added 2 commits November 3, 2026 16:47
GitHub's macOS arm64 pool is short of machines. Waiting jobs are cancelled
with "not acquired by Runner", and verify / verify then fails although no
test ran. On 2026-11-03 this caused most red runs and needed manual reruns.

A workflow_run job now reruns such a run, up to three attempts in all. It
reruns only when every failed or cancelled job is a runner shortage or the
verification gate that follows from one. It reads job results only and never
checks out PR code. Versioned release stays manual.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Mat4m0
Mat4m0 enabled auto-merge (squash) November 4, 2026 06:20
@Mat4m0
Mat4m0 merged commit 00c11d2 into main Nov 4, 2026
18 checks passed
@Mat4m0
Mat4m0 deleted the ci/retry-runner-shortage branch November 4, 2026 06:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant