ci: run the merge gate on our own runners, not the contended hosted pool

[avrabe]

The gate is not slow. It is queued behind GitHub's hosted pool for work our own machine is idle enough to take.

Measured, not assumed

• Last merge to main (aa48860): 76 jobs, 630 job-minutes, 59 min wall-clock.
• While writing this, 78 checks were queued at once across two PRs.
• The thread sanitizer sat QUEUED for over two hours with no runner assigned — while the two self-hosted sanitizer jobs in the same workflow were picked up in three minutes (pulseengine-ci-01-6 and -10, running concurrently).
• The pool is real: instances -5, -6, -7, -9, -10, -11, -12 have all taken jobs recently. Today 1 of 58 job definitions targets it.

What moves here, and why these

The cheap required gates — that is where the leverage is. Of the 22 contexts main requires, 19 need no container and could move; these nine need nothing but actions/checkout plus cargo or python, both already proven on this runner by fuzz-smoke and sanitizers:

workflow	jobs
kill-criteria	data-overlap, fixture-freshness, object-freshness, graph-env, required-contexts, wcet-evt, wcet-sidecar, proof-completeness, renode-targets, retry-loops, zephyr-fork-pin (11)
rivet-v-closure	guard, v-closure (2)
gust-targets	gust-targets (1)

Together they are ~9 job-minutes. That is the point: trivial work sitting in a queue behind 34 Zephyr matrix jobs, so a merge waits hours on gates that run in seconds.

Not moved — each for a stated reason, not by omission

• mpu-enforcement needs sudo apt-get install qemu-system-arm, and the self-hosted containers have no passwordless sudo. Same class as the deliberately-hosted thread sanitizer (TSan's vm.mmap_rnd_bits sysctl). Stays on ubuntu-24.04 until qemu-system-arm is in the runner image.
• Rust Coverage and the cross-arch seam gate likewise need apt (curl, binutils-arm-none-eabi).
• Everything with a container: — 17 jobs including all of Zephyr and LLVM-LTO — is blocked on the podman work on pulseengine-ci-01 (graphroot volume + subuid/newuidmap). That is where 67% of the compute is: Zephyr C Coverage alone is 39 minutes, and the 34 qemu_cortex_m3 jobs are 360 job-minutes between them. Moving those is what would drain the queue; moving these is what stops merges waiting on it meanwhile.

The failure mode this introduces, made visible

Every moved job gains timeout-minutes: 30 where it had none. If the self-hosted pool is ever down, these fail loudly instead of hanging a required context forever. runs-on has no fallback, so that is the tradeoff being taken deliberately rather than discovered later.

🤖 Generated with Claude Code

https://claude.ai/code/session_011QG86sovTbfnPNY9SfhSmo

[avrabe]

[fathom (gale) — it works, with an A/B inside a single run, and one honest wrinkle]

kill-criteria run 35832723443, created 07:37:53Z. Twelve minutes later:

job	runner	wait	run
a Rocq proof that is merely STATED cannot pass	pulseengine-ci-01-5	10.8 min	0.2 min ✅
VER-OS-WCET-001 kill-criterion can still fail	pulseengine-ci-01-5	11.1 min	0.2 min ✅
every renode_test target defined is actually run	pulseengine-ci-01-7	10.9 min	0.2 min ✅
committed Renode ELF fixtures are not older…	pulseengine-ci-01-10	11.2 min	0.3 min ✅
no CI retry loop can swallow its own failure	pulseengine-ci-01-5	11.3 min	0.2 min ✅
a denied write really faults (left on ubuntu-24.04)	UNASSIGNED	>12 min	still queued

The last row is the control, and it is in the same run as the others: the one job I deliberately did not move — because it needs sudo apt-get install qemu-system-arm — is the one still waiting for a runner. Earlier today a hosted job (thread sanitizer) sat queued for over two hours.

The gates take twelve seconds. That is the whole finding. Work measured in seconds was waiting behind a queue measured in hours.

The wrinkle, because it changes the next ask

Our own pool is not instant either: ~11 minutes of queue before pickup. Only four instances took work in this run — -5 (4 jobs), -10 (3), -6 (1), -7 (1) — although -9, -11 and -12 have taken jobs on other days. So the effective width right now is about four, and with 14 jobs moved here that is ~3 waves.

So this PR converts hours of hosted queue into minutes of our own queue, which is the win, but the residual 11 minutes is pool width and not something the repo can fix. Two levers, both on the host:

1. More runner instances online — the cheapest way to cut the 11 minutes, and it needs no repo change.
2. 67% of CI compute cannot move to our own runners: 17 container: jobs blocked on podman-in-podman on pulseengine-ci-01 #419's podman work — unrelated to this wait, but it is what lets the 67% of compute (Zephyr) leave the hosted pool at all.

I would rather state the 11 minutes than quietly present the 12-second run time as the result.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!