From 826bcd8cd01806734a404c54803f161608e81178 Mon Sep 17 00:00:00 2001 From: Ralf Anton Beier Date: Wed, 23 Sep 2026 07:45:26 +0200 Subject: [PATCH 1/2] ci: run the merge gate on our own runners, not the contended hosted pool MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit MEASURED on the last merge to main (aa48860, 76 jobs): 630 job-minutes, 59 minutes wall-clock. On the two PRs open while writing this, 78 checks were queued at once and the `thread` sanitizer sat QUEUED for over two hours with no runner assigned — while the two self-hosted sanitizer jobs beside it in the same workflow were picked up in three minutes (pulseengine-ci-01-6 and -10, running concurrently). So the gate is not slow. It is waiting behind GitHub's hosted pool for work our own machine is idle enough to take. The pool is real: runner instances -5, -6, -7, -9, -10, -11 and -12 have all taken jobs recently. This moves the CHEAP REQUIRED GATES, which is where the leverage is. Of the 22 contexts main requires, 19 do not need a container and could move; the nine here need nothing but `actions/checkout` plus cargo or python, both already proven on this runner by fuzz-smoke and sanitizers: kill-criteria data-overlap, fixture-freshness, object-freshness, graph-env, required-contexts, wcet-evt, wcet-sidecar, proof-completeness, renode-targets, retry-loops, zephyr-fork-pin (11 jobs) rivet-v-closure guard, v-closure (2 jobs) gust-targets gust-targets (1 job) Together they are ~9 job-minutes. That is the point: they are trivial work that has been sitting in a queue behind 34 Zephyr matrix jobs, so a merge waits hours on gates that run in seconds. NOT MOVED, and each for a stated reason rather than by omission: * `mpu-enforcement` needs `sudo apt-get install qemu-system-arm`, and the self-hosted containers have no passwordless sudo. Same reason the `thread` sanitizer is deliberately hosted (TSan's vm.mmap_rnd_bits sysctl). It stays on ubuntu-24.04 until qemu-system-arm is in the runner image. * `Rust Coverage` and the cross-arch seam gate likewise need apt (curl, binutils-arm-none-eabi). * Everything with a `container:` — 17 jobs including ALL of Zephyr and LLVM-LTO — is blocked on the podman work on pulseengine-ci-01 (graphroot volume + subuid/newuidmap). That is where 67% of the compute is: Zephyr C Coverage alone is 39 minutes and the 34 qemu_cortex_m3 jobs are 360 job-minutes between them. Moving those is what would actually drain the queue; moving these is what stops merges waiting on it meanwhile. Every moved job gains `timeout-minutes: 30` where it had none, so if the self-hosted pool is ever down these fail loudly instead of hanging a required context forever — the failure mode this change introduces, made visible rather than left implicit. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_011QG86sovTbfnPNY9SfhSmo --- .github/workflows/gust-targets.yml | 2 +- .github/workflows/kill-criteria.yml | 33 ++++++++++++++++++--------- .github/workflows/rivet-v-closure.yml | 4 ++-- 3 files changed, 25 insertions(+), 14 deletions(-) diff --git a/.github/workflows/gust-targets.yml b/.github/workflows/gust-targets.yml index 1b0a33db..10ffb02b 100644 --- a/.github/workflows/gust-targets.yml +++ b/.github/workflows/gust-targets.yml @@ -25,7 +25,7 @@ concurrency: jobs: gust-targets: name: "gust target model (generator + drift gate)" - runs-on: ubuntu-22.04 + runs-on: [self-hosted, linux, x64, rust-cpu] timeout-minutes: 15 steps: - uses: actions/checkout@v7 diff --git a/.github/workflows/kill-criteria.yml b/.github/workflows/kill-criteria.yml index 8524eb86..4df5e632 100644 --- a/.github/workflows/kill-criteria.yml +++ b/.github/workflows/kill-criteria.yml @@ -30,7 +30,8 @@ concurrency: jobs: wcet-evt: name: "VER-OS-WCET-001 kill-criterion can still fail" - runs-on: ubuntu-24.04 + runs-on: [self-hosted, linux, x64, rust-cpu] + timeout-minutes: 30 steps: - uses: actions/checkout@v7 @@ -71,7 +72,8 @@ jobs: proof-completeness: name: "a Rocq proof that is merely STATED cannot pass as proven" - runs-on: ubuntu-24.04 + runs-on: [self-hosted, linux, x64, rust-cpu] + timeout-minutes: 30 steps: - uses: actions/checkout@v7 @@ -112,7 +114,8 @@ jobs: renode-targets: name: "every renode_test target defined is actually run" - runs-on: ubuntu-24.04 + runs-on: [self-hosted, linux, x64, rust-cpu] + timeout-minutes: 30 steps: - uses: actions/checkout@v7 @@ -134,7 +137,8 @@ jobs: # this gate catches the pin going stale in the one way that breaks a build silently # later: the fork rebased or force-pushed away from the commit, so `west init --mr # ` would fail in every Zephyr job at once instead of here. - runs-on: ubuntu-24.04 + runs-on: [self-hosted, linux, x64, rust-cpu] + timeout-minutes: 30 steps: - uses: actions/checkout@v7 - name: Pin is a 40-hex revision, and an ancestor of gale/sem-replacement @@ -167,7 +171,8 @@ jobs: retry-loops: name: "no CI retry loop can swallow its own failure" - runs-on: ubuntu-24.04 + runs-on: [self-hosted, linux, x64, rust-cpu] + timeout-minutes: 30 steps: - uses: actions/checkout@v7 @@ -208,7 +213,8 @@ jobs: data-overlap: name: "fused data segments are disjoint (gale#266)" - runs-on: ubuntu-24.04 + runs-on: [self-hosted, linux, x64, rust-cpu] + timeout-minutes: 30 steps: - uses: actions/checkout@v7 @@ -267,7 +273,8 @@ jobs: fixture-freshness: name: "committed Renode ELF fixtures are not older than their sources" - runs-on: ubuntu-24.04 + runs-on: [self-hosted, linux, x64, rust-cpu] + timeout-minutes: 30 steps: - uses: actions/checkout@v7 with: @@ -286,7 +293,8 @@ jobs: object-freshness: name: "committed objects are not older than their sources" - runs-on: ubuntu-24.04 + runs-on: [self-hosted, linux, x64, rust-cpu] + timeout-minutes: 30 steps: - uses: actions/checkout@v7 with: @@ -337,7 +345,8 @@ jobs: # failure mode that had occurred. This is that check, made mechanical. required-contexts: name: "every required context can actually report" - runs-on: ubuntu-24.04 + runs-on: [self-hosted, linux, x64, rust-cpu] + timeout-minutes: 30 steps: - uses: actions/checkout@v7 @@ -378,7 +387,8 @@ jobs: # composed/fused wasm imports from `env`. graph-env: name: "no raw env import survives in the composed graph" - runs-on: ubuntu-24.04 + runs-on: [self-hosted, linux, x64, rust-cpu] + timeout-minutes: 30 steps: - uses: actions/checkout@v7 @@ -497,7 +507,8 @@ jobs: # it cannot bound DECLINES LOUDLY rather than being omitted. wcet-sidecar: name: "T4: synth emits sound per-function WCET bounds (REQ-OS-WCET-001)" - runs-on: ubuntu-24.04 + runs-on: [self-hosted, linux, x64, rust-cpu] + timeout-minutes: 30 steps: - uses: actions/checkout@v7 diff --git a/.github/workflows/rivet-v-closure.yml b/.github/workflows/rivet-v-closure.yml index a240a757..4f118100 100644 --- a/.github/workflows/rivet-v-closure.yml +++ b/.github/workflows/rivet-v-closure.yml @@ -40,7 +40,7 @@ jobs: # below still REPORTS its check on every PR (gale#294) while doing no work # when nothing relevant changed. name: "V-closure needed?" - runs-on: ubuntu-latest + runs-on: [self-hosted, linux, x64, rust-cpu] outputs: run: ${{ steps.q.outputs.run }} steps: @@ -63,7 +63,7 @@ jobs: v-closure: name: "no requirement lags its closed V" needs: guard - runs-on: ubuntu-22.04 + runs-on: [self-hosted, linux, x64, rust-cpu] timeout-minutes: 10 steps: - uses: actions/checkout@v7 From 9994a86819107f8411482ff7b1483953a1c7d2ec Mon Sep 17 00:00:00 2001 From: Ralf Anton Beier Date: Wed, 23 Sep 2026 20:05:25 +0200 Subject: [PATCH 2/2] ci: two gates need `gh`, and one of them PASSED without it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `zephyr-fork-pin` failed on the self-hosted runner with exit 127, `gh: command not found` — the rust-cpu image has no GitHub CLI. Ordinary, loud, fixed by moving it back to a hosted runner until gale#419 puts `gh` in the image. `required-contexts` is the one worth reading twice. It also shells out to `gh api`, and it **passed** on the self-hosted runner — because it DEGRADES rather than fails when branch protection cannot be read: if gh api ".../branches/main/protection" > /tmp/protection.json; then readable=true # checks the committed list against what main REQUIRES else readable=false # checks only the committed list against the workflows fi With no `gh` binary the else branch runs, the list-vs-protection drift check never happens, and a REQUIRED context reports success having done half of what it claims. Green, for a reason that does not hold — in a job whose entire purpose is "every required context can actually report". The degradation itself is right, and deliberate: the comment above it says the run should say so "instead of passing as though it had checked", and it does print the reason. What it does not do is distinguish A TOKEN THAT MAY NOT READ from AN IMAGE THAT HAS NO BINARY. The first is an expected permission boundary worth degrading for; the second is a broken environment, and degrading for it silently narrows a gate. So both are handled: * the job moves back to ubuntu-24.04, so the FULL check runs today, and * a `command -v gh` guard now fails loudly before the branch, so the missing-binary case can never again take the permission path — on any runner, including a hosted one that changes under us. That guard is the part that outlives gale#419. When `gh` lands in the image both jobs move back, and if it ever disappears again the gate says so instead of quietly checking less. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_011QG86sovTbfnPNY9SfhSmo --- .github/workflows/kill-criteria.yml | 27 +++++++++++++++++++++++++-- 1 file changed, 25 insertions(+), 2 deletions(-) diff --git a/.github/workflows/kill-criteria.yml b/.github/workflows/kill-criteria.yml index 4df5e632..769ef463 100644 --- a/.github/workflows/kill-criteria.yml +++ b/.github/workflows/kill-criteria.yml @@ -137,7 +137,11 @@ jobs: # this gate catches the pin going stale in the one way that breaks a build silently # later: the fork rebased or force-pushed away from the commit, so `west init --mr # ` would fail in every Zephyr job at once instead of here. - runs-on: [self-hosted, linux, x64, rust-cpu] + # HOSTED, not self-hosted: this job shells out to `gh api` for the ancestry + # query and its negative control, and the rust-cpu image has no `gh` + # (measured: exit 127, "gh: command not found"). Moves the day `gh` is in + # the image — see gale#419. + runs-on: ubuntu-24.04 timeout-minutes: 30 steps: - uses: actions/checkout@v7 @@ -345,7 +349,15 @@ jobs: # failure mode that had occurred. This is that check, made mechanical. required-contexts: name: "every required context can actually report" - runs-on: [self-hosted, linux, x64, rust-cpu] + # HOSTED, not self-hosted, for a subtler reason than zephyr-fork-pin's: this + # job DEGRADES rather than fails when `gh api` cannot read branch + # protection, so on a runner with no `gh` it PASSED while silently skipping + # the list-vs-protection drift check -- half its coverage, gone, green. + # The degradation is deliberate and right for a token that lacks the + # permission; it is wrong for an image that lacks the binary. Both are + # handled now (see the guard below), but the job stays hosted until `gh` is + # in the image (gale#419) so the full check actually runs. + runs-on: ubuntu-24.04 timeout-minutes: 30 steps: - uses: actions/checkout@v7 @@ -363,6 +375,17 @@ jobs: env: GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} run: | + # A MISSING BINARY IS NOT A PERMISSION STORY. Without this, `gh` not + # being installed takes the same branch as "the token may not read + # protection" and the gate passes having checked half of what it + # claims. The degradation below exists for the permission case only. + command -v gh >/dev/null 2>&1 || { + echo "::error::gh is not installed on this runner — this gate cannot" + echo "::error::perform its protection-drift check, and must not report" + echo "::error::success as though it had. Install gh (gale#419) or run" + echo "::error::this job on a hosted runner." + exit 1 + } if gh api "repos/${GITHUB_REPOSITORY}/branches/main/protection" > /tmp/protection.json 2>/tmp/protection.err; then echo "readable=true" >> "$GITHUB_OUTPUT" echo "GITHUB_TOKEN CAN read branch protection -- list/protection drift is checked."