31 of 53 CI jobs have no timeout at job or step level — on a repo whose bottleneck is runner starvation #382

[fathom (gale) — measured while waiting on a 22-minute qemu job]

sched_metairq (qemu_cortex_m3) held #380 for ~22 minutes. It was not hung — it
completed and the PR went green. But checking whether it could hang turned up
something worth recording.

The measurement

Counting jobs with no timeout-minutes at either job or step level:

workflow                         jobs  job-TO  step-TO  UNBOUNDED
bazel-tests.yml                     3       0        0   3
compliance.yml                      1       0        0   1
coverage.yml                        2       0        0   2
drv-components.yml                  1       0        0   1
drv-cross-arch.yml                  1       0        0   1
engine-bench-smoke.yml              1       0        0   1
formal-verification.yml             5       0        4   1
kill-criteria.yml                   9       0        0   9
llvm-lto.yml                        2       0        0   2
pages.yml                           2       0        0   2
release-wasm.yml                    1       0        0   1
release.yml                         2       0        0   2
rivet-v-closure.yml                 2       1        0   1
zephyr-tests.yml                    4       0        0   4

jobs with NO timeout at job or step level: 31 of 53

Only formal-verification.yml uses step-level timeouts (4 of 5 jobs), and
rivet-v-closure.yml has the single job-level one. Everything else runs to
GitHub's 6-hour default.

Why it matters here specifically

This repo's documented bottleneck is runner starvation — the 2-PR ceiling exists
because ~75 jobs per PR against ~1-2 concurrent runners already serialises the
queue (measured earlier: 24 runs queued, 1 in progress, oldest waiting 11
minutes). A single job that hangs holds a runner for six hours against that.

The exposure is not hypothetical in kind: qemu tests that wait on serial output
are exactly the shape that hangs when the guest never reaches its exit path, and
zephyr-tests.yml — 4 jobs, the largest matrix — has no timeout at any level.

Not proposing a blanket number

Timeouts should be sized from measured durations, not guessed, or they become a
new flake source. The honest next step is to measure the p95 per job over recent
runs and set ~2-3x that. I have not done the measurement, so I am not opening a
PR with invented numbers.

Two jobs are worth doing first regardless, because they are the ones that block
everything else: zephyr-tests.yml (observed 22 min for one qemu test) and
kill-criteria.yml (9 jobs, all short, so a generous bound costs nothing and
catches a genuine hang immediately).

Method note

My first pass at this audit was wrong and said every workflow had a timeout.
grep -c prints 0 and exits 1 when there is no match, so a
grep -c … || echo 0 idiom produced "0\n0", which never compared equal to
"0". Same output-vs-exit-status conflation the retry-loop sweep was about
(#370), in the audit that was looking for it.