Skip to content

fix(ci): go-test-ran ignores unrelated skips; TestRunsWait stable under load - #549

Merged
acamarata merged 3 commits into
mainfrom
fix/go-test-ran-skips
Oct 29, 2026
Merged

acamarata merged 3 commits into
mainfrom
fix/go-test-ran-skips

Conversation

@acamarata

Copy link
Copy Markdown
Contributor

Two fixes for flakes that hit many P7 verify lines.

  1. scripts/ci/go-test-ran.py: failed on any skip, but ./cmd/... always has 2 skips (also on main). It now requires a passing event for every named prefix and fails on any fail event, a non-JSON line, or a skipped test named by a prefix. Skips of other tests and package-level skips are ignored. Tests: scripts/ci/tests/test_go_test_ran.py (unittest, run by the hygiene gate).
  2. free/ci TestRunsWait: the 200 ms capped-wait subtest had to pay for store open + recovery inside its budget, so under load the budget ended first (state unknown, not running; 5 of 20 on mm4). It now uses a resolver that answers at once. The stalled resolver's look counter was a never-reset package var; it now lives on the value.

Proof: go test -count=20 -run TestRunsWait ./cmd/ on mm4 at load 11-14 passes twice; before, 5 of 20 failed.

A package run such as ./cmd/... always carries a few skips, so requiring zero
skip events failed every verify line that used the script. It now requires a
passing event for each named prefix, and still fails on any fail event, a
non-JSON line, or a skipped test whose name starts with a named prefix.
The capped-wait subtest gave a 200 ms budget to a run that had to open the
store and recover dead coordinators first, so on a loaded host the budget
ended before the first read (state unknown, not running). It now uses a
resolver that answers at once. The local-store timeout subtest accepts the
same load tail. The stalled resolver counted its looks in a package variable
that was never reset, so every repeat after the first started mid-count; the
count now belongs to the resolver value.
A budget that ended before the store opened and the first read finished made
the wait report state unknown with exit 10 for a run that exists. The first
local read now gets up to firstReadGrace (half the invoke-deadline margin)
even when the budget is shorter; later polls and remote lookups keep the
budget. The wait subtests assert running again.
@acamarata
acamarata merged commit 3d7da3f into main Oct 29, 2026
31 of 41 checks passed
@acamarata
acamarata deleted the fix/go-test-ran-skips branch October 29, 2026 20:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant