Skip to content

test(fresh-baseline): stop asserting a measured duration is exactly 0 - #191

Merged
myselfsiddharth merged 1 commit into
mainfrom
fix/fresh-baseline-walltime-flake
Sep 6, 2026
Merged

test(fresh-baseline): stop asserting a measured duration is exactly 0#191
myselfsiddharth merged 1 commit into
mainfrom
fix/fresh-baseline-walltime-flake

Conversation

@myselfsiddharth

Copy link
Copy Markdown
Contributor

Summary

tests/unit/fresh-baseline-runner.test.ts had a load-dependent flake. It surfaced as a red CI on #186 (@anthropic-ai/sdk 0.117.1 → 0.122.0), which is innocent — that PR is fine to merge once this lands.

The StubFreshBaselineClient case asserted:

expect(result.cost_fresh).toEqual({ tokens_in: 0, tokens_out: 0, wall_clock_ms: 0 });

The stub returns normally rather than throwing, so it takes FreshBaselineRunner's measured path (measureWallClock, src/runner/fresh-baseline-runner.ts:120) — not the forced-zeroCost() catch at :136-143. So wall_clock_ms is real elapsed time. Asserting it equals 0 asserts the machine was fast, not that the code was honest.

Evidence

Condition Result
#186's branch (SDK 0.122.0) locally, idle 11/11 pass — SDK is not the cause
Unmodified main, idle, 12 runs 0 failures — my machine rounds to 0
Unmodified main, 12-way CPU saturation, 10 runs 1 failure — reproduced
This branch, same saturation, 12 runs 0 failures

What the test still defends

The claim is "never a silent success," so tokens are still pinned to exactly 0 and task_success to false. Only the duration assertion is relaxed — to non-negative and finite, which is the real invariant for a measured value.

Deliberately not changed

The three sibling wall_clock_ms: 0 assertions (lines 134, 140, 163) are left alone. Those cases throw, so zeroCost() makes 0 a genuine forced invariant rather than a timing accident. Relaxing them would weaken real coverage.

The StubFreshBaselineClient case asserted
cost_fresh.wall_clock_ms === 0. The stub returns normally instead of
throwing, so it takes FreshBaselineRunner's measured path
(measureWallClock, fresh-baseline-runner.ts:120) rather than the
forced-zeroCost() catch at :136-143. wall_clock_ms is therefore real
elapsed time, and pinning it to 0 asserts the machine was fast, not
that the code was honest.

Surfaced as a red CI on #186 (@anthropic-ai/sdk 0.117.1 -> 0.122.0),
which had nothing to do with it. Reproduced with the bump applied
locally: 11/11 pass idle. Reproduced on unmodified main under CPU
saturation: 1/10 failures. 0/12 after this change under the same load.

The three sibling wall_clock_ms === 0 assertions are left alone: those
cases throw, so zeroCost() makes 0 a genuine invariant rather than a
timing accident.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@myselfsiddharth
myselfsiddharth requested a review from a team as a code owner September 6, 2026 08:08
@github-actions github-actions Bot added the size/S <= 50 changed lines label Sep 6, 2026
@github-actions
github-actions Bot requested a review from OM152002 September 6, 2026 08:09
@myselfsiddharth
myselfsiddharth merged commit 5361c37 into main Sep 6, 2026
13 checks passed
@myselfsiddharth
myselfsiddharth deleted the fix/fresh-baseline-walltime-flake branch September 6, 2026 08:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/S <= 50 changed lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant