test(fresh-baseline): stop asserting a measured duration is exactly 0 - #191
Merged
Merged
Conversation
The StubFreshBaselineClient case asserted cost_fresh.wall_clock_ms === 0. The stub returns normally instead of throwing, so it takes FreshBaselineRunner's measured path (measureWallClock, fresh-baseline-runner.ts:120) rather than the forced-zeroCost() catch at :136-143. wall_clock_ms is therefore real elapsed time, and pinning it to 0 asserts the machine was fast, not that the code was honest. Surfaced as a red CI on #186 (@anthropic-ai/sdk 0.117.1 -> 0.122.0), which had nothing to do with it. Reproduced with the bump applied locally: 11/11 pass idle. Reproduced on unmodified main under CPU saturation: 1/10 failures. 0/12 after this change under the same load. The three sibling wall_clock_ms === 0 assertions are left alone: those cases throw, so zeroCost() makes 0 a genuine invariant rather than a timing accident. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
tests/unit/fresh-baseline-runner.test.tshad a load-dependent flake. It surfaced as a red CI on #186 (@anthropic-ai/sdk0.117.1 → 0.122.0), which is innocent — that PR is fine to merge once this lands.The
StubFreshBaselineClientcase asserted:The stub returns normally rather than throwing, so it takes
FreshBaselineRunner's measured path (measureWallClock,src/runner/fresh-baseline-runner.ts:120) — not the forced-zeroCost()catch at:136-143. Sowall_clock_msis real elapsed time. Asserting it equals0asserts the machine was fast, not that the code was honest.Evidence
main, idle, 12 runsmain, 12-way CPU saturation, 10 runsWhat the test still defends
The claim is "never a silent success," so tokens are still pinned to exactly
0andtask_successtofalse. Only the duration assertion is relaxed — to non-negative and finite, which is the real invariant for a measured value.Deliberately not changed
The three sibling
wall_clock_ms: 0assertions (lines 134, 140, 163) are left alone. Those cases throw, sozeroCost()makes0a genuine forced invariant rather than a timing accident. Relaxing them would weaken real coverage.