Skip to content

fix(harness): three measurement and run-integrity defects - #42

Merged
shekharprateek merged 1 commit into
mainfrom
fix/measurement-and-run-integrity
Oct 26, 2026
Merged

shekharprateek merged 1 commit into
mainfrom
fix/measurement-and-run-integrity

Conversation

@shekharprateek

Copy link
Copy Markdown
Contributor

Grouped because they share one test file and none is big enough to warrant its own PR. Each regression test was confirmed to fail against main before the fix and pass after — verified by reverting the sources while keeping the tests:

sources reverted to main          fixes restored
  BedrockSlugPrefix        FAILED   →  OK
  RetryGuardsFinishedWork  FAILED   →  OK
  DisjointAgentVllmCache   FAILED   →  OK

1. Token totals inflated ~2× for codex and strands on vLLM

codex, strands and strands-harness report input_tokens as the whole prompt, and their counts are declared disjoint, so everything downstream sums input + output + cache_read + cache_write.

That's correct when the cache figure comes from the agent's own API. When it comes from vLLM's Prometheus counters instead, cache_read is a subset of input_tokens, so the cached prompt gets counted twice:

1000-token prompt, 800 cached
Before 1000 + 100 + 800 + 200 = 1900
After 200 + 100 + 800 + 0 = 1100

_summary_metrics now moves the cached part out of input and records no separate cache write — the uncached prefill is the remaining input — and sources.input_tokens states the subtraction. A run that never reported an input count keeps the old shape, since there's no whole prompt to split.

This matters beyond the token count: on the self-hosted path, cost per task is derived from tokens processed.

2. A retry deleted finished work

_summary_is_retryable only refused on ok or error_max_turns. A task that wrote all six artifacts and then exited on a non-stop reason — omp can end on stopReason toolUse straight after its final summary — was classed transient. And a retry clears the task's artifacts before re-running.

So the one case where the work was already complete is the case where it got thrown away. A task with design and patch both on disk is now never retried: the judge scores the files, not the exit status.

3. model_to_slug missed three of Bedrock's four region prefix forms

The regex was ^[a-z]{2}\., so us. and eu. stripped while global., apac. and us-gov. did not — and global. is the form this repo's own docs recommend, for the 10% lower cross-region rate.

The slug names the artifact folder the /swe skill writes to, so a global.-prefixed model wrote its artifacts under global.anthropic.claude-.../ while the harness and the judge looked under claude-.../.

us.anthropic.claude-opus-5       → claude-opus-5
global.anthropic.claude-opus-5   → claude-opus-5   (was: unchanged)
apac.anthropic.claude-opus-5     → claude-opus-5   (was: unchanged)
us-gov.anthropic.claude-opus-5   → claude-opus-5   (was: unchanged)
moonshotai.kimi-k2-thinking      → moonshotai.kimi-k2-thinking   (correctly untouched)

That last case is deliberate: a bare <vendor>.<model> has no region component, so there's no prefix to strip (issue #205). The fix is scoped to the four real region forms rather than "strip anything before the second dot".

Checks

642 tests pass, 7 new. ruff, ruff format, mypy, bandit (0 issues at every severity) clean. The one failure, test_judge_common.test_lowercase_preferred_on_collision, fails on main too — it assumes a case-sensitive filesystem.

Remaining from the audit

.gitignore not ignoring run output or runner.yaml (while four docs claim it does), CLAUDE.md not importing AGENTS.md, and the omp fuzzy model-match guard. Then versioning.

Grouped because they share one test file and none is big enough to be worth a
PR of its own. Each has a regression test that was confirmed to FAIL against
main before the fix and pass after.

1. Token totals were inflated for codex and strands on vLLM (issue #200).

Those agents report input_tokens as the WHOLE prompt, and their counts are
declared disjoint, so everything downstream sums input + output + cache_read +
cache_write. That is right when the cache figure comes from the agent's own
API. When it comes from vLLM's Prometheus counters instead, cache_read is a
SUBSET of input_tokens, so the cached prompt was counted twice: a 1000-token
prompt with 800 cached totalled 1900 instead of 1100. _summary_metrics now
moves the cached part out of input and records no separate cache write, since
the uncached prefill IS the remaining input, and sources.input_tokens says so.
A run that never reported an input count keeps the old shape, because there is
no whole prompt to split.

2. A retry deleted finished work.

_summary_is_retryable only refused to retry on ok or error_max_turns. A task
that wrote all six artifacts and then exited on a non-stop reason -- omp can end
on stopReason "toolUse" straight after its final summary -- was classed
transient, and the retry clears the task's artifacts before re-running. So the
one case where the work was already complete is the case where it got thrown
away. A task with design and patch both on disk is now never retried: the judge
scores the files, not the exit status.

3. model_to_slug missed three of Bedrock's four region prefix forms.

The regex was ^[a-z]{2}\. so us. and eu. stripped, while global., apac. and
us-gov. did not -- and global. is the form this repo's own docs recommend, for
the 10% lower cross-region rate. The slug names the artifact folder the /swe
skill writes to, so a global.-prefixed model wrote its artifacts under
global.anthropic.claude-.../ while the harness and the judge looked under
claude-.../. A bare <vendor>.<model> such as moonshotai.kimi-k2-thinking has no
region component and is still left alone (issue #205).

642 tests pass, 7 new. ruff, ruff format, mypy and bandit (0 issues at every
severity) clean.
@shekharprateek
shekharprateek merged commit 70c6886 into main Oct 26, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant