Repository navigation
fix(harness): three measurement and run-integrity defects - #42
Merged
Merged
Conversation
Grouped because they share one test file and none is big enough to be worth a
PR of its own. Each has a regression test that was confirmed to FAIL against
main before the fix and pass after.
1. Token totals were inflated for codex and strands on vLLM (issue #200).
Those agents report input_tokens as the WHOLE prompt, and their counts are
declared disjoint, so everything downstream sums input + output + cache_read +
cache_write. That is right when the cache figure comes from the agent's own
API. When it comes from vLLM's Prometheus counters instead, cache_read is a
SUBSET of input_tokens, so the cached prompt was counted twice: a 1000-token
prompt with 800 cached totalled 1900 instead of 1100. _summary_metrics now
moves the cached part out of input and records no separate cache write, since
the uncached prefill IS the remaining input, and sources.input_tokens says so.
A run that never reported an input count keeps the old shape, because there is
no whole prompt to split.
2. A retry deleted finished work.
_summary_is_retryable only refused to retry on ok or error_max_turns. A task
that wrote all six artifacts and then exited on a non-stop reason -- omp can end
on stopReason "toolUse" straight after its final summary -- was classed
transient, and the retry clears the task's artifacts before re-running. So the
one case where the work was already complete is the case where it got thrown
away. A task with design and patch both on disk is now never retried: the judge
scores the files, not the exit status.
3. model_to_slug missed three of Bedrock's four region prefix forms.
The regex was ^[a-z]{2}\. so us. and eu. stripped, while global., apac. and
us-gov. did not -- and global. is the form this repo's own docs recommend, for
the 10% lower cross-region rate. The slug names the artifact folder the /swe
skill writes to, so a global.-prefixed model wrote its artifacts under
global.anthropic.claude-.../ while the harness and the judge looked under
claude-.../. A bare <vendor>.<model> such as moonshotai.kimi-k2-thinking has no
region component and is still left alone (issue #205).
642 tests pass, 7 new. ruff, ruff format, mypy and bandit (0 issues at every
severity) clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Grouped because they share one test file and none is big enough to warrant its own PR. Each regression test was confirmed to fail against
mainbefore the fix and pass after — verified by reverting the sources while keeping the tests:1. Token totals inflated ~2× for codex and strands on vLLM
codex,strandsandstrands-harnessreportinput_tokensas the whole prompt, and their counts are declared disjoint, so everything downstream sumsinput + output + cache_read + cache_write.That's correct when the cache figure comes from the agent's own API. When it comes from vLLM's Prometheus counters instead,
cache_readis a subset ofinput_tokens, so the cached prompt gets counted twice:_summary_metricsnow moves the cached part out of input and records no separate cache write — the uncached prefill is the remaining input — andsources.input_tokensstates the subtraction. A run that never reported an input count keeps the old shape, since there's no whole prompt to split.This matters beyond the token count: on the self-hosted path, cost per task is derived from tokens processed.
2. A retry deleted finished work
_summary_is_retryableonly refused onokorerror_max_turns. A task that wrote all six artifacts and then exited on a non-stopreason — omp can end on stopReasontoolUsestraight after its final summary — was classed transient. And a retry clears the task's artifacts before re-running.So the one case where the work was already complete is the case where it got thrown away. A task with design and patch both on disk is now never retried: the judge scores the files, not the exit status.
3.
model_to_slugmissed three of Bedrock's four region prefix formsThe regex was
^[a-z]{2}\., sous.andeu.stripped whileglobal.,apac.andus-gov.did not — andglobal.is the form this repo's own docs recommend, for the 10% lower cross-region rate.The slug names the artifact folder the
/sweskill writes to, so aglobal.-prefixed model wrote its artifacts underglobal.anthropic.claude-.../while the harness and the judge looked underclaude-.../.That last case is deliberate: a bare
<vendor>.<model>has no region component, so there's no prefix to strip (issue #205). The fix is scoped to the four real region forms rather than "strip anything before the second dot".Checks
642 tests pass, 7 new.
ruff,ruff format,mypy,bandit(0 issues at every severity) clean. The one failure,test_judge_common.test_lowercase_preferred_on_collision, fails onmaintoo — it assumes a case-sensitive filesystem.Remaining from the audit
.gitignorenot ignoring run output orrunner.yaml(while four docs claim it does),CLAUDE.mdnot importingAGENTS.md, and the omp fuzzy model-match guard. Then versioning.