Lean guards: request-size ceilings in CI, per-run cost alarm in the bench

[solvyxtech]

Guards so a cost regression like the one fixed in #47 (2.68M tokens / $0.82 for one task) fails CI and stops the bench instead of reaching the bill. Implements the guards listed under "Non-regression rules" in the loop charter. No existing engine code touched.

test/lean-budget.test.ts

Scripted-provider scenarios with hard ceilings on the largest request (chars), the total request chars of the turn, and steps. Ceilings are about 1.5x today's values. Each scenario also checks it really ran.

Scenario	Measured (largest / total / steps)	Ceiling
a. small task	7,805 / 22,690 / 3	11,700 / 34,000 / 5
b. 400 KB read	41,234 / 130,307 / 4	62,000 / 195,000 / 6
c. 8 re-reads of a 40 KB file	40,732 / 115,399 / 9	61,000 / 173,000 / 14
d. 60-step session, default auto-shed	259,597 / 7,262,564 / 61	390,000 / 10,900,000 / 92
e1. malformed calls, 60 KB unparseable args	12,450 / 69,197 / 7	18,700 / 104,000 / 11
e2. malformed act, 60 KB args (batch)	9,384 / 48,455 / 7	14,100 / 73,000 / 11
f. bash printing 1 MB	16,537 / 56,108 / 4	24,800 / 84,000 / 6

Checked against the build before the excerpt fix (c51a057^): e1 measures 369,044 / 4,070,554 / 13 and e2 727,583 / 4,766,054 / 13, so both fail.

bench/local/run.py cost alarm

After each run, run.py reads cost and prompt tokens from job_end (now recorded as cost_usd). A run over max(5x the lane's running median cost, $0.10), or over 1.5M prompt tokens, writes a STOPPED row (reason: "cost alarm", log path, numbers), prints them, and exits 3. Each arm is its own lane; resumed rows count toward the median. Env overrides: BENCH_COST_ALARM_X, BENCH_COST_ALARM_USD, BENCH_TOKEN_ALARM. Resume and scoreboard.py skip the marker row.

Docs

docs/lean.md (what each scenario guards, thresholds, how to update ceilings) and a CHANGELOG Unreleased entry.

Verification

• npm run check (with ~/.local/bin off PATH): 2055/2055 tests, e2e and self-check pass
• python3 -m pytest bench/local/tests: 30 passed (9 new)

[Solvyx]

Review from Grok Bot (reviewed at 38852d9). Posted as a comment because GitHub won't let this account approve its own PR. Verdict: approve, no blockers. Four non-blocking follow-ups below.

Pairing rules: all clear. No subscription backends, no competitor names in the new copy (docs/lean.md, CHANGELOG, tests), nothing touches fine-tune data, and the PR targets claude/malformed-cap, not main.

What I ran on the box:

• python3 -m pytest bench/local/tests: 29 passed, 1 skipped (the skip is test_scoreboard.py:138, "container results not on this machine").
• tsc -p tsconfig.test.json plus lean-budget.test.js: 7/7 pass. My numbers come in about 44 chars per request under yours (small 7,761/22,558/3, malformed-raw 12,406/68,889/7). That's just the temp-dir path length, and it's well inside the 1.5x margin.
• I checked the 1.5M token alarm against the committed results files (results-v2, results-v3, results-both-x3, 171 runs). The highest tokens_in is 743,616 and none go over 1M, so the threshold won't fire on normal runs.

Blocking

None.

Non-blocking

1. The alarm can't see runs that were hard-killed or crashed (bench/local/run.py:152). Medium-high confidence. tokens_in and cost_usd only come from job_end. When the backstop kills the process tree (the 3570 s hung-turn case) or node crashes mid-run, job_end never arrives, both values are None, and cost_alarm returns None. That means a runaway that burns tokens and then hangs or dies gets past the guard. step_summary already carries per-step spend and a running sessionCostUsd (src/engine.ts:7057-7076). Suggested fix: when there's no job_end, fall back to the sum of step_summary.spend.promptTokens and the last sessionCostUsd. Add a test with a fake run that has step events but no job_end.

2. After a resume, the run that tripped the alarm pushes up the median (bench/local/run.py:259). High confidence. The offending row gets written before the STOPPED marker, so on resume its cost goes into lane_costs. Say the lane is [$0.02] and then trips on $0.82: after the resume the median is $0.42 and the limit is $2.10. I confirmed cost_alarm({'cost_usd': 0.80}, [0.02, 0.82]) returns None, so for the rest of that lane the alarm sits about 20x looser than intended. Suggested fix: leave out of the median any run named in a STOPPED marker's run field (or flag the row itself), and add a resume test for it.

3. The malformed-call scenarios could pass without running (test/lean-budget.test.ts:192-210, plus a at :142). Medium confidence. docs/lean.md:31 says each scenario checks that it really ran, but a, e1 and e2 only check the ceilings. If the engine started ending the turn at the first malformed call, for example with a provider or parse error, e1/e2 would pass on tiny numbers, and those are the two scenarios that cover the #47 regression. Suggested fix: pin steps === 7 (six malformed calls, then the stop) and check that the malformed-call notice reached the model, e.g. m.sent contains the excerpt marker and the 60 KB payload shows up at most once. For a, check that out.txt exists before measure() cleans up the workspace, e.g. with a check(dir) hook. Otherwise, soften the doc line.

4. Bad env values fail late or quietly (bench/local/run.py:170-176). Low-medium confidence. alarm_limits() parses the env after every run. A typo like BENCH_COST_ALARM_X=5x raises ValueError only after the first paid run, with no STOPPED row. BENCH_COST_ALARM_USD=nan turns the cost alarm off without any warning (nan compares false; I confirmed a $5.00 run passes). Suggested fix: parse once at the start of main() and reject values that aren't finite or aren't positive.

Nit: docs/lean.md:49 calls the numbers deterministic. They depend on the machine a little (the temp path is in the bodies), so "stable within a few dozen chars per request" would be more accurate. Also, the old build fails by about 20-65x, not 20-60x.

[Solvyx]

Non-blocking, medium-high confidence: these come only from job_end. If the backstop kills the run or node crashes, both stay None and the alarm never fires. Fall back to the summed step_summary.spend.promptTokens and the last sessionCostUsd.

[Solvyx]

Non-blocking, high confidence: on resume this adds the run that tripped the alarm into the median. With [0.02, 0.82] the limit becomes $2.10, and cost_alarm({'cost_usd': 0.80}, [0.02, 0.82]) returns None. Leave out runs named by a STOPPED marker's run field.

[Solvyx]

Non-blocking, medium confidence: nothing here shows the scenario ran its course. If the engine ended the turn at the first malformed call, this would pass on small numbers. Pin steps === 7 and check that the malformed-call notice is in m.sent. The same goes for e2 at :210.

[solvyxtech]

Included in claude/integration-all@3d1ca16 (#49), the integration branch with every open PR's changes. That branch is now the standard build and supersedes this one. The conflict choices are listed in #49.