Repository navigation
Refused tool calls stay cheap: excerpt on the wire, stop after six in a row - #47
solvyxtech wants to merge 2 commits into
Conversation
… a row end the turn A worker that sent 17 malformed act calls had every one resent in full on each step: in the call itself, in the complaint quoting it, and in the shed digest. Prompts reached 630k tokens and one task cost $0.82 for a 181-byte file. The record and captures keep the full text; the wire gets an excerpt.
Solvyx
left a comment
Solvyx
left a comment
There was a problem hiding this comment.
Grok review of c51a057 (comment-only, no merge). Verdict: changes requested — one blocker, the rest is small.
The fix itself is right and well measured: excerpting refused calls on the wire, capping _unparsed and the shed digest, and a streak stop are exactly what the 630k-token run needed. I built the branch, tsc --noEmit is clean and test/malformed-cap.test.ts passes 5/5 locally. Pairing rules: clean (no subscription backends, no competitor names in code or copy, nothing touching fine-tune data; base is claude/completion-reserve, not main).
Blocking
1. A malformed-call stop is recorded as a provider failure, and the receipt says so. src/engine.ts:5965-5966 sets turnEndedBy = "provider" and calls judgeOnDisk("provider", ""). That value flows unchanged into the receipt (src/receipts.ts:526-530), which then prints "The provider failed after work had been done; the sealed checks ran on the tree as it stood.", into the passedAtEnd info line (src/engine.ts:4469, "when the provider stopped the model"), and into the bench capture (bench/local/run.py:144 records endedBy).
Failure scenario: a model writes a file, then sends six broken act calls. The turn ends, the bar runs, and the receipt states the provider failed. Nothing failed except the model. Receipts are the thing Maat promises are true, and the bench can't tell a model that can't form tool calls from a flaky endpoint, so a model's own fault gets filed as infrastructure. Confidence: high (traced through the code; not a guess about intent).
Fix: add a third reason, e.g. "malformed", to the endedBy union (engine.ts:2357, engine.ts:4080, types.ts:645, receipts.ts:63, receipts.ts:376) with its own receipt sentence and info wording, and pass it to judgeOnDisk. A test asserting the receipt line for this path would lock it in.
Non-blocking
2. Hallucinated tool names don't count toward the streak (medium confidence). argumentProblems returns [] when the tool has no schema (src/toolargs.ts:24), so a call to a tool that doesn't exist is not malformed and resets the streak to 0 (engine.ts:5074). A model spamming unknown tool: foo with big payloads is the same cost leak this PR closes. Consider counting !TOOLS.find(...) (and unknown MCP names) as refused, and excerpting those on the wire too.
3. The new constants detach an existing doc comment (high confidence, cosmetic). engine.ts:304-306 sits between the auto-shed JSDoc and DEFAULT_AUTO_SHED_TOKENS, so that comment now documents MALFORMED_WARN in editors and generated docs. Move the constants above the shed comment.
4. The malformed check landed under the wall-clock comment (high confidence, cosmetic). At engine.ts:5954-5959 the "The wall clock, checked where the other ceilings are…" comment now introduces the malformed block, not the deadline block below it. Give the new block its own comment and move it under the deadline one, or swap them.
5. Test gaps (medium confidence). Nothing asserts the firm warning text appears at MALFORMED_WARN, and the reset test only checks the stop message is absent. Worth adding: warning present at 3; the stop with writes on disk produces a judged outcome (and, after fix 1, the right endedBy); a mixed act where a good sub-action resets the streak.
6. Excerpt marks don't survive a transcript restore (low confidence; only matters if sessions are rebuilt from the record). malformedCalls lives in memory only, so a restored transcript would resend refused calls in full. If /regrow or resume rebuilds Transcript, persist the mark on the message (molt metadata) instead of a side Set.
CI (typecheck-and-test) was still pending when this was written.
…t; marks live on the message Review of #47: endedBy gains "malformed" with its own receipt sentence instead of being filed as a provider failure; calls to a tool that does not exist count toward the streak; the firm warning reaches the model on the act path too; the refused marks are message metadata (survive a restored transcript); constants and the stop block sit under their own comments; tests for each.
|
Addressed at 7a49ff1:
|
2 participants
Measured bug: a DeepSeek run made 17 malformed
actcalls; each was resent in full every step (the call, the complaint quoting it, and the shed digest). Prompts reached 630k tokens; one task cost $0.82 for a 181-byte file (lane went from ~$0.01 to ~$0.10 per run).Fix:
{"_refused": "<first 300 chars>…[N more not resent]"}. Record/captures keep the full text._unparsedcopies are capped the same way.actthat reaches a tool call as itself (unreadable actions) counts as malformed.Verified without spending: 10 scripted 50 KB malformed calls → request sizes 7–12k chars (was up to 509k), 7 provider calls. test/malformed-cap.test.ts. npm run check green (2048), e2e + self-check pass.