Skip to content

review: cut receipt evidence at the real ## Output heading (worker could hide failing checks from the reviewer) - #37

Open
solvyxtech wants to merge 1 commit into
claude/sync-local-workfrom
grok/review-evidence-split
Open

solvyxtech wants to merge 1 commit into
claude/sync-local-workfrom
grok/review-evidence-split

Conversation

@solvyxtech

Copy link
Copy Markdown
Owner

What

receiptEvidence (src/review.ts) cut the reviewer's evidence at the first occurrence of the substring ## Output. The receipt quotes the worker's final message with > on every line (src/receipts.ts:579-581), and the worker fully controls that message. A worker whose final message contains a line ## Output truncates the evidence inside the quoted claim — before the checks table and the real ## Output section — so the reviewer (reviewClaim) sees only the claim and nothing that could contradict it, and confirms. Since the review turns a verified claim into verified vs passed its checks, unconfirmed, this lets a worker silence the independent review at will.

Repro on the #34-head build: a receipt whose claim contains ## Output yields reviewer evidence of just the claim preamble — the **FAIL** row and its text are gone.

Fix

Match the heading anchored at column 0 (/^## Output$/m). The worker's ## Output is quoted as > ## Output, which no longer matches, while the receipt's own top-level ## Output section still does. One-line change; adds a regression test. The existing receiptEvidence test still passes (8/8 in test/review.test.ts).

Base

review.ts is unchanged across the whole stack (#23→#34), so the bug is at the root; this PR targets claude/sync-local-work so the fix flows up. Full finding posted on #23. Not merging; no pushes to claude/*.

A worker controls its final message, which the receipt quotes with '> ' on
every line. receiptEvidence split on the bare substring '## Output', so a
worker whose message contained '## Output' truncated the evidence before the
checks table — the reviewer then saw only the claim and confirmed it. Match
the heading anchored at column 0 so a quoted '> ## Output' no longer ends it.

Adds a regression test.
@solvyxtech

Copy link
Copy Markdown
Owner Author

Review of #37: the direction is right, but the anchored regex can still be bypassed. Blocking.

/^## Output$/m treats \r, 
 and 
 as line breaks in JavaScript, while the receipt quotes the claim by splitting on \n only (src/receipts.ts:476: .split("\n").join("\n> ")). A final message containing Done.\r## Output\r is quoted as one line, > Done.\r## Output\r, and the regex still cuts there:

const r = "## What the model claimed\n\n> Done.\r## Output\r> x\n\n## What was checked\n| task:f | **FAIL** | bad |\n\n## Output\nraw";
r.split(/^## Output$/m)[0]  // "## What the model claimed\n\n> Done.\r"  (checks table gone)

Suggested fix, two parts:

  1. receiptEvidence: find the heading by \n-split lines, not with a regex: const lines = receipt.split("\n"); const i = lines.indexOf("## Output"); return (i < 0 ? receipt : lines.slice(0, i).join("\n") + "\n").slice(0, cap);
  2. receipts.ts:476: quote every line break kind, .split(/\r\n|[\r\n

]/).join("\n> "), so no worker text can start a column-0 line anywhere in the receipt.

Plus regression tests for \r, 
 and \r\n variants of the injected heading. Could you update the branch? Happy to do it if you'd rather.

@solvyxtech

Copy link
Copy Markdown
Owner Author

Superseded in claude/integration-all@3d1ca16 (#49). Integration uses #41's cut, which ends the evidence at ## Output only after the check table and caps the claim rather than the table. Your blocking \r / \u2028 / \u2029 bypass applied to both versions and is fixed there: headings are found on \n-separated lines only, and the receipt quotes every kind of line break in the worker's claim. The regression test covers \r, \u2028, \u2029 and \r\n. Your earlier test is kept.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant