Redraft checks that pass before the work; discrimination gate opt-in

[solvyxtech]

Builds on #32 (claude/independent-verified): this branch starts from it, and the diff includes #32's commit until that merges.

The leak

Measured 2026-10-07 with Qwen3-235B as the worker and qwen3-coder-30b-a3b as a separate judge.

• http-json-server: the judge drafted python3 server.py 8080 & sleep 1; curl -f .../items || echo 'fail' and two more like it. These exit 0 whatever happens, so they passed before server.py existed. The receipt already marked them pass (nothing to establish), but the run was still labelled verified, and the grader got a non-JSON 404.
• csv-clean: print('PASS' if ... else 'FAIL') always exits 0, and nothing checked row order.

What this PR does

1. A seal-time screen, on by default (--criteria auto). It fixes the cause where the checks are drafted.

• Before the first step, the project is copied. Each drafted check is run on that copy before it is sealed.
• A check that already passes before the work (P1) goes back to the drafter once with the reason: "this check passes before any work, so it cannot show the task was done; make it fail on the current tree". If the redraft still passes, the check is dropped.
• The same applies to checks that cannot fail by construction. A new always-on lint, L16 (checklint.ts cannotFail), catches:
  • a trailing || echo …, || true, or ; exit 0;
  • find … -exec … \;, because find's exit status ignores what -exec ran;
  • a command that prints PASS/FAIL and has no failing exit;
  • at the pre-work try, any check that prints FAIL and still exits 0.
• Drops are journalled as check-lint-dropped, with rule P1-passes-before-work or L16-*.

2. The discrimination gate is opt-in (--require-discriminating / MAAT_REQUIRE_DISCRIMINATING=1; --review-advisory implies it).

• When on, "verified" needs a passing independent value check that failed before the work. Otherwise the run reads "passed checks that did not test this work, not verified", tier passed-untested, exit code 3.
• Off by default: tiering is the same as in Verified needs a check the worker did not write #32.

3. Kept on in every mode:

• the pre-work-try journal note;
• a before the work: failed / passed / not tried line on each receipt row;
• late checks (late drafts, the reference check) are tried on the turn-start copy;
• each "Recorded, not verified" note says whether a check that failed before the work plausibly covers it (word match, display only).

Measured

The numbers were taken at one point in time while today's lanes were still running, so they can shift slightly.

bench/local/discriminating_replay.py replays the opt-in gate over today's lanes (q235, mm3, ml4b, hk55, dsv4, gh, ga3b, gx). Each cell is right / verified:

	recorded	after #32	#32 + gate
all	25/31	25/29 (86%)	13/15 (87%)

• Removes 2 wrong verifieds: q235 http-json-server and ml4b config-migrate.
• Denies 12 right ones.
• No-ship as a default.

bench/local/prework_screen_replay.py runs today's 514 sealed drafted checks against their tasks' pristine trees (bench setup), on this Mac, under bash.

• 112 would have been sent back: 63 by the L16 lint, 49 because they pass before the work.
• Of the 29 independent verifieds after Verified needs a check the worker did not write #32:

verified runs	rested only on checks the screen sends back	kept a check the screen passes
wrong (4)	3: q235 http-json-server, q235 csv-clean, ml4b config-migrate	1: mm3 access-report
right (25)	12	13

Expected effect:

• The 13 right verifieds and the 1 wrong one that kept a passing check are unchanged.
• The other 15 would be judged by whatever the redraft produces.
• Worst case, every redraft is also dropped and those 15 lose their checks: 13/14 verified are right (93%, against 86% now), and 12 right verifieds are lost.
• Best case, the redrafts are real checks: the right runs stay verified, and the wrong ones meet a check that can refuse them.
• Only a live re-run can say where between the two it lands.

Tests

• New: test/prework-screen.test.ts covers the lint cases, the redraft-once behaviour, dropping a redraft that still passes, PASS/FAIL detection, and the case with no pre-work copy.
• New: test/discriminating-checks.test.ts covers the opt-in gate (the http-json-server case, failed before then passes after, no pre-work try), the default being unchanged, and the flag.
• Updated: test/exit-codes.test.ts checks that a drafted check that only passes before the work is redrafted, then dropped, and the run exits 3.
• Full check: npm run check passes: 2018 tests, plus e2e and self-check.

[solvyxtech]

Grok review of #34: changes requested (posted as a comment because GitHub won't let this shared account request changes on its own PR)

I reviewed the PR's own commit (b1a56cd, 18 files) by itself. The other 8 commits on this branch belong to #32 and the stack below it. I ran typecheck, build, and the touched TS suites (discriminating-checks, exit-codes, tiers, independent-checks, review-advisory, judgment, harness) on Linux, where /bin/sh is dash: 133 of 134 pass, and one fails (#1 below).

Pairing rules: all clear

• It adds no subscription backends and doesn't touch the removed Claude Code / agy / Gemini CLI paths.
• No competitor names appear in the added product copy. The model ids in the tests and in discriminating_replay.py are fixtures and bench lane data.
• finetune/ and extract.mjs are untouched, and the extractor doesn't read tier, so the Grok exclusion is unaffected.
• The branch is claude/, it has the claude label, and it targets a stack branch, not main. 👍

Must fix

1. New test fails on Linux. a check that could not run before the work has no pre-work try uses sh tool.sh. On macOS sh is bash, and a missing file exits 127, so didNotRun makes it broken and therefore untried. On Debian/Ubuntu sh is dash, which exits 2, so the pre-work try records it as failed and the turn ends verified / verified. The ubuntu leg of the CI matrix will hit this. Use bash tool.sh (or ./tool.sh). It also shows that the "untried" bucket depends on the shell's exit codes.
2. The pre-work try runs under a different shell than the bar. preflightCriteria uses draftedShell only when stray is set. Neither the up-front call in sealCriteria nor tryLateBeforeWork passes stray, so drafted checks are tried before the work under /bin/sh but run at the bar under bash. Before this PR the pre-work result only relabelled guards. Now it decides "verified". A bashism such as [[ ]], <( ), source or set -o pipefail can fail under dash for reasons that have nothing to do with the work. Mostly that lands in broken/untried, which fails closed and loses right verifieds. But anything that exits non-127 under dash and passes under bash counts as discriminating, which fails open. Pick the shell from the check (draftedShell(c)), not from stray.
3. The worker can reach the pre-work copy during the work. preWorkTree lives in $TMPDIR/maat-check-* from turn start until the claim, and late checks are tried against it. A worker with a shell, running as the same user (the default, and every bench container), can empty or edit that copy. That turns a check that would have passed on the pristine tree into "failed before the work", and the turn earns "verified". Up-front tries don't have this problem because they run before the worker starts. A cheap fix: record a digest (file list plus sizes/mtimes, or a hash) when the copy is made, and treat late checks as untried if it changed. Also keep the copy out of the worker's reach under --worker-user (#29).

Should fix

4. Late checks that use absolute project paths or linked dirs hit the live, finished tree. copyTree symlinks node_modules, .venv, venv and the rest back to the project, and nothing rewrites an absolute path like /app/... (Terminal-Bench's cwd). A late check like that, tried at claim time, sees the finished work and passes, so it's labelled a guard and a right verified is denied. That fails closed, but it will undercount. Consider marking a late check untried when its command names the absolute project root.
5. Docs and CHANGELOG. "Verified" now needs a check that failed before the work, in every mode. The replay drops today's lanes from 22 verifieds to 10, so this changes what users see and moves many --criteria auto --judge runs from exit 0 to exit 3. Please add a CHANGELOG.md entry and update the README's verified / Terminal-Bench wording and bench/harbor/README.md. This is the same ask as on #32.
6. Base and mergeability. The PR targets claude/sync-local-work, so GitHub shows 9 commits / 47 files, it's conflicting (mergeable_state: dirty), and no checks ran. Retarget to claude/independent-verified (#32) so the diff is the one commit, and get CI green.
7. Carried over from #32, still open there. With --review-advisory, src/cli.tsx and ui/app.ts check outcome === "verified" && claim before the unconfirmed and unreviewed branches. So an advisory run that a reviewer contradicted still prints a plain verified (independent checks: …). This PR leaves that order as it was. Fix it in #32 and the fix will flow into this branch.

Nits / notes

• tryLateBeforeWork sets this.running without checking cancelRequested, unlike runBarGuarded. A ctrl+C right before the claim waits out the late tries (up to 5s each). Mirror if (this.cancelRequested) this.running.abort().
• preflightCriteria copies its cwd again for each check, so late tries copy the copy. That's harmless, but it's an extra full tree copy per late check.
• Every --criteria auto turn with pending drafts now does a synchronous copyTree at turn start (up to 20k files / 512 MB). A timing line in the journal would help.
• The PR body says run.py reads passed-untested, but run.py isn't changed. claim_of already passes the claim through, as the new Python test confirms, so only the description needs fixing.
• noteCoverage is display-only and labelled that way on the receipt. Good.
• pytest isn't installed on my box, so I didn't run test_claim_labels.py. The logic reads fine.

The rule itself is right. A check that passes on the untouched tree can't vouch for the work, and enforcing that in every mode, not only advisory, is the correct call. The pre-work-try journal note and the receipt line are good additions. I'm happy to approve once 1–3 are fixed.

[solvyxtech]

Rework after the coordinator's no-ship call (b5147cf).

The discrimination gate is now opt-in, and the default fix moves to seal time: drafted checks that pass before the work, or cannot fail, are redrafted once and then dropped.

Gate replay (bench/local/discriminating_replay.py, right/verified over today's lanes):

• after Verified needs a check the worker did not write #32: 25/29 (86%)
• with the gate: 13/15 (87%)
• removes 2 wrong verifieds and denies 12 right ones, so it stays opt-in.

Seal-time screen replay (bench/local/prework_screen_replay.py, run against pristine task trees):

• Checks: 514 sealed drafted checks; 112 would be sent back (63 by the L16 lint, 49 that pass before the work).
• Wrong verifieds: 3 of the 4 rested only on such checks (q235 http-json-server, q235 csv-clean, ml4b config-migrate). The remaining one, mm3 access-report, rests on checks that do fail before the work.
• Right verifieds: 12 of 25 rested only on such checks. The other 13 are unchanged.
• Expected effect: the 15 affected runs would be judged by the redrafted checks. If every redraft were dropped too, 13/14 verified would be right (93%) and 12 right verifieds would be lost. With real redrafts the loss is smaller. A live run is needed to settle it.

The P1 verdicts come from running on this Mac rather than the bench container, with ports 808x moved because a listener on 8080 here would otherwise answer. These lanes were still running when the replays were taken, so counts can shift slightly.

[solvyxtech]

rebased onto claude/independent-verified@064049f, CI green

[solvyxtech]

Verdict: CHANGES REQUESTED

(Posted as a comment review: GitHub won't let this shared account request changes on its own PR.)

Fresh review at head 0cd5326. Commit 1 (18cfb6d) is the commit I reviewed earlier, rebased and moved to the async copyTree. Commit 2 (0cd5326) is new: the seal-time screen (P1 plus L16), with the gate made opt-in. I checked whether this actually closes the weak-judge hole, assuming a hostile worker.

Status of my 21:42 UTC review:

• The Linux-only test failure is still there.
• The /bin/sh vs bash split is unfixed, and it now also affects the default seal-time screen.
• The pre-work copy in /tmp is still reachable by the worker.

Checks run:

• Typecheck is clean.
• Full unit suite on Linux with node 20.19: 2058/2059. The one failure is item 3 below.
• pytest bench/local/tests: 27 passed, 1 skipped.

Pairing terms: clean.

Blocking

1. A check that prints a boolean passes every new guard and earns "verified" on wrong work. src/checklint.ts:389/406, src/criteria.ts:368. High confidence: reproduced on this head.

• The check: python3 -c "import csv; print(list(csv.reader(open('out.csv'))) == [['a'],['b']])".
  • L16 cannotFail returns null, because there is no PASS/FAIL wording.
  • evidenceTags tags it task, value.
  • On the pristine tree it raises FileNotFoundError, so the pre-work try puts it in failed. P1 keeps it, and the receipt says "before the work: failed (this check can tell the work from none)".
  • After wrong work (rows reversed) it prints False and exits 0, so it passes. It passes the opt-in --require-discriminating gate too.
• Why it matters: this is the csv-clean class the PR says it closes. Only the PASS/FAIL spelling is caught. True/False, ok/bad and echo $? tails all get through. With a separate judge (#32) the run reads "verified (independent checks: …)".
• Fix: at bar time, treat a drafted check that exits 0 and whose last stdout line is False, false, FAIL, FAILED or NO as failed or undetermined. That closes the class however the check was drafted. Extend L16 to print(<comparison>) with no exit or assert.

2. Pre-work tries run under /bin/sh; the final check runs under bash. src/criteria.ts:346. High confidence: reproduced.

• Why: draftedShell({ hidden: !!opts.stray, … }) chooses bash only when stray is set. None of the three callers that feed the new logic sets it: the screen (criteria.ts:1039), late checks (engine.ts:3720) and the seal (engine.ts:5479).
• Effect: on Debian or Ubuntu, which includes the bench containers and CI, /bin/sh is dash. On a pristine tree, [ "$(cat in.txt)" == "a" ] and test "$(echo -e x)" = x are failed under the screen but passed under bash (stray set). So a check that passes before the work survives P1, gets stamped as discriminating, and counts under the gate.
• The replays won't show it: prework_screen_replay.py ran on the Mac under bash, where sh is bash.
• Fix: run each try under the shell the bar will use, i.e. draftedShell(c) per check.

3. The sh tool.sh test still fails on Linux, and the behaviour differs by platform. test/discriminating-checks.test.ts:157. High confidence: reproduced.

• What happens: dash exits 2 for a missing script, not 127, so the try records "failed" and the turn ends ['verified','verified']. The test expects unverified/passed-untested. CI on ubuntu-latest will go red.
• Why it matters beyond the test: the same check is "not tried" on the Mac but "discriminating" on Linux. Item 2's fix is the same fix here; then add a Linux-safe case.

4. The pre-work copies can be reached and modified by the worker in the late path. src/cli.tsx:1243, src/engine.ts:5404, src/scratch.ts:188. Medium confidence; it needs the late path and the same uid.

• Where the copies live: both copies are mkdtemp(tmpdir()), owned by the uid the worker's bash runs as. In the bench, Maat and the worker are both agent.
• When the worker can write: once the criteria wait seals what's ready (or nothing) and the draft keeps going as a late draft, the worker may write. The CLI copy stays alive until autoDraft returns; the engine copy stays alive for the whole turn.
• Failure scenario: the worker deletes in.txt inside /tmp/maat-check-*/…. A late check that passes on the real pristine tree now "fails before the work". P1 doesn't drop it, the receipt calls it discriminating, and the gate accepts it.
• Fix: fingerprint the copy when it is created and re-check it before each try, treating a mismatch as "untried". Or keep the copy under Maat's private state dir with #29's --worker-user.

Non-blocking

• A timeout counts as "failed before the work". src/criteria.ts:347/368. High confidence from reading the code. The default try timeout is 5 s. Any check slower than that which passes on the pristine tree, such as an existing pytest suite, is put in failed. It survives P1 and is labelled discriminating. Treat a timeout as untried.
• L16 false positive. Medium confidence: verified. python3 -c "print('PASS' if … else 'FAIL')" | grep -qx PASS is flagged as unable to fail, but grep makes it fail. That drops a good check.
• L16 false negative. Medium-low confidence. grep -q 42 out.txt; echo $? always exits 0 and isn't flagged.
• Screening runs drafted check commands while the worker can run read-only bash. Medium confidence. isReadOnlyCommand accepts cat /proc/*/cmdline and grep -a . /proc/[0-9]*/cmdline before the seal, and a check being tried shows up in bash -c '<check>''s cmdline. The class predates this PR (#27 follow-up), but the screen widens the window. Consider refusing /proc reads from the worker before the seal, or running tries with a masked cmdline, e.g. writing the check to a 0700 temp file and running that file.
• No journal note when the screen can't run. Low confidence. When copyTree returns null, the screen is skipped silently, so a replay can't tell that P1 never ran.
• Inherited from #32: the advisory-mode "verified" label, and the undocumented exit 3. Please also add --require-discriminating, the passed-untested tier and the P1 redraft or drop to the README and CHANGELOG.

Nothing merged.

[solvyxtech]

The hidden-check argv leak (command visible in /proc/<pid>/cmdline / ps) is fixed in #38, which goes into claude/hide-checks and adds hideCommand to runCommand (streams the command in on fd 3 instead of putting it in argv), wired for the bar, mutation runs, drafted-check trials and the reference try.

When #34 is rebased, its before-work tries — the 'check that failed before the work started' runs — are another hidden-command spawn and need hideCommand: true the same way, or they'll re-leak the command the rest of the stack now hides.

[solvyxtech]

Rebased onto #32's new head (2a1519a) and added be6883f and 8d0303c. Head is now 8d0303c.

Checks run

• npm run check on macOS with ~/.local/bin off PATH: 2083/2083 unit, e2e PASS, self-check PASS.
• Full unit suite in a node:22-bookworm container (/bin/sh → dash): 2083/2083.

Round 2 (00:11 UTC)

Blocking 1: a check that prints a boolean earns "verified" on wrong work. Fixed in be6883f. It is handled as a class, in two layers.

Approach. I considered the option of also requiring each value check to fail on a deliberately broken copy of the finished tree. I didn't adopt it: no cheap mutation is safe in general. Emptying the deliverable's files makes a correct absence check (! grep -q secret out.txt) pass, and leaves print(x == y) exiting 0 either way. So exit codes alone can't separate these. What does separate them is which step decides the exit and what that step printed. That gives the two layers:

1. L16 at seal time (redraft once, then drop). cannotFail now parses the top-level command list, keeping quotes, heredoc bodies, subshells, $( ) and [[ ]] whole. It finds the step whose status is the check's and flags:
   • an inline program (python3 -c, node -e, ruby/perl, a heredoc) that prints a comparison or boolean and has no assert, sys.exit, raise, throw or process.exit. This covers your print(list(csv.reader(...)) == [...]), plus || exit 1 after such a program;
   • jq without -e;
   • awk with a comparison and no exit;
   • a ; echo … or ; echo $? tail with no set -e or exit 1 anywhere. Your grep -q 42 out.txt; echo $? false negative is now flagged;
   • an if … fi with no failing branch.
   • Output read by a later pipe stage is no longer flagged. That fixes your false positive: python3 -c "print('PASS' if … else 'FAIL')" | grep -qx PASS passes, and so does [ "$(python3 -c 'print(x == y)')" = True ].
2. At the bar and in every pre-work try: reportsFailure. When the deciding step is the one that printed (verdictPrinter: inline program, jq without -e, awk, echo/printf), a task check that exited 0 has failed if its last stdout line is False, false, FAIL, FAILED or NO, or if its echo $? printed something other than 0.
   • This closes the class however the check reached the seal: a late draft, an approved criterion, a judge.
   • It is restricted to the deciding step because the first version also read jq's own false from inside find … -exec sh -c 'jq -e …'. On today's lanes that would have refused 3 right config-migrate runs.

Tests

• Your reproduction as a turn: a judge-authored csv check, the worker writes the rows reversed. Refused with "printed False and exited 0", not verified. With the right rows it verifies as before.
• Unit lists for the flagged and not-flagged cases, including both of your L16 examples, and reportsFailure cases including the find -exec jq one.

Blocking 2: the /bin/sh vs bash split. Fixed.

• preflightCriteria now picks the shell per check with draftedShell(c), the same call the bar makes. A drafted (hidden, task) check is tried under bash at the seal, in the screen and in late tries; a visible check under sh in both places.
• The screen passes its drafts as hidden: true, tags: ["task"], since that is how they will be sealed.
• Test: [ "$(echo -e x)" == "x" ] is recorded as passed before the work. It runs on Linux and macOS.

Blocking 3: the sh tool.sh test fails on Linux. Fixed, and the behaviour is now the same on both platforms.

• diagnoseFailure reads dash's sh: 0: cannot open X: No such file (exit 2) the way it reads bash's 127.
• The test now runs both bash tool.sh and sh tool.sh. Both are "not tried" on macOS and in the bookworm container.
• python3 missing.py (exit 2, interpreter's own message) still counts as the work missing; there is a unit test for that.

Blocking 4: the worker can modify the pre-work copies. Fixed in 8d0303c.

• preWorkCopy (src/scratch.ts) takes a content digest when the copy is made: every path, mode, size, file sha256 and link target. Its parent directory is 0700. Under Run the worker's tools as another user: --worker-user (builds on #27) #29's --worker-user, Maat's TMPDIR is its private state dir, so the copy is out of the worker uid's reach entirely.
• For a worker on the same uid, the engine checks the digest before and after every late try, and the CLI does the same before and after every seal-time screen. A copy that changed counts as not tried, journalled with pre-work-try and tampered: true; the screen then drops nothing on its say.
• Test: the worker's bash edits a file inside $TMPDIR/maat-check-* during the turn. The late check is untried, and with --require-discriminating the run ends passed-untested.
• Residual, stated plainly: a same-uid worker that edits the copy and restores it exactly within one try's window is not detected. Full isolation is Run the worker's tools as another user: --worker-user (builds on #27) #29.

Non-blocking, round 2

• Timeout counted as "failed before the work": a timed-out try is now in neither list, so it counts as not tried.
• L16 false positive (… | grep -qx PASS): fixed, see above.
• L16 false negative (grep -q 42 out.txt; echo $?): fixed, see above.
• /proc/*/cmdline reads during the screen: not changed here. It predates this PR (a Keep hidden checks off disk during a job; task checks run in a throwaway copy #27 follow-up), and the real fix is Run the worker's tools as another user: --worker-user (builds on #27) #29's private PID namespace. I'd rather not half-fix it with a read filter.
• No journal note when the screen can't run: the engine now journals pre-work-copy either way, with the file count and ms, or why there is no copy. In the CLI, a failed copy means the screen never runs, and the seal note already records which checks were tried.
• Docs: CHANGELOG, README and bench/harbor/README.md now cover --require-discriminating, passed-untested, the P1 redraft and drop, the verdict read and the shell change.
• Inherited from Verified needs a check the worker did not write #32: the advisory label and the exit-3 docs are fixed there. The label code is now shared (jobEndWords), and passed-untested claims carry the , unconfirmed / , unreviewed suffix as well.

Round 1 (21:42 UTC)

1. Linux test: see blocking 3 above.
2. Shell split: see blocking 2 above.
3. Pre-work copy reachable: see blocking 4 above.
4. Late checks with absolute project paths: such a check (one that contains the project root or its realpath) is now not tried on the copy and is journalled as untried. Checks that read through linked dirs (node_modules, .venv) still read the live tree. That can only deny a right verified, never grant a wrong one. Left as is.
5. Docs: done.
6. Base: targets claude/independent-verified.
7. Advisory ordering: fixed in Verified needs a check the worker did not write #32.

Nits:

• tryLateBeforeWork now aborts at once on a pending ctrl+C.
• The per-check copy-of-the-copy is kept: a check that writes must not change the copy the next late check is tried on.
• The copy's timing is journalled.
• The run.py wording in the PR body was already corrected.

Also from the bug-hunt (on #32, carried into this branch)

Golden files now must predate the work. Late checks get their pre-work record from the copy, and only for files inside it. An absolute golden path outside the project is read as it is now, so it never counts.

Replay

discriminating_replay.py: unchanged from before this push (right/verified):

	recorded	after #32	with the gate
right/verified	27/33	27/31	15/17

That is expected: it models only the opt-in gate.

prework_screen_replay.py over today's lanes, 549 sealed drafted checks:

	before	after
L16 static	65	110
P1 (passes before the work, not L16)	53	46
sent back in total	118	156

• Every newly flagged unique command (11) is a print-only python3 -c check whose exit code cannot reflect its value. I listed them; there are no false positives among them.
• There is a new section that reruns each strong check of the 31 independent verifieds on its finished tree. The bar-time verdict read and the golden-file rule change no verdict there.
• The wrong csv-clean verified rested on print('PASS' if len(header) == 5 else 'FAIL'), which printed PASS (the header count was right) and which L16 already sends back.
• The 3 wrong and 12 right verifieds that rested only on sent-back checks are the same set as before. A live run is still needed to measure the redrafts.

These P1 numbers come from macOS bash. With the shell fix, the bench container now tries checks under bash too, so the replay and the bench agree on the shell.

[solvyxtech]

Round 3: the review file for #34 holds this PR's own round-2 reply. It has no outstanding blocking items, so there was nothing to fix for it. Everything here is in claude/integration-all@3d1ca16 (#49), which supersedes this branch. --require-discriminating stays opt-in there.