Verified needs a check the worker did not write by solvyxtech · Pull Request #32 · solvyxtech/molt

solvyxtech · 2026-10-07T21:18:11Z

Builds on #30 (claude/overflow-classify → #27 → #24–#26). Merge those first; this diff is one commit on top.

Why

A model was allowed to find the work and also judge it. On the 2026-10-06/07 container bench:

self-judged arms (the worker drafts its own hidden checks): 125 of 174 "verified" claims right (72%)
separate-judge arms: 48 of 55 right (87%), with 0 wrong on the latest builds
the wrong "verified" results on the latest builds were all self-judged (MiniMax M3 on regex, where its checks shared its own misreading of the IP trap, and Qwen3-235B)

Every one of those claims was labelled "verified (self-checked)", including the ones a separate judge checked.

What changes

Authorship is recorded at seal time. Each check gets author: { kind: worker | judge | person | reference, model? }. CLI --criteria auto drafts are judge when --judge/MAAT_JUDGE_MODEL is set, otherwise worker. Criteria a person approved in the window, gave with --criterion, put in done.yml or a mission contract count as person. A reference check is reference with the model that wrote it. If no author was recorded, a hidden check counts as the worker's.
Tier rule (tierOf). A run is verified only when a passing check that ran the work and asserted a value was written by a person, or by a model other than the worker. Model ids are compared after provider normalization, so openrouter/m:free counts as the same model as m, and so does a dated alias. Both the configured worker id and the id the backend reported are compared. If all such checks are the worker's, the run gets the new tier passed-own-checks, with outcome unverified and the same exit code (3). A reviewer contradiction and an unreviewed claim still take priority, as before.
Labels. verified (independent checks: <judge model>), verified (your checks), passed own checks (<worker model>), not verified. These appear on the terminal and in the window, in job_end's new claim field (plus checkAuthors), in the receipt index row and in the receipt.
Receipts list who wrote each check: a written by: line per check in the Output section, a "written by" note on each sealed task criterion, and a Claim: line.
Bench. run.py's claim field now takes job_end's claim, falling back to the old outcome + " (self-checked)" for older builds. stats.py adds claim_basis / claim_judge, which read both old and new text, and passed-own-checks joins TIERS. summarize.py handles the new tier. startswith("verified") parsers keep working: the new verified labels still start with "verified", and the passed-own-checks label does not.

Interactive and desktop sessions are unchanged, because their checks are person-approved.

Known gap, left as is: the reference check is still written by the worker model even when a judge is set, so a reference never counts as independent unless its writer is a different model.

Tests

test/independent-checks.test.ts: model-id normalization; worker-only checks → not verified; judge-written → verified; a judge with the same model id (including under a provider prefix) → not independent; person-approved → verified; the label text in each case; receipts and index rows record authorship and the claim.
test/exit-codes.test.ts: --criteria auto without a judge, or with a same-model judge, exits 3; with --judge j it exits 0.
test/reference.test.ts: a reference written by the worker's own model is passed-own-checks.
bench/local/tests/test_claim_labels.py: run.py's claim_of on new and old job_end events, plus claim_basis / claim_judge / is_verified on old and new text.
Existing fixtures that sealed hidden checks to test other features now mark them as written by a judge model.

npm run check passes with ~/.local/bin off PATH: 1999 node tests, e2e PASS, self-check PASS. pytest bench/local/tests: 27 passed.

solvyxtech

Grok review of #32: changes requested (posted as a comment because GitHub won't let this shared account request changes on its own PR)

I reviewed the PR's own commit (fc5aa1d, 27 files) on its own. The other 7 commits on this branch belong to #24–#27 and #30.

Pairing rules: all clear

It adds no subscription backends and doesn't touch the removed Claude Code / agy / Gemini CLI paths.
No competitor names appear in product copy. The vendor ids in normalizeModelId are code, and the ones in test/independent-checks.test.ts are test fixtures, so both are fine.
Nothing touches finetune/ or the export path. claim / checkAuthors are new job_end fields, so please make sure they don't change the fine-tune Grok exclusion filter later.
The branch is claude/, it has the claude label, and it targets a stack branch, not main. 👍

Must fix

Advisory review mode loses "unconfirmed" and "unreviewed" on the terminal and in the window. With reviewAdvisory, tierOf still returns verified (plus a reviewNote) when the reviewer contradicted the task or didn't run. Then claimLabel sets claim, and the new ev.outcome === "verified" && ev.claim branch in src/cli.tsx (around L1426) is checked before the review && !review.confirmed and unreviewed branches. So a run that used to print passed its checks, unconfirmed / passed its checks, unreviewed now prints verified (independent checks: j) with no qualifier. ui/app.ts (around L1144) has the same ordering bug. Either move the claim branch below those two checks, or append , unconfirmed / , unreviewed to the claim when review.confirmed === false or ev.unreviewed is set. Please add a test: an advisory run with a judge-authored check and a 2/3 contradicting review should still say unconfirmed.

Should fix

Behavior change needs docs. A headless --criteria auto run without --judge now exits 3 where it used to exit 0. That will break CI scripts and any bench arm that has no judge. Please add a CHANGELOG.md entry and update the README "Terminal-Bench" paragraph (around L402–406: "the model drafts acceptance checks… it is held to them") to say that "verified" needs --judge <another model> or person-approved checks. bench/harbor/README.md should say the same, since molt_agent.py runs --criteria auto.
Base / mergeability. The PR targets claude/sync-local-work, so GitHub shows 8 commits / 45 files, it's CONFLICTING, and no checks ran on the branch. Please retarget to claude/overflow-classify (#30) so the diff is the one commit, or rebase once the stack lands, and get CI green before merge.

Nits / notes

sealDraft reads process.env.MAAT_JUDGE_MODEL directly instead of env("JUDGE_MODEL"), so a legacy MOLT_JUDGE_MODEL won't mark checks as judge-written. judge.ts has the same behavior, so this is consistent, but it's worth knowing.
Checks approved in the interactive interview (app.tsx, taskChecksFrom with no hidden) are labelled verified (your checks) even though a model proposed them. That matches the PR's design. "approved by you" might read more accurately, but it's your call.
normalizeModelId errs toward "same model" (it strips everything before the last /, plus date and alias suffixes). That's the right direction because it fails closed. The tests cover deepseek-v3 vs v2 and qwen3-235b vs 32b, which is good.
The known gap where the reference is written by the worker even with a judge set is well documented. A follow-up issue to draft the reference with the judge model would close it.

The core rule is solid: unknown authorship defaults to the worker, ids are compared under every name the worker ran as, and person checks keep "verified". I'm happy to approve once #1 is fixed and #2 is documented.

The gate from the previous commit is now opt-in (--require-discriminating, MAAT_REQUIRE_DISCRIMINATING=1; --review-advisory implies it). Replayed over 2026-10-07's lanes it cost more right verifieds than it removed wrong ones, so default tiering is #32's again. The pre-work try is still journalled and shown on each receipt row, late checks are still tried on the turn-start copy, and the coverage notes stay. The cause is fixed where the checks are drafted instead. --criteria auto copies the project before the first step, and each drafted check is tried on that copy before it is sealed. One that already passes there cannot show the task was done: it goes back to the drafter once with "this check passes before any work, so it cannot show the task was done; make it fail on the current tree", and is dropped if the redraft still passes. Checks that cannot fail by construction get the same treatment through an always-on lint (L16): a trailing `|| echo ...`, `|| true`, `; exit 0`, `find ... -exec ... \;` (find ignores what -exec returns), PASS/FAIL printed with no failing exit, and, at the try, FAIL printed with exit 0. Drops are journalled as check-lint-dropped with rules P1-passes-before-work and L16-*. bench/local/prework_screen_replay.py runs today's 514 sealed drafted checks against their tasks' pristine trees: 112 would have been sent back (63 by the lint, 49 passing before the work). Of the 29 independent verifieds, 15 rested only on such checks: 3 of the 4 wrong ones (q235 http-json-server, q235 csv-clean, ml4b config-migrate) and 12 right ones, which would be judged by the redraft instead.

A run was labelled verified on checks the worker model drafted for itself. On the 2026-10-06/07 container bench, self-judged arms were right in 125 of 174 verified claims (72%) and separate-judge arms in 48 of 55 (87%); the wrong verifieds on the latest builds were all self-judged. Who wrote each task check is now recorded when it is sealed: the worker, a separate judge (--judge / MAAT_JUDGE_*), a person (done.yml, a mission contract, criteria approved in the window or given with --criterion), or the reference writer. A run is verified only when a passing check that ran the work and asserted a value came from someone other than the worker model, compared by provider-normalized id, or from a person. Otherwise the tier is passed-own-checks: outcome unverified, exit code 3. Claims read "verified (independent checks: <judge>)", "verified (your checks)" or "passed own checks (<worker>), not verified" on the terminal, the window, job_end's new claim field, the receipt index and the receipt. Receipts say who wrote each check. bench/local/run.py carries job_end's claim and checkAuthors; stats.py reads both the new and the old text.

solvyxtech · 2026-10-07T23:03:10Z

rebased onto claude/completion-reserve@a48570d, CI green

The gate from the previous commit is now opt-in (--require-discriminating, MAAT_REQUIRE_DISCRIMINATING=1; --review-advisory implies it). Replayed over 2026-10-07's lanes it cost more right verifieds than it removed wrong ones, so default tiering is #32's again. The pre-work try is still journalled and shown on each receipt row, late checks are still tried on the turn-start copy, and the coverage notes stay. The cause is fixed where the checks are drafted instead. --criteria auto copies the project before the first step, and each drafted check is tried on that copy before it is sealed. One that already passes there cannot show the task was done: it goes back to the drafter once with "this check passes before any work, so it cannot show the task was done; make it fail on the current tree", and is dropped if the redraft still passes. Checks that cannot fail by construction get the same treatment through an always-on lint (L16): a trailing `|| echo ...`, `|| true`, `; exit 0`, `find ... -exec ... \;` (find ignores what -exec returns), PASS/FAIL printed with no failing exit, and, at the try, FAIL printed with exit 0. Drops are journalled as check-lint-dropped with rules P1-passes-before-work and L16-*. bench/local/prework_screen_replay.py runs today's 514 sealed drafted checks against their tasks' pristine trees: 112 would have been sent back (63 by the lint, 49 passing before the work). Of the 29 independent verifieds, 15 rested only on such checks: 3 of the 4 wrong ones (q235 http-json-server, q235 csv-clean, ml4b config-migrate) and 12 right ones, which would be judged by the redraft instead.

solvyxtech

Verdict: CHANGES REQUESTED

(Posted as a comment review: GitHub won't let this shared account request changes on its own PR.)

Fresh review at head 064049f. Range-diff against fc5aa1d, the commit my 21:20 UTC review looked at, shows only the rebase onto #36. The content is identical, so both items from that review are still open. I re-read the whole commit with a hostile worker in mind.

Checks run: typecheck is clean. The full unit suite passes 2040/2040 on Linux with node 20.19; the runner exits 1 only because node's coverage reporter crashes. pytest bench/local/tests: 26 passed, 1 skipped.

Pairing terms: clean. The only vendor names are the prefix regex and comment in normalizeModelId, which aren't product copy. No subscription backends, and fine-tune data is untouched.

Blocking

1. A passing check from the default maat init bar counts as "your checks", so self-judged runs still read "verified". src/tiers.ts:278 and 298. High confidence: reproduced with tierOf.

The rule as coded: person = passing.some((r) => authorOf(r).kind === "person"). A visible check with no recorded author is a person's. So any passing non-hidden result earns basis: "person", with no runs or value requirement. That includes the session builtins every maat init bar ships (src/detect.ts:230): work-landed: files-changed, record-intact, claims-grounded, work-accounted, spec-intact.
Failure scenario: in any initialised project, run maat run --criteria auto headless with no --judge. The worker drafts every hidden check and changes a file, so work-landed passes. The run is tier verified, basis person, labelled "verified (your checks)", exit 0. tierOf({results:[work-landed ok (builtin), task:out ok (hidden, value, author worker m)], worker:["m"]}) returns {"tier":"verified","basis":"person"}. With only a surface check plus work-landed, the result is the same.
Why it matters: this is the self-judging hole the PR exists to close. Your own description says a person's check must be one that "ran the work and asserted a value". The receipt also prints "written by: a person (your check)" for a builtin nobody wrote for this task.
Fix: person should need a passing person-authored command check, at least excluding kind: "builtin" and session-tagged checks. Builtins shouldn't print as person-written.

2. The advisory-mode label still prints "verified" when the review contradicts the work or doesn't run. src/cli.tsx:1458 and ui/app.ts:1144. High confidence. Unchanged since my last review.

What happens: with --review-advisory, tierOf returns verified with a reviewNote. The new ev.outcome === "verified" && ev.claim branch comes before the review.confirmed === false and unreviewed branches. So the terminal and the window print "verified (independent checks: j)" instead of "passed its checks, unconfirmed" or "unreviewed".
Fix: test review.confirmed === false and ev.unreviewed before using the claim, or put them into the claim text.

3. The exit-code change is still undocumented. README and CHANGELOG. High confidence.

What happens: headless --criteria auto without --judge, or with a judge that is the same model, now exits 3 instead of 0, and there's a new passed-own-checks tier. Neither README nor CHANGELOG mentions either.
Why it matters: with the npm publish planned for tomorrow, CI jobs and bench scripts that treat exit 0 as success will flip without warning.

Non-blocking

normalizeModelId misses common aliases of the same model. src/tiers.ts:134. Medium-low confidence. A worker qwen/qwen3-235b-a22b-2507 and a judge Qwen/Qwen3-235B-A22B-Instruct-2507 normalise to different ids (…-instruct-2507), so a model ends up judging itself while counted as independent. The operator sets this, not the worker, so it's not exploitable from inside a run. Consider stripping -instruct/-chat/-it, or warning when the two ids share a stem.
Authorship is now the only gate, so a weak judge's checks still decide. Medium confidence; cross-reference to #34. A judge's check that prints a boolean and exits 0 is tagged runs+value and earns "verified (independent checks: …)" on wrong work. Example: python3 -c "import csv; print(list(csv.reader(open('out.csv'))) == [['a'],['b']])". I reproduced it on #34's head, and #34 doesn't catch it either.
An empty worker id makes a real judge count as not independent. Low confidence; I didn't verify which backends hit this. independentOf returns false when there are no worker names, so if cfg.model and the reported model are both empty, a real judge never counts.

Nothing merged.

solvyxtech · 2026-10-08T00:25:13Z

Grok bug-hunt (verdict correctness) — two findings in this PR's diff, both reproduced on the #34-head build. Not a full review (that pass is separate); these are from the hostile-worker verdict pass.

1. `assertsValue` counts a worker-written golden file as a value assertion — blocking, medium confidence (`src/tiers.ts:77-81`)

The diff/cmp branch returns true whenever an operand's name contains expect|golden|want|answer|baseline|correct, with no knowledge of whether that file existed before the work. assertsValue is a pure string function, so it cannot know — but a golden file should only confer "value" if it predates the work.

Confirmed:

assertsValue("diff out.txt expected.txt")   -> true
assertsValue("diff output.json golden.json") -> true
assertsValue("cmp got.bin baseline.bin")     -> true
assertsValue("diff out.txt in.txt")          -> false   (correctly)

Why it matters: with --judge, a judge check like diff out.txt expected.txt where the task ships no expected.txt tags value, is independent-authored, and — if the worker creates both out.txt and expected.txt — passes at the bar, earning verified. The seal-time P1 screen does not catch it: on the pre-work copy neither file exists, so diff errors (exit≠0) and the check is recorded as failed before the work, not passed before, so it is never dropped. In default mode (#34 gate opt-in) nothing else requires the golden to predate the work. The honest fix lives with the discriminating signal, not in the string heuristic: a golden-file diff should only count once the referenced file is known to have existed on the untouched tree (e.g. gate this branch on membership in the pre-work snapshot / failedBefore).

2. The reference check passes with no comparison when the deliverable calls `os._exit(0)` on import — blocking, high confidence (`src/reference.ts:171-209`)

The DRIVER loads check.py/second.py and calls a.run_deliverable(x), which (per REFERENCE_SYSTEM, src/reference.ts:116-117) "runs the deliverable … the command line, import path or output file the task names." For an import-style task, run_deliverable imports the worker's module in-process. A worker module with os._exit(0) at top level terminates the whole driver with exit 0 before any comparison; the reference check (expectExit 0) is then recorded as a pass.

Confirmed end-to-end through runBar: check.reference(x)=x*2, worker sol.f(x)=x*999 (wrong), sol.py top-level os._exit(0) →

reference check ok (counted as pass)? true  exit 0
output: "`task:reference` exited 0 in 47ms"

No input was ever compared; judgePass does not flag the empty output. Root cause: importing the worker's module runs worker code inside the grader process, so the worker can end it cleanly (and, separately, read check.py/second.py's expected values from HERE). Why it matters: the reference is the strongest work-sensitive check; a silent pass removes the one thing that would fail the bar on wrong work, which — alongside passing independent judge checks — flips the turn to verified. Suggested fix: run run_deliverable out-of-process (subprocess), or import it in a forked child and treat a child exit without a printed verdict as a reference self-error (exit 3), not a pass; at minimum require the driver's final summary line on stdout for a pass rather than trusting a bare exit 0.

Nothing merged; no pushes to claude/*.

…driver's own count The driver imported an import-style deliverable into its own process, so a module with os._exit(0) at top level ended the driver with exit 0 before any input was compared, and the reference check passed on wrong work. Each input now runs in a fresh child (runner.py) that loads check.py, drops reference() and INPUTS from the module, calls run_deliverable and sends repr() of the result back on its own pipe. A child that ends without a result (os._exit, sys.exit, a crash) is a result that matches nothing. Expected values stay in the driver's memory: never on disk, never in a child. The driver ends with "REFERENCE COMPARED n/n", and the bar requires that line for a pass: an exit 0 without it is a failure.

A passing check from the default `maat init` bar (work-landed, record-intact, claims-grounded, work-accounted, spec-intact) had no recorded author, so it counted as a person's check and turned a self-judged run into "verified (your checks)". Only a command check a person wrote or approved now carries the person basis; builtins and session-tagged checks never do, a result with no recorded kind fails closed, and receipts say these checks are Maat's, not "a person". `diff out.txt expected.txt` (any expect/golden/want/baseline/answer operand) counted as asserting a value even when the worker wrote both files. The seal now records each such operand's content before the first step, and at the tier a check whose value rests only on such a file counts only when the file predates the work and is unchanged (src/golden.ts). With --review-advisory, a review that contradicted the task or did not run no longer disappears behind the claim: claimLabel appends ", unconfirmed" or ", unreviewed", and the terminal and the window share one jobEndWords that tests both before using a claim. normalizeModelId also drops -instruct/-chat/-it, so a judge reached under another provider's spelling of the worker model is not counted independent. CHANGELOG, README (exit codes, Terminal-Bench) and bench/harbor/README.md document that headless --criteria auto without --judge now exits 3.

solvyxtech · 2026-10-08T00:40:58Z

Pushed efcbdb4 and 2a1519a on top of 064049f (base is still claude/completion-reserve). npm run check passes on macOS with ~/.local/bin off PATH: 2053/2053 unit, e2e PASS, self-check PASS. The full unit suite also passes in a node:22-bookworm container, where /bin/sh is dash: 2053/2053.

Round 2 (00:11 UTC)

Blocking 1: default maat init builtins counted as "your checks". Fixed in 2a1519a.

tierOf now gives the person basis only to a passing command check (kind: "command") that is not tagged session and was written or approved by a person (personCheck in src/tiers.ts).
Builtins (work-landed, record-intact, claims-grounded, work-accounted, spec-intact, imports-tracked, …) and session-tagged commands never carry it, even when an author map says "person".
A result with no recorded kind fails closed.
Receipts print these checks as "written by: Maat (a session check: … does not count toward verified)" instead of "a person (your check)".
Your repro, tierOf({results:[work-landed ok (builtin), task:out ok (hidden, value, worker m)], worker:["m"]}), now returns passed-own-checks / own. Builtins alone now give passed-checks.
New tests: the unit cases, plus a turn on a real proposeBar bar beside a worker-drafted check. That turn ends passed own checks (m), not verified, and the receipt names Maat as the writer.
Behaviour change, documented: a project whose bar is only builtins now ends unverified (exit 3), where it used to end verified.
One judgment call, flagged: command checks that maat init detected (npm test, tsc, …) sit in the person's own done.yml, so they still count as the person's. Tell me if you want init-drafted commands excluded too, e.g. by tagging them at init.

Blocking 2: advisory label hid "unconfirmed" / "unreviewed". Fixed.

In advisory mode tierOf now sets reviewGap, and claimLabel appends , unconfirmed or , unreviewed. So the job_end claim, the receipt and the receipt index all carry it.
The terminal (src/cli.tsx) and the window (ui/app.ts) now share one jobEndWords. It tests the contradicted and unreviewed cases before using a claim, and adds the qualifier to a claim from an older build that lacks it.
Test: an advisory run with a judge-authored check and a 2/3 contradicting review reads verified (independent checks: j), unconfirmed. There are also cases for unreviewed and for the window/terminal words.

Blocking 3: exit-code change undocumented. Done.

CHANGELOG.md now has entries for passed-own-checks, exit 3 for headless --criteria auto without --judge, builtins, golden files, the reference driver, and the claim labels.
README: the exit-code line and the Terminal-Bench paragraph now say "verified" needs --judge <another model> or person-approved checks.
bench/harbor/README.md: the adapter passes no judge, so runs end passed-own-checks (exit 3) unless MAAT_JUDGE_* is set. The exit code never fails a trial.

Non-blocking

normalizeModelId aliases: it now drops -instruct, -chat and -it. qwen/qwen3-235b-a22b-2507 and Qwen/Qwen3-235B-A22B-Instruct-2507 are the same model, and so is gemma-3-27b-it. This errs toward "same", which fails closed.
A weak judge's boolean-printing check: fixed in Redraft checks that pass before the work; discrimination gate opt-in #34. That PR adds a bar-time read of the check's own verdict line and an L16 extension; details are on Redraft checks that pass before the work; discrimination gate opt-in #34.
Empty worker id: unchanged on purpose. It fails closed, so with no worker name no judge counts as independent (a lost verified, never a wrong one). I didn't audit which backends can leave both ids empty.

Round 1 (21:20 UTC)

Item 1 (advisory ordering) is the same as round-2 item 2 above. Item 2 (docs) is the same as round-2 item 3.
Item 3 (base): already retargeted to claude/completion-reserve; CI ran green on that base.
sealDraft reading process.env.MAAT_JUDGE_MODEL: left as is, consistent with judge.ts. Moving both to env("JUDGE_MODEL") is a separate change.
"verified (your checks)" wording for interview-approved checks: kept.
Drafting the reference with the judge model: still open as a follow-up.

Bug-hunt (verdict correctness)

1. A worker-written golden file counted as a value assertion. Fixed in 2a1519a (src/golden.ts).

At the seal, before the first step, Maat records what each expected-looking diff/cmp operand (expect, golden, want, baseline, answer, correct) holds.
At the tier, a check whose value tag rests only on such a file keeps the tag only if the file existed then and is byte-identical now. A literal comparison or a heredoc/<( ) still counts as before.
assertsValue(run, goldenPredates) takes the pre-work record, and tierOf takes valueUnproven.
A late check has no pre-work record, so its golden file proves nothing. Redraft checks that pass before the work; discrimination gate opt-in #34 adds the record from the turn-start copy.
Tests in a turn:
- The worker writes both out.txt and expected.txt: passed-checks, not verified.
- The task ships expected.txt: verified.
- The worker rewrites the shipped expected.txt: not verified.

2. The reference check passed when the deliverable called os._exit(0). Fixed in efcbdb4.

The driver no longer runs the work's code. For each input it starts a fresh child (runner.py). The child loads check.py, deletes reference and INPUTS from the module, calls run_deliverable, and sends repr() of the result on its own pipe.
A child that ends without a message (os._exit, sys.exit, a crash) gives "the deliverable ended the process …". That matches nothing, so the check fails: the work is at fault, not the reference.
Expected values exist only in the driver's memory. They are never written to disk and never passed to a child.
The driver's last line is REFERENCE COMPARED n/n. src/bar.ts requires it, with n > 0 and every judged input compared, before an exit 0 counts as a pass. Without it the check fails.
Tests:
- Your repro: os._exit(0) at import with a wrong f. Refused, with "the deliverable ended the process" in the failure.
- sys.exit(0) at import, and os._exit(0) inside the call.
- A right import-style deliverable still verifies.
- A deliverable that looks for reference() in sys.modules gets nothing.
- referenceComparedAll unit cases.
Still open: the child runs as Maat's uid. Same-uid code can still find check.py in its 0700 temp directory and recompute an expected value, though it can no longer end the driver or see stored answers. Making the files unreadable needs Run the worker's tools as another user: --worker-user (builds on #27) #29's worker uid. Run the worker's tools as another user: --worker-user (builds on #27) #29 is not in this stack. When it lands, the child can be launched as the worker user (the driver already isolates it in its own process for that reason).

Replay

The builtin rule changes no bench verdict: none of today's 259 bench trees has a done.yml, so no builtin ever ran there.
The golden rule and the bar-time verdict read are measured over today's lanes on Redraft checks that pass before the work; discrimination gate opt-in #34 (prework_screen_replay.py, new section). Results are on that PR.

The gate from the previous commit is now opt-in (--require-discriminating, MAAT_REQUIRE_DISCRIMINATING=1; --review-advisory implies it). Replayed over 2026-10-07's lanes it cost more right verifieds than it removed wrong ones, so default tiering is #32's again. The pre-work try is still journalled and shown on each receipt row, late checks are still tried on the turn-start copy, and the coverage notes stay. The cause is fixed where the checks are drafted instead. --criteria auto copies the project before the first step, and each drafted check is tried on that copy before it is sealed. One that already passes there cannot show the task was done: it goes back to the drafter once with "this check passes before any work, so it cannot show the task was done; make it fail on the current tree", and is dropped if the redraft still passes. Checks that cannot fail by construction get the same treatment through an always-on lint (L16): a trailing `|| echo ...`, `|| true`, `; exit 0`, `find ... -exec ... \;` (find ignores what -exec returns), PASS/FAIL printed with no failing exit, and, at the try, FAIL printed with exit 0. Drops are journalled as check-lint-dropped with rules P1-passes-before-work and L16-*. bench/local/prework_screen_replay.py runs today's 514 sealed drafted checks against their tasks' pristine trees: 112 would have been sent back (63 by the lint, 49 passing before the work). Of the 29 independent verifieds, 15 rested only on such checks: 3 of the 4 wrong ones (q235 http-json-server, q235 csv-clean, ml4b config-migrate) and 12 right ones, which would be judged by the redraft instead.

The copy taken at turn start for late checks lived in $TMPDIR, owned by the uid the worker's bash runs as. A worker could edit it so a check that passes on the pristine tree "failed before the work" and counted as discriminating. preWorkCopy takes a content digest when the copy is made (0700 parent; under --worker-user that is Maat's private state dir) and checks it before and after every late try and every seal-time screen. A copy that changed counts as not tried, journalled with tampered: true. A late check that names the project's absolute path reads the finished tree, so it is not tried either. Golden files a late check diffs against are recorded from the copy, so the #32 golden rule covers late checks too. The late try now honours a pending cancel, the copy's size and timing are journalled (pre-work-copy), and a missing copy is journalled instead of skipped silently. CHANGELOG, README and bench/harbor/README.md describe --require-discriminating, passed-untested, the P1 redraft and drop, the verdict read and the shell change.

solvyxtech · 2026-10-08T05:55:31Z

Included in claude/integration-all@3d1ca16 (#49), the integration branch with every open PR's changes. That branch is now the standard build and supersedes this one. The conflict choices are listed in #49.

solvyxtech added the claude Owned by claude label Oct 7, 2026

solvyxtech commented Oct 7, 2026

View reviewed changes

This was referenced Oct 7, 2026

Keep unattended workers on the task: bench isolation, scoped file tools, no-progress guard #33

Open

Redraft checks that pass before the work; discrimination gate opt-in #34

Open

solvyxtech changed the base branch from claude/sync-local-work to claude/completion-reserve October 7, 2026 22:56

solvyxtech force-pushed the claude/independent-verified branch from fc5aa1d to 064049f Compare October 7, 2026 22:56

solvyxtech commented Oct 8, 2026

View reviewed changes

solvyxtech mentioned this pull request Oct 8, 2026

Send an output cap, and shrink it when the completion reserve overflows the window #36

Open

solvyxtech mentioned this pull request Oct 8, 2026

test: adversarial regression suite for a hostile worker #39

Open

This was referenced Oct 8, 2026

Close the adversarial gaps with no PR yet #41

Open

Post-work audit: checks drafted after the work, gated on the pre-work tree and on mutants #42

Open

Integration: all 2026-10-07 fixes #49

Merged

Navigation Menu

Repository navigation

Verified needs a check the worker did not write - #32

Verified needs a check the worker did not write#32
solvyxtech wants to merge 3 commits into
claude/completion-reservefrom
claude/independent-verified

solvyxtech commented Oct 7, 2026

Uh oh!

solvyxtech left a comment

Uh oh!

solvyxtech commented Oct 7, 2026

Uh oh!

solvyxtech left a comment

Uh oh!

solvyxtech commented Oct 8, 2026

Uh oh!

solvyxtech commented Oct 8, 2026

Uh oh!

solvyxtech commented Oct 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

solvyxtech commented Oct 7, 2026

Why

What changes

Tests

Uh oh!

solvyxtech left a comment

Choose a reason for hiding this comment

Pairing rules: all clear

Must fix

Should fix

Nits / notes

Uh oh!

solvyxtech commented Oct 7, 2026

Uh oh!

solvyxtech left a comment

Choose a reason for hiding this comment

Blocking

Non-blocking

Uh oh!

solvyxtech commented Oct 8, 2026

1. assertsValue counts a worker-written golden file as a value assertion — blocking, medium confidence (src/tiers.ts:77-81)

2. The reference check passes with no comparison when the deliverable calls os._exit(0) on import — blocking, high confidence (src/reference.ts:171-209)

Uh oh!

solvyxtech commented Oct 8, 2026

Round 2 (00:11 UTC)

Round 1 (21:20 UTC)

Bug-hunt (verdict correctness)

Replay

Uh oh!

solvyxtech commented Oct 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

1. `assertsValue` counts a worker-written golden file as a value assertion — blocking, medium confidence (`src/tiers.ts:77-81`)

2. The reference check passes with no comparison when the deliverable calls `os._exit(0)` on import — blocking, high confidence (`src/reference.ts:171-209`)