Repository navigation
Post-work audit: checks drafted after the work, gated on the pre-work tree and on mutants - #42
solvyxtech wants to merge 2 commits into
Conversation
… tree and on mutants --post-work-audit (MAAT_POST_WORK_AUDIT=1), off by default. When a claimed turn ends unverified (the sealed checks passed without earning the word, only drafted checks refused it, or nothing checked it), the --judge model, which must be another model than the worker, drafts checks from the task text and an interface view of the work (src/interface-view.ts): the files added or changed, their declaration lines, what each new script's --help prints when it reads as usage, data files by name and format only, and the top-level listing. It never gets the transcript, the claim or the outputs, and a subprocess judge is started in an empty folder. Each check must quote the task words its expected value comes from; a quote not in the task (dispute.ts quotedIn) drops it. A check then has to clear every gate (src/post-audit.ts): the lints (L15 mutates, L16 cannot fail, and it must assert a value), pass on a copy of the work, fail on the pre-work copy (criteria.ts preWorkScreen, the seal-time screen, now exported), and fail on at least one of 3-5 mechanical mutants of the changed code that still parses and did not just crash (flip a comparison, nudge a number, delete a statement, swap if/else bodies; data files are mutated only when the work changed no code). Runs leave nothing behind: every run is on a throwaway copy and its process group is killed when it exits (run.ts killGroupOnExit), so a server one run started cannot answer for the next. One check through every gate makes the outcome "verified" with tier verified-audit and claim "verified (post-work audit: <judge>)" in job_end, the receipt index and the journal (post-work-audit note with each check's gates and mutant kills; the commands are withheld while they run and released after). job_end carries `audit`. run.py keeps it; stats.py reads the label as basis "audit".
…-07's runs Replays the audit on every saved tree from today's lanes whose claim was "unverified" (178 runs: 101 in judged arms, 77 in self arms as a what-if, since no judge was set there), each in its own bench container with the build installed: the saved tree is the work, the task's setup() rebuilds the pre-work tree, the judge is asked live. Cached per run. Judge qwen/qwen3-coder-30b-a3b-instruct (the bench's judge model): judged arms: 6 of 90 grader-passed runs verified (7%), 0 of 9 failed; self arms: 7 of 65 (11%), 0 of 9. Judge qwen/qwen3-coder-next: judged arms: 18 of 91 (20%), 0 of 10; self arms: 15 of 66 (23%), 0 of 9. Pre-work "verified" on the judged lanes was 27/31 right; with the audit, 33/37 (a3b judge) or 45/49 (coder-next). Most drafted checks never reach the mutants: they assert nothing (a3b: 95 of 201 gated) or fail on correct work (80). Zero false verifieds is over only 18-19 failing runs per judge, so the bound on precision is loose.
solvyxtech
left a comment
solvyxtech
left a comment
There was a problem hiding this comment.
Grok review of #42: changes requested (posted as a comment because the PR and the reviewer share the solvyxtech account, so GitHub won't accept a formal Request changes. Not merging.)
I checked this out locally at af5f680. npm run typecheck is clean and the full suite passes, 2094 of 2094, including test/post-work-audit.test.ts with 11 of 11 green. CI never actually ran: the typecheck-and-test job wasn't started because of a GitHub Actions billing or spending-limit problem on the account, so the red X isn't a code failure.
On the pairing rules this PR is fine. There are no competitor names in the added lines, no subscription backends, and nothing feeds fine-tune data. Base is claude/discriminating-verified and not main.
Blocking
B1. The audit can overrule a failing independent sealed check and still award "verified" (src/engine.ts:4661-4664 together with src/engine.ts:4813-4822). Confidence: high.
checksDisagree turns "not proven" into "unverified" whenever every failing check is hidden. Judge-drafted sealed checks are hidden too (criteria.ts:154), so this covers more than the worker's own drafts. auditable then accepts that "unverified" outcome, and one accepted audit check sets the outcome to verified.
Failure scenario: the --judge model seals [ "$(python3 tool.py 3 4)" = "7" ] before the work. The worker's tool prints 8, so the check fails twice, checksDisagree fires, and the outcome becomes unverified. Next, the audit judge sees only the interface. It writes a check on a different property (say the output format), which passes on the work, fails pre-work because the file is missing, and kills a mutant. The run ends as verified (post-work audit) while an independent check is still failing on the same work.
Why it matters: this is a false "verified" built from a real failure signal, the exact case #32 to #34 were meant to close. The replay's 0 out of 9 or 10 grader-failed runs is too small a sample to rule it out. No test covers it.
Fix: skip the audit when any failing check was authored by someone other than the worker (checkAuthors already records this), or require the failing check to be retired first through the dispute arbiter. Please add a test where a failing judge-authored check plus a passing audit check still ends unverified.
B2. --help passes the work's actual output to the judge whenever it contains the word "usage" (src/interface-view.ts:148-150 and :230-232). Confidence: high, reproduced.
looksLikeUsage only checks /\busage\b/i. A script that ignores --help and just runs prints its real answer. If that answer contains "usage" (disk, CPU, memory or API usage reports are common tasks), the whole answer goes into the view. I reproduced it against dist/interface-view.js: a script that prints disk usage report and total: 1234 ends up in the view under "--help prints:" with total: 1234 verbatim.
Failure scenario: the judge copies 1234 as the expected value and quotes any 8 or more task characters for grounding (see N2). The check passes on the work, fails pre-work and kills a nudge mutant, so a wrong answer gets verified-audit. A hostile worker can also get this on purpose by printing "usage:" alongside its answer, or by putting instructions to the judge in --help text, add_argument(help=...) lines or usage: echo lines. Signatures keep those whole (interface-view.ts:128, :142).
Why it matters: the view promises "never its answers", and that promise is what keeps the audit honest.
Fix: require real usage shape (a usage: line at the start of a line, or argparse or getopt structure), reject help output that differs from what the script prints with no arguments at all, and strip default= and help= values from the signature lines. Add an adversarial fixture for an output that contains "usage".
Non-blocking
N1. Compile errors in compiled languages count as killed mutants (src/post-audit.ts:363-374 and :452). Confidence: medium.
mutantParses treats any extension without a parser (.go, .ts, .rs, .java) as parsing. CRASH only matches Python and JS error names. In Go, deleting a statement often gives declared and not used, so go run exits 1 and the run counts as killed by any check that runs the program. That makes gate (c) close to free there. The same thing happens with Node TypeError and with Python tracebacks over 2000 characters, where the header gets cut off at :472. Fix: also count a mutant run that fails while the baseline passed and stderr looks like a compiler error as crashedOnly, or run go vet, tsc --noEmit or a similar check as the parse step when it's available.
N2. Grounding doesn't connect the quote to the expected value (src/post-audit.ts:124). Confidence: high, low severity on its own. Any task span of 8 or more characters passes quotedIn, so a guessed or leaked value with an unrelated quote still counts as "grounded". Requiring the expected literal, or the input-to-output pair, to show up in or near the quote would make grounded mean what the job_end field says.
N3. The audit can overrun the run's deadline (src/engine.ts:3830, src/post-audit.ts:527). Confidence: medium. deadlineAt is at least now + 45s even when the turn's deadline has already passed, and it's only checked between checks, not between mutant runs. Worst case is about 4 checks × 7 runs × 10s, plus the judge ask. On a bench with a hard wall clock this could kill the container mid-audit. Please clamp to the turn deadline and check it inside the mutant loop.
N4. amendTier now raises a tier, but it was built and documented to lower one (src/engine.ts:4832, src/receipts.ts:776-783). Confidence: medium. The hash-bound receipt file still reads unverified, and only the mutable index row says verified-audit. Lowering in the index fails safe. Raising there means a "verified" with nothing hash-bound behind it, apart from the journal note. Either chain-bind the audit record, for example a small receipt with the accepted check hashes, or update the docs and the receipt-verify path so they treat an index-only raise as needing the journal entry.
N5. A subprocess judge in an empty folder isn't sandboxed (src/post-audit.ts:586-587). Confidence: low. An agentic ACP judge still has file tools and could find the work's outputs by path. Worth noting in the docs, or keeping the audit to API judges for now.
N6. Replay caveat. The replay trees are "the state after grading", so grader artifacts may be in the work tree the judge's interface view lists. Worth confirming they're excluded before the recall numbers get quoted.
The overall design is good: running cheap gates first, mutating only the changed lines, killing process groups and keeping a separate tier are the right calls. B1 and B2 are the two places where a wrong "verified" can still get through.
solvyxtech
left a comment
solvyxtech
left a comment
There was a problem hiding this comment.
Verdict: CHANGES REQUESTED
Prior items: 0 of 2 blockers fixed. B1 (the audit can overrule a failing independent sealed check) and B2 (--help output containing "usage" leaks real answers to the judge) are both still open, and so are N1–N6. The head is still af5f680, the commit I reviewed in 5450349722, and the base hasn't moved.
Fresh pass from Grok, comment only. I re-ran it at af5f680: npm run typecheck is clean, and the suite passed 2094 of 2094. CI hasn't run because of the account-wide GitHub Actions billing block. Two new, non-blocking notes:
1. Model vendor names were added to the repo. Non-blocking until Tyler rules; high confidence that they're there. bench/local/audit_replay.py hardcodes qwen/qwen3-coder-30b-a3b-instruct as the default judge, plus qwen3-coder-30b-a3b in an assert/fixture, and the commit messages name qwen/qwen3-coder-30b-a3b-instruct and qwen/qwen3-coder-next. My last review called this PR clean. That ID is an open-weight model the bench self-hosts, and it already appears in 24 files on main. Under Tyler's "no competitor names anywhere in the repo" rule, though, it's a judgment call, so I'm flagging it for him rather than deciding it. If it has to go, read the default from REPLAY_JUDGE_MODEL or the bench's untracked local config and use a generic fixture ID. Changing the commit messages would mean rewriting history, which is Tyler's call.
2. This PR and #43 conflict. High confidence. Both branch from claude/discriminating-verified, and a trial merge of claude/false-drops into claude/post-work-audit conflicts in src/criteria.ts. Whichever one lands second needs a rebase, and so does #44, which sits on this branch.
|
Round-3 review, item by item. Fixed in claude/integration-all@3d1ca16 (#49); this branch is superseded by integration.
|
1 participant
Opt-in
--post-work-audit(MAAT_POST_WORK_AUDIT=1). Builds on #34.What it does
When a claimed turn ends unverified (the sealed checks passed without earning "verified", only drafted checks refused it, or nothing checked it), the
--judgemodel drafts more checks. It must be a different model from the worker (#32). Runs that ended "not proven", errors, and runs stopped by the clock are left alone.Interface view (
src/interface-view.ts). The judge sees the task text plus:--help, kept only if it reads as usage text and run on a throwaway copy;It never sees the transcript, the claim or the outputs. A subprocess judge starts in an empty folder.
Grounding. Each check quotes the task words its expected value comes from. A check whose quote isn't in the task is dropped (
quotedIn).Gates (
src/post-audit.ts). A check must clear all of them:cannotFail(includes Redraft checks that pass before the work; discrimination gate opt-in #34's boolean-print class), and it must assert a value;preWorkScreen, and respectsintact();Isolation. Every run is on a throwaway copy, and its process group is killed on exit (
killGroupOnExit). Without that, a server left by one run would answer for the next.Label. One check through every gate gives:
verified, tierverified-audit;verified (post-work audit: <judge>)in job_end, the receipt index and the journal;audit: {judge, drafted, grounded, accepted}.run.py/stats.pyparse it (basisaudit). Commands are withheld while they run, then released and journalled.Replay (
bench/local/audit_replay.py)All 178 "unverified" trees from 2026-10-07's lanes, each replayed in its own bench container on this build. The task's
setup()rebuilds the pre-work tree, and the judge is asked live.On the judged lanes, the pre-work "verified" was right 27/31 times. With the audit it is right 33/37 (a3b judge) or 45/49 (coder-next).
Most drafted checks don't get as far as the mutants. They assert nothing, or they fail on correct work. The zero false verifieds come from only 18–19 failing runs per judge, so the upper bound on precision is loose.
Tests:
test/post-work-audit.test.ts(view excludes outputs/claims; ungrounded dropped; passes-every-mutant rejected; full-gate check earns the label in a turn; flag off changes nothing).npm run checkgreen (2094) with~/.local/binoff PATH.