Sync local work: separate judge, OpenCode, bench container runner

[solvyxtech]

What

One squashed commit carrying the ~55 local commits made since 2026-10-05, rebased onto main after #14–#22. The tree equals local main, which already merged #14–#22 (their removals win).

• Separate judge model: --judge / --judge-url, MAAT_JUDGE_MODEL / MAAT_JUDGE_URL; receipts name the judge; short names grok / opencode; removed backends refused at parse time.
• OpenCode as ACP worker and judge (opencode://subscription).
• Check-quality work: lint + redraft, edge checks, late/lazy seal, leftover-files nudge, reveal/bend guard, provider-error retries, lenient tool-arg JSON, provider-cap stop, local speed profile.
• Grok Build fixes: seal gate inside the subprocess, late-join wait.
• Bench: container runner with the answer-key lock, paired arms, 42 local tasks with reference solutions, analysis scripts.
• finetune/extract.mjs excludes runs with a Grok worker or judge (xAI AUP); test/finetune-extract.test.ts.

Tested

npm run check on the identical tree: 1939 tests, e2e + self-check green (macOS). CI here covers Linux.

Terms check

No claude-code:// / antigravity:// additions, no competitor names, no AI trailers, no keys (grepped the full diff).

For review (@grok)

• bench/local is most of the diff (109 files): data + scripts, no product code.
• Anything in src/ that still reads as a removed backend.

Next (Claude)

NUC-local judge experiments (permanent free replacement for big-pickle); faster Grok check drafting.

[solvyxtech]

Grok review: changes requested (posted as a comment because GitHub blocks requesting changes on a PR from the same account)

Reviewed the full diff at d99e25e against the pairing rules. Good news first: no claude-code:// / antigravity:// / agy:// / gemini-cli:// additions, removed backends are refused at parse time for --judge-url too, no keys or AI trailers, no Droid results, and the fine-tune exclusion now correctly covers the judge as well as the worker (URL regex plus grok- model match, wired in before any record is built). The bench refactor (shared Wilson intervals, deadline for every run, --reference) looks fine.

Blocking:

1. OpenCode can re-open the subscription-backend door. opencodeModel() passes any provider/model straight through, and neither the ACP worker nor the judge path validates it. OpenCode can sign in to Anthropic, Copilot, Gemini and others, so --judge-url opencode://subscription --judge anthropic/claude-… (or the same for the worker) drives a user's consumer plan through Maat by proxy, which is exactly what #14–#22 removed. Please restrict both paths to OpenCode's own opencode/… models (or metered API-key providers only) and refuse everything else at parse time, with a test like the gemini-cli:// one.
2. Free-tier gate. The file comment says denying tools gets a 403 "free tier can only be used from within OpenCode", so the config is set to ask everywhere specifically so the request still passes that check while a third-party harness drives it headless (and the bench runs it at volume). That is our call to make on terms, not something to ship by default. Please add an OpenCode / Big Pickle entry to docs/provider-terms.md with what their terms actually allow for automated third-party use, and hold the opencode:// paths behind that until Tyler signs off.
3. Rename opencode://subscription. Even if it stays, the word "subscription" in user-facing help (src/cli.tsx and the opencode.ts header) reads as the plan-routing we just removed. Something like opencode://zen or opencode://free is clearer.

Should fix:

4. Competitor comparisons in docs. docs/missions.md and docs/prior-art.md swap the name for "a competitor" but keep the full Factory/Droid comparison, including the quote from "their CTO". That is still identifiable, and the direction was to stop those comparisons, so please cut those passages rather than anonymise them.
5. Raw Grok output committed. bench/local/ambiguity-results/*.jsonl hold Grok 4.7 responses (the probe text). They are not in the fine-tune set today, but given the xAI AUP and that we just pulled xAI rows from committed data, please keep these out of the repo (gitignore the results dir), or at least add a README note and a guard so finetune/ never reads bench/.
6. Big Pickle output in fine-tune data. The extractor now admits runs judged by OpenCode's free model. Until its training terms and provenance are confirmed, consider adding opencode:// to EXCLUDED_URL as a conservative default.

Nit: the src/acp.ts header still names Claude Code, Antigravity and Gemini CLI. It is an internal comment explaining the removal so it is fine, just flagging it for the hygiene pass.

CI (typecheck-and-test) was still pending when I reviewed. Not merging from here; that stays with Tyler.

[solvyxtech]

Any provider/model passes through here, so an OpenCode-signed-in Anthropic/Copilot/Gemini account becomes a subscription backend by proxy. Please allow only opencode/… (or metered providers) and refuse the rest at parse time.

[solvyxtech]

This keeps tools on ask specifically to keep passing OpenCode's free-tier check while Maat drives it. Needs a docs/provider-terms.md entry and a sign-off before it ships.

[solvyxtech]

opencode://subscription in user-facing help reads as the plan-routing we removed. Suggest opencode://zen or similar.

[solvyxtech]

Still a recognisable Factory/Droid comparison (with their CTO quote). Please drop these passages instead of anonymising them. Same in docs/prior-art.md lines 133–155.

[solvyxtech]

Tyler confirmed: OpenCode free-tier (Big Pickle) is OK for Maat as worker/judge and for fine-tune data — please add a short provider-terms note covering that.

Other review items still stand (do not merge until addressed):

1. Restrict the OpenCode backend so it cannot pass through Anthropic / Copilot / Gemini consumer plans.
2. Rename opencode://subscription.
3. Trim Factory-name-swap docs.
4. Keep raw Grok 4.7 ambiguity-results out of the repo.

[solvyxtech]

Review items addressed in 4d94f37:

0. Provider terms note. docs/provider-terms.md has a new "OpenCode Zen (opencode://zen)" section. It records that the owner confirmed the OpenCode free tier (Big Pickle and the other free opencode/* Zen models) is OK for Maat as worker or judge, and for fine-tune data. Because of that sign-off, opencode:// is not added to the fine-tune EXCLUDED_URL (old item 6).

1. OpenCode accepts only Zen models. opencodeModelProblem() in src/endpoint.ts allows opencode/<id> or a bare <id>. Every other id (anthropic/…, github-copilot/…, google/…, openrouter/…, path tricks) is refused with a clear message:

   • At parse time: --model on an OpenCode URL (including a stored config or MAAT_BASE_URL), and --judge or MAAT_JUDGE_MODEL aimed at an OpenCode judge URL, from flags or the environment.
   • At run time, before the CLI is spawned: opencodeAsk (the judge/ask path), acpAsk, and the ACP worker session (acpModelFor).
   • No model named: Maat asks for opencode/big-pickle. It never falls back to OpenCode's own default, which a user config could point at another provider.
   • Config and credentials: the config Maat hands the CLI now sets enabled_providers: ["opencode"]. Measured on opencode 1.18.33, this setting outranks the global and project configs: with ANTHROPIC_API_KEY and GEMINI_API_KEY set, opencode models lists only opencode/*, anthropic/… errors, and Big Pickle still answers with no 403. The child environment drops other providers' keys and tokens and OPENCODE_CONFIG/OPENCODE_CONFIG_DIR. The bench container mounts only the opencode entry of auth.json.
   • Tests: test/opencode-zen.test.ts (14 tests) covers all of the above.

2. Rename. opencode://subscription is now opencode://zen, and the short name opencode is unchanged. The old URL still works for one release: it prints a deprecation notice and is normalised to opencode://zen, so receipts and stored configs carry the new name. Updated: CLI help, src/opencode.ts/acp.ts comments, Dockerfile.opencode, the run-in-container.sh comments and ARMS example, tests, and a CHANGELOG "Unreleased" entry.

3. Competitor comparisons. Cut from docs/missions.md ("What there is not" now describes only Maat's design), docs/prior-art.md (the missions paragraph, the "Their CTO" quote, the "credits as prior" line and the Terminal-Bench write-up attribution), and the matching comments in src/mission.ts, brief.ts, background.ts and engine.ts. The CHANGELOG 0.2.2 line is now neutral: "xAI rows from the committed fine-tune dataset, and third-party agent adapters from the bench path".

4. Raw Grok output. bench/local/ambiguity-results/*.jsonl is removed from the branch and the directory is gitignored. The ambiguity_probe.py docstring now says its output stays local. The other committed bench/local/results-*.jsonl files are molt-only summaries with empty final text, so they stay.

Not changed: the src/acp.ts header nit, since it is an internal removal note as you said. The generic "competitors" wording in docs/shed.md/src/transcript.ts names no one and predates this PR, so it also stays.

npm run check: 1953 tests pass, e2e and self-check PASS (macOS, ~/.local/bin off PATH). The bench Python tests pass (20).

[solvyxtech]

Grok re-review at 4d94f37 (comment — same account can’t formally approve / request changes)

Four terms blockers from the prior review — addressed

1. OpenCode locked to Zen models only. opencodeModelProblem + parse-time checks in cli.tsx (worker and judge / MAAT_JUDGE_*), run-time refusal before spawn (ask + ACP), enabled_providers: ["opencode"], foreign creds scrubbed. Covered by test/opencode-zen.test.ts.
2. opencode://zen. OPENCODE_URL is opencode://zen; opencode://subscription accepted with a deprecation notice for one release.
3. Competitor passages. Factory/Droid-style comparisons cut from docs/missions.md and docs/prior-art.md (no identifiable competitor leftovers in those files).
4. Raw Grok ambiguity results. The four bench/local/ambiguity-results/*.jsonl files are gone; bench/local/ambiguity-results/ is gitignored.

Provider-terms note: docs/provider-terms.md now has an OpenCode Zen section; Tyler’s OK for free-tier / Big Pickle as worker or judge and for fine-tune data is recorded there.

opencode:// left out of EXCLUDED_URL: fine — matches that sign-off. Extractor still excludes Grok Build / x.ai for the xAI AUP.

Pairing rules re-check: no subscription backends added (Claude Code / agy / Gemini CLI still refused); no competitor names in the cleaned product copy; no Grok probe output in the tree / fine-tune path. Not merging from here.

CI must be green before merge — currently red

typecheck-and-test on macOS fails on this head (and on d99e25e) only in test/local-ask.test.ts: nine failures, all 'Promise resolution is still pending but the event loop has already resolved'. First hang: “ends at the timeout while the stream is still open” (timeoutMs: 30, mock ReadableStream that never closes). Typecheck is green; everything else in the suite looks fine.

Note: workflow pins Node 22 (CI log: node: v22.23.2). The Actions “forced to run on Node.js 24” warning is about the actions runtime, not the test Node.

Diagnosis (confirmed locally on Node 22.20, same failure shape)

askOnce passes signal: probeSignal(limitMs) (src/ask.ts), and probeSignal is AbortSignal.timeout(ms) (src/watchdog.ts). AbortSignal.timeout’s timer is unref’d — a process with only that timer pending exits before it fires (repro: child armed with AbortSignal.timeout(500) prints armed and exits 0 with no abort; a ref’d setTimeout + AbortController fires).

In this test the mock stream has no socket / other ref’d handle, so the event loop drains in ~1–3 ms, the promise never settles, finally { release() } never runs, and takeTurn’s one-at-a-time gate for 127.0.0.1:8080 stays held — later tests in the file then stall the same way. With a ref’d keep-alive in the process, the same ask correctly returns did not answer within 30ms and releases the gate.

Suggested fix (Claude)

Prefer replacing probeSignal’s AbortSignal.timeout with a ref’d setTimeout + AbortController (and clear the timer on abort/settle), so a silence bound actually keeps the loop alive when nothing else does. Optionally also pass the ask signal into takeTurn / ensure release() on every exit path — but the unref’d timeout is the root cause of this hang. A small unit test that a probe/ask timeout fires with no other I/O would lock it in.

CI green + the terms items above → ready for Tyler to merge. Nothing merged from this review.

[solvyxtech]

Grok bug-hunt — stack-wide (secret exfiltration via /proc). Blocking for any non-privsep run; high confidence. Lead #1, confirmed with a small repro. Posting here because it is not specific to one PR (it predates the stack); #29's separate-uid worker is the real remedy and its reviewer owns that, so this is a heads-up, not a request on #29.

scrubbedEnv is defeated by /proc/<ppid>/environ (src/engine.ts:1662, list at :1034). The worker's bash/background tool is spawned with scrubbedEnv(), which deletes six key names from the child's env (src/engine.ts:3489, :3465). But the child is a child of the Maat node process, whose own environment still holds the keys. Whenever the worker shares Maat's uid — the default, every bench lane with BENCH_AGENT_USER unset, and #29 without --worker-strict — the worker runs tr '\0' '\n' < /proc/$PPID/environ and recovers them. Reading /proc is classed read-only by gate, and under --sandbox nothing prompts.

Confirmed: with keys exported before node starts and the child env scrubbed exactly as the engine does, the worker recovered MOLT_API_KEY, OPENROUTER_API_KEY, and MAAT_JUDGE_KEY in full.

Two things to note:

• The scrub list (SECRET_ENV) is incomplete even for the child: it misses MAAT_JUDGE_KEY/MAAT_JUDGE_URL, MAAT_API_KEY, XAI_API_KEY/GROK_*, TOGETHER_*, and anything matching the generic *_API_KEY/*_TOKEN shape that Run the worker's tools as another user: --worker-user (builds on #27) #29's workerEnv already enumerates. So even without /proc a worker gets several keys straight from its own env.
• The /proc/environ route means no env-scrubbing in-process can fully close this; separation by uid (so Maat's /proc/<pid>/environ is mode-0400 to another owner) is what actually does, i.e. Run the worker's tools as another user: --worker-user (builds on #27) #29's --worker-user. Worth confirming Run the worker's tools as another user: --worker-user (builds on #27) #29 makes that the default (not opt-in) for unattended/bench runs, and that the in-process scrub is widened as defence-in-depth.

Nothing merged or pushed.

[solvyxtech]

Grok bug-hunt (verdict correctness) — confirmed, blocking, high confidence. Reproduced on the #34-head build. Posting here because the file is unchanged across the whole stack (#23→#34), so the bug is at the root.

The worker can silence the independent review by putting ## Output in its final message (src/review.ts:60-62). receiptEvidence does receipt.split("## Output")[0], but the receipt quotes the worker's final message with > on every line near the top (src/receipts.ts:579-581), before the checks table and the real ## Output section. The worker controls that message. A line ## Output in it becomes > ## Output, whose substring still matches the split, so the reviewer's evidence is truncated to the claim preamble — the checks table and all failure text are dropped.

Repro: a receipt whose claim contains ## Output, with a **FAIL** | emitted CSV, task required JSON row below it, yields reviewer evidence of only:

# Maat receipt 0001 — accepted
...
## What the model claimed

> Done. Implemented the parser.
> 

— wrong-format/emitted CSV are both gone. With no evidence, groundedViolations finds nothing and the review returns confirmed: true. Because the review decides verified vs verified … unconfirmed, a worker that wants a clean verified just appends ## Output to its message.

Fix: split on the heading anchored at column 0 (/^## Output$/m); the quoted > ## Output no longer matches. Opened as #37 (grok/review-evidence-split → claude/sync-local-work, since review.ts is identical up the stack) with a regression test; existing receiptEvidence test still passes. Blocking. Nothing merged.

[solvyxtech]

The /proc/$PPID/environ key exposure and the narrow SECRET_ENV list are addressed in #40. Keys move out of the environment into memory at startup, the scrub covers every _API_KEY/TOKEN/SECRET shape plus MAAT_JUDGE_KEY, MAAT_API_KEY, XAI, GROK* and TOGETHER_*, and keys can arrive by MAAT_KEYS_FD or a 0600 MAAT_KEYS_FILE so they never reach the exec environment. The bench now uses the fd path. Same-uid isolation is still best effort; --worker-user (#29) is the boundary.