Integration: all 2026-10-07 fixes

.github/workflows/check.yml

@@ -3,7 +3,10 @@ name: check
 on:
   push:
     branches: [main]
+  # Stacked PRs target claude/* branches, not main; both get the checks
+  # (privsep-linux included).
   pull_request:
+    branches: [main, "claude/**"]
 
 jobs:
   typecheck-and-test:
@@ -28,3 +31,48 @@ jobs:
           node dist/cli.js --version
           # doctor may exit >1 when default localhost endpoint is down (no Ollama in CI)
           node dist/cli.js doctor || true
+
+  # The full suite on Linux, where /bin/sh is dash: macOS's /bin/sh is bash,
+  # and a check that reads a shell's own wording (dash's "eval: x: not found")
+  # passed there and failed here (#49 status sweep, N2).
+  test-linux:
+    runs-on: ubuntu-latest
+    env:
+      ELECTRON_SKIP_BINARY_DOWNLOAD: "1"
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 22
+          cache: npm
+      - run: npm ci
+      - run: npm run typecheck
+      - run: npm test
+
+  # Privilege separation (--worker-user, --check-user, --worker-strict) is
+  # Linux-first and needs root to switch users: its tests skip on macOS and
+  # run here, in a root container. The test creates its own worker and check
+  # users. The container gets CAP_SYS_ADMIN, so the worker's PID namespace and
+  # each check run's PID namespace and private /tmp can be made, and
+  # MAAT_TEST_REQUIRE_PIDNS=1 makes the namespace tests fail here rather than
+  # skip when they are not. Docker's default AppArmor profile on Ubuntu hosts
+  # denies mount(2) (needed for the namespace's own /proc and the check's
+  # tmpfs /tmp) even with the capability, so it is lifted for this container.
+  privsep-linux:
+    runs-on: ubuntu-latest
+    container:
+      image: node:22-bookworm
+      options: --cap-add=SYS_ADMIN --security-opt apparmor=unconfined
+    env:
+      ELECTRON_SKIP_BINARY_DOWNLOAD: "1"
+      MAAT_TEST_REQUIRE_PIDNS: "1"
+    steps:
+      - uses: actions/checkout@v4
+      - run: npm ci
+      - run: npm run test:build && npm run build
+      # The job container's HOME is /github/home, a runner bind mount the
+      # worker can read and root does not own, so Maat (correctly) refuses it
+      # as the judge's HOME. root's own HOME is private.
+      - run: node --test --test-reporter=spec dist-test/test/privsep.test.js
+        env:
+          HOME: /root

.gitignore

@@ -40,3 +40,8 @@ out-cli/
 *.tgz
 .molt/out/
 bench/local/work/
+bench/local/.api2.env
+
+# raw model output from bench probes stays local (xAI's AUP forbids using Grok
+# output to develop ML models, so transcripts are never committed)
+bench/local/ambiguity-results/

CHANGELOG.md

@@ -1,5 +1,152 @@
 # Changelog
 
+## Unreleased
+
+### Added
+
+- **`--review-executable`** (experimental, also `MAAT_REVIEW_EXECUTABLE=1`):
+  every `--review` objection must carry a read-only command that demonstrates
+  it. Maat runs it on a throwaway copy of the tree, with no API keys, tokens or
+  provider variables in its environment; an objection counts only when its
+  command reads the work (names a project path, or runs the project's tests or
+  build) and exits non-zero, or prints the value it named without that value
+  being in the command itself. A demonstrated objection shows the command
+  failed, not that it tested what the objection says. With no copy of the tree
+  available the command is not run. Each objection is recorded as counted,
+  refuted (the reviewer was wrong) or unchecked (it could not be run).
+- **Lean-budget guards.** `test/lean-budget.test.ts` runs seven scripted
+  scenarios (small task, 400 KB read, repeated re-reads, a long session that
+  sheds, malformed calls with huge arguments on the tool and `act` paths, a
+  command printing 1 MB) with hard ceilings on the largest request, the total
+  request characters and the steps of the turn; the build before the malformed
+  excerpt fix fails it by 20-60x. `bench/local/run.py` stops the lane with a
+  `STOPPED` row (`reason: "cost alarm"`) and exit code 3 when one run costs
+  more than max(5x the lane's running median, $0.10) or sends more than 1.5M
+  prompt tokens (`BENCH_COST_ALARM_X`, `BENCH_COST_ALARM_USD`,
+  `BENCH_TOKEN_ALARM`). See `docs/lean.md`.
+
+### Changed
+
+- **"Verified" needs a check the worker model did not write.** Each task
+  check now records who wrote it (the worker, a separate judge, a person, the
+  reference writer). A run is `verified` only when a passing check that ran
+  the work and asserted a value came from another model (`--judge`,
+  `MAAT_JUDGE_*`, compared by provider-normalised id), or a person wrote or
+  approved a passing command check (done.yml, a mission, criteria approved in
+  the window, `--criterion`). Otherwise the tier is the new
+  **`passed-own-checks`**: outcome `unverified`, **exit code 3**.
+  **This changes headless `--criteria auto` without `--judge` (or with a judge
+  that is the worker model under another name): it used to exit 0 and now
+  exits 3.** CI jobs and bench scripts that read exit 0 as success should
+  pass `--judge <another model>` or approve the checks.
+- **Builtins never make a run verified.** `work-landed`, `record-intact`,
+  `claims-grounded`, `work-accounted`, `spec-intact` and every other builtin
+  or `session` check say the turn was well-formed, not that the task is
+  right. A bar of builtins alone now ends `passed-checks` (unverified, exit 3)
+  where it used to end `verified`, and receipts say these checks are Maat's,
+  not "a person (your check)".
+- **An expected file counts only if it predates the work.** `diff out.txt
+  expected.txt` (or a golden/want/baseline/answer file) asserts a value only
+  when that file was in the project before the first step and is unchanged
+  at the claim. A worker that writes both files no longer earns "verified".
+- **The reference check runs the deliverable out of process.** Maat's driver
+  used to import an import-style deliverable into its own process, so a
+  module that called `os._exit(0)` ended the check with exit 0 before any
+  comparison, and it passed. Each input now runs in a fresh child; a child
+  that ends without a result is a mismatch, `reference()` is removed from the
+  module the work's code can reach, and expected values never leave the
+  driver's memory. A pass also needs the driver's final `REFERENCE COMPARED
+  n/n` line: an exit 0 without it is a failure.
+- **Drafted checks that pass before the work are redrafted, then dropped.**
+  With `--criteria auto` each drafted check is tried on a copy of the project
+  taken before the first step. One that already passes there cannot show the
+  task was done: it goes back to the drafter once, and is dropped if the
+  redraft still passes (journal: `check-lint-dropped`, rule
+  `P1-passes-before-work`). An always-on lint (L16) does the same for checks
+  that cannot fail by construction: `|| echo …`, `|| true`, `; exit 0`,
+  `…; echo $?`, `find … -exec … \;`, an `if … fi` with no failing branch,
+  and inline programs whose exit code does not depend on what they assert
+  (`python3 -c "print(x == y)"`, `node -e "console.log(a === b)"`, `jq`
+  without `-e`, `awk` with no `exit`, a printed PASS/FAIL).
+- **A task check that prints its failure has failed.** A drafted or approved
+  check that exits 0 but whose last line of output is `False`, `FAIL`,
+  `FAILED` or `NO` (or whose `echo $?` tail printed non-zero) is read as a
+  failure at the bar and before the work. `print(rows == expected)` printing
+  False on wrong work used to pass and earn "verified".
+- **`--require-discriminating`** (`MAAT_REQUIRE_DISCRIMINATING=1`, off by
+  default; `--review-advisory` implies it): "verified" needs an independent
+  value check that failed on the tree before the work and passes now.
+  Otherwise the new tier **`passed-untested`** ("passed checks that did not
+  test this work, not verified"): outcome `unverified`, exit code 3. Each
+  receipt row says how its check fared before the work.
+- **Checks are tried before the work under the shell the bar runs them
+  with** (bash for drafted checks), so a bashism no longer "fails before the
+  work" under dash and passes at the bar. dash's exit 2 for a missing
+  `sh script` is read like bash's 127. A try that timed out counts as not
+  tried, not as failing.
+- **The pre-work copy is checked before it is trusted.** Late checks (drafts
+  past the time cut, the reference check) are tried on a copy taken at turn
+  start. Its digest is taken when it is made and compared around every try;
+  a copy that changed during the work, or a check that names the project's
+  absolute path, counts as not tried. The copy's size and timing are
+  journalled (`pre-work-copy`).
+- **Checks that pass before the work are kept as refuse-only guards.** A
+  drafted check the pre-work screen drops (P1) is still sealed, tagged
+  `guard`: it runs at the bar and a failure refuses the claim, but it never
+  counts toward "verified", and receipts label it "refuse-only guard". Up to
+  four, beside (not inside) the four-check cap. Not kept: a duplicate of a
+  sealed check, and one that pins today's output of the project's program on
+  the project's own data (`node summarize.js transactions.json | jq -e
+  '.zoe == "-3.50"'`, `sqlite3 app.db 'PRAGMA user_version' | grep -q '^1$'`),
+  which on the 2026-10-07 lanes failed 13 finished trees the grader accepted.
+- **Fewer good drafted checks dropped before sealing** (measured with
+  `bench/local/drop_audit.py` over the 2026-10-07 lanes):
+  - the absolute-path rule reads what each program opens (a shell parser,
+    `src/shellwords.ts`), not the raw text: a sed program's `/g`, an awk
+    program, a grep pattern, a jq filter, printf's data and `sys.argv[1] +
+    "/old.py"` are not paths. Its four drops that day were all wrong.
+  - `./rotate.sh …` before rotate.sh exists is the deliverable missing:
+    tried before the work it now counts as failing there (it discriminates),
+    not as a command that could not run. A tool missing from PATH is still
+    broken.
+  - the mutation rule (L15) skips the value of `touch -d/-t/-r` and
+    `mkdir -m`, follows variables built on a mktemp directory
+    (`f="$d/x"; echo > "$f"`), and lets `> "$(mktemp -d)/out"` through.
+  - the cannot-fail rule (L16) no longer flags `…; exit 0` after an earlier
+    `exit 1` in the same shell, or `cond && exit 1 || exit 0`.
+  - a check that names the project by its absolute path is tried on the
+    pre-work copy, and run in the bar's throwaway copy, instead of reaching
+    the live folder.
+- **Claims say who stood behind them** on the terminal, the window, job_end's
+  `claim`, the receipt index and the receipt: `verified (independent checks:
+  <judge>)`, `verified (your checks)`, `passed own checks (<worker>), not
+  verified`. With `--review-advisory`, a review that contradicted the task or
+  did not run appends `, unconfirmed` / `, unreviewed`.
+
+- **OpenCode runs OpenCode Zen models only.** The OpenCode backend (worker and
+  judge) accepts `opencode/<model>` (e.g. `opencode/big-pickle`) and refuses
+  any other provider at parse time and at run time; the CLI is handed a config
+  that enables only the `opencode` provider, and no other provider's
+  credentials.
+- **`opencode://zen`** is the OpenCode endpoint (short name `opencode`). The old
+  `opencode://subscription` still works for this release, with a deprecation
+  notice.
+- **Unattended runs stop when they stop advancing** (`MAAT_NO_PROGRESS_CALLS`,
+  default 30, `0` turns it off; on for every unattended run on an HTTP
+  backend, not on ACP backends yet). A call advances when it changes a file
+  in the project, reads something not read before, runs a command not run
+  before, or gets output from a command that differs from every earlier run
+  of it (timings aside), wherever the work happens: a build into `dist/`, a
+  venv, a config under `/etc`. After N calls in a row that advance nothing the
+  model is told once to finish or stop; after N more the turn ends, the work
+  on disk is judged, and the receipt says `ended: no progress` (never
+  verified: the model did not say it was done).
+- **While an unattended job with hidden checks runs, the file tools stay in
+  the task.** `read_file`, `list_dir`, `grep` and `inspect` refuse paths
+  outside the project and under `.maat/` (except spilled output and
+  background logs, and paths the task names). bash is not blocked; this is a
+  budget control, not isolation.
+
 ## 0.2.2 — subscription backends removed
 
 ### Removed
@@ -9,8 +156,8 @@
   written. Stale `claude-code://`, `antigravity://`, `agy://`, and
   `gemini-cli://` configs now fail with a clear removal message. Grok Build
   (ACP) and the metered xAI API provider remain.
-- **Droid benchmarking** from the harbor/local bench path, and **xAI rows**
-  from the committed fine-tune dataset (provider-terms alignment). See
+- **xAI rows** from the committed fine-tune dataset, and third-party agent
+  adapters from the bench path (provider-terms alignment). See
   `docs/provider-terms.md`.
 
 ### Changed

README.md

@@ -200,7 +200,12 @@ molt acp                                    # serve ACP on stdio, for an editor'
 
 Exit codes: `0` the bar was met · `1` it was not · `2` usage · `3` finished
 without a verdict (no bar, or `--only`/`--skip` left required checks unrun, which
-is never reported as a pass).
+is never reported as a pass), or passed without earning "verified": every check
+that could prove the work was written by the worker model itself
+(`passed-own-checks`), with `--require-discriminating` none of them failed
+before the work (`passed-untested`), or nothing but builtins passed. "Verified" needs a
+passing check from another model (`--judge`) or a command check you wrote or
+approved; builtins never carry it.
 
 ## The bar
 
@@ -401,9 +406,19 @@ loop, and the judge is the commands. `docs/missions.md` has the whole thing.
 
 `bench/harbor/molt_agent.py` runs molt under harbor on Terminal-Bench 2.0,
 installing this tree's CLI into each task container. `bench/harbor/README.md`
-has the recipe. Headless runs get `--criteria auto`: the model drafts
-acceptance checks for the task before the work, they are sealed, and it is
-held to them.
+has the recipe. Headless runs get `--criteria auto`: acceptance checks for
+the task are drafted before the work, they are sealed, and the worker is held
+to them. Passing checks the worker model drafted for itself are never
+"verified": without `--judge <another model>` (or checks a person approved)
+the best a run earns is `passed own checks (<model>), not verified`, exit 3.
+Drafted checks are tried on a copy of the project before the work: one that
+already passes there, or cannot fail by construction, is sent back to the
+drafter once and dropped if the redraft is no better; one that already
+passed is still sealed as a refuse-only guard, which can fail the claim but
+never makes it verified. `--require-discriminating`
+goes further: "verified" then needs an independent value check that failed
+before the work, and anything less is `passed checks that did not test this
+work, not verified` (exit 3).
 
 ## Docs
 

bench/harbor/README.md

@@ -8,6 +8,19 @@ the share of tasks whose tests pass. It is run through
 `molt_agent.py` is the adapter. It installs this tree's CLI into each task
 container and runs `molt run --yes --criteria auto` on the instruction.
 
+The adapter passes no `--judge`, so the checks are drafted by the worker
+model, and since 2026-10-07 a run that passes only its own checks ends
+`passed own checks (<model>), not verified` (job_end `tier:
+passed-own-checks`, exit 3), never `verified`. Set `MAAT_JUDGE_MODEL` (and
+`MAAT_JUDGE_URL` / `MAAT_JUDGE_KEY`) in the container to another model to make "verified"
+reachable. Drafted checks that pass on the untouched task, or cannot fail
+by construction, are redrafted once and then dropped before they are
+sealed; the ones that only passed before the work are kept as refuse-only
+guards (they can fail a run, never verify it). With `--require-discriminating` (or `MAAT_REQUIRE_DISCRIMINATING=1`)
+a run whose independent checks never failed before the work ends
+`passed-untested`, also exit 3. The exit code never fails a trial (see
+below).
+
 ## What it measures
 
 The score of **this tree**, not of a published package. The adapter installs
@@ -140,7 +153,7 @@ In this order, because each is cheaper than the next:
 2. `molt.jsonl` ends with `proof_exhausted` — molt refused its own claim
    `attempts` times. Read the last `proof_result`: the failing criterion says
    what the model thought done meant and could not reach.
-3. `job_end` says `verified` and the test still failed — the drafted
+3. `job_end` says `verified` (or `passed-own-checks`) and the test still failed — the drafted
    criteria did not capture the task. This is the interesting case: it is the
    gap between what the model checked and what the grader checked.
 4. The log stops mid-step — the harness killed molt at the task's timeout.

bench/harbor/compare.py

@@ -30,6 +30,10 @@
 from datetime import datetime
 from pathlib import Path
 
+# Shared with bench/local/scoreboard.py (Wilson intervals, one definition).
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "local"))
+from stats import rate  # noqa: E402
+
 
 def outcome_of(trial: Path) -> str | None:
     """
@@ -215,6 +219,15 @@ def prec(h):
         return f"{100 * h['claimed_right'] / (h['claimed_right'] + h['claimed_wrong']):.0f}%"
 
     print(f"{'  done was true':16} {prec(ha):>{w}}  {prec(hb):>{w}}")
+    def pr(h, kind):
+        if h is None:
+            return "—"
+        r, wr, un = h["claimed_right"], h["claimed_wrong"], h["passed_unclaimed"]
+        return rate(r, r + wr) if kind == "p" else rate(r, r + un)
+
+    # Wilson 95% intervals: at 89 tasks a 5-task gap is noise, so say how wide.
+    print(f"{'precision':16} {pr(ha, 'p')}  |  {pr(hb, 'p')}   P(pass | said done)")
+    print(f"{'recall':16} {pr(ha, 'r')}  |  {pr(hb, 'r')}   P(said done | pass)")
     only_a, only_b = sorted(set(ta) - set(tb)), sorted(set(tb) - set(ta))
     if only_a or only_b:
         print(f"not compared — only in {na}: {len(only_a)} task(s); only in {nb}: {len(only_b)} task(s)")