Verified needs a check the worker did not write

CHANGELOG.md

@@ -4,6 +4,42 @@
 
 ### Changed
 
+- **"Verified" needs a check the worker model did not write.** Each task
+  check now records who wrote it (the worker, a separate judge, a person, the
+  reference writer). A run is `verified` only when a passing check that ran
+  the work and asserted a value came from another model (`--judge`,
+  `MAAT_JUDGE_*`, compared by provider-normalised id), or a person wrote or
+  approved a passing command check (done.yml, a mission, criteria approved in
+  the window, `--criterion`). Otherwise the tier is the new
+  **`passed-own-checks`**: outcome `unverified`, **exit code 3**.
+  **This changes headless `--criteria auto` without `--judge` (or with a judge
+  that is the worker model under another name): it used to exit 0 and now
+  exits 3.** CI jobs and bench scripts that read exit 0 as success should
+  pass `--judge <another model>` or approve the checks.
+- **Builtins never make a run verified.** `work-landed`, `record-intact`,
+  `claims-grounded`, `work-accounted`, `spec-intact` and every other builtin
+  or `session` check say the turn was well-formed, not that the task is
+  right. A bar of builtins alone now ends `passed-checks` (unverified, exit 3)
+  where it used to end `verified`, and receipts say these checks are Maat's,
+  not "a person (your check)".
+- **An expected file counts only if it predates the work.** `diff out.txt
+  expected.txt` (or a golden/want/baseline/answer file) asserts a value only
+  when that file was in the project before the first step and is unchanged
+  at the claim. A worker that writes both files no longer earns "verified".
+- **The reference check runs the deliverable out of process.** Maat's driver
+  used to import an import-style deliverable into its own process, so a
+  module that called `os._exit(0)` ended the check with exit 0 before any
+  comparison, and it passed. Each input now runs in a fresh child; a child
+  that ends without a result is a mismatch, `reference()` is removed from the
+  module the work's code can reach, and expected values never leave the
+  driver's memory. A pass also needs the driver's final `REFERENCE COMPARED
+  n/n` line: an exit 0 without it is a failure.
+- **Claims say who stood behind them** on the terminal, the window, job_end's
+  `claim`, the receipt index and the receipt: `verified (independent checks:
+  <judge>)`, `verified (your checks)`, `passed own checks (<worker>), not
+  verified`. With `--review-advisory`, a review that contradicted the task or
+  did not run appends `, unconfirmed` / `, unreviewed`.
+
 - **OpenCode runs OpenCode Zen models only.** The OpenCode backend (worker and
   judge) accepts `opencode/<model>` (e.g. `opencode/big-pickle`) and refuses
   any other provider at parse time and at run time; the CLI is handed a config

README.md

@@ -200,7 +200,11 @@ molt acp                                    # serve ACP on stdio, for an editor'
 
 Exit codes: `0` the bar was met · `1` it was not · `2` usage · `3` finished
 without a verdict (no bar, or `--only`/`--skip` left required checks unrun, which
-is never reported as a pass).
+is never reported as a pass), or passed without earning "verified": every check
+that could prove the work was written by the worker model itself
+(`passed-own-checks`), or nothing but builtins passed. "Verified" needs a
+passing check from another model (`--judge`) or a command check you wrote or
+approved; builtins never carry it.
 
 ## The bar
 
@@ -401,9 +405,11 @@ loop, and the judge is the commands. `docs/missions.md` has the whole thing.
 
 `bench/harbor/molt_agent.py` runs molt under harbor on Terminal-Bench 2.0,
 installing this tree's CLI into each task container. `bench/harbor/README.md`
-has the recipe. Headless runs get `--criteria auto`: the model drafts
-acceptance checks for the task before the work, they are sealed, and it is
-held to them.
+has the recipe. Headless runs get `--criteria auto`: acceptance checks for
+the task are drafted before the work, they are sealed, and the worker is held
+to them. Passing checks the worker model drafted for itself are never
+"verified": without `--judge <another model>` (or checks a person approved)
+the best a run earns is `passed own checks (<model>), not verified`, exit 3.
 
 ## Docs
 

bench/harbor/README.md

@@ -8,6 +8,13 @@ the share of tasks whose tests pass. It is run through
 `molt_agent.py` is the adapter. It installs this tree's CLI into each task
 container and runs `molt run --yes --criteria auto` on the instruction.
 
+The adapter passes no `--judge`, so the checks are drafted by the worker
+model, and since 2026-10-07 a run that passes only its own checks ends
+`passed own checks (<model>), not verified` (job_end `tier:
+passed-own-checks`, exit 3), never `verified`. Set `MAAT_JUDGE_MODEL` (and
+`MAAT_JUDGE_URL` / `MAAT_JUDGE_KEY`) in the container to another model to make "verified"
+reachable. The exit code never fails a trial (see below).
+
 ## What it measures
 
 The score of **this tree**, not of a published package. The adapter installs
@@ -140,7 +147,7 @@ In this order, because each is cheaper than the next:
 2. `molt.jsonl` ends with `proof_exhausted` — molt refused its own claim
    `attempts` times. Read the last `proof_result`: the failing criterion says
    what the model thought done meant and could not reach.
-3. `job_end` says `verified` and the test still failed — the drafted
+3. `job_end` says `verified` (or `passed-own-checks`) and the test still failed — the drafted
    criteria did not capture the task. This is the interesting case: it is the
    gap between what the model checked and what the grader checked.
 4. The log stops mid-step — the harness killed molt at the task's timeout.

bench/harbor/summarize.py

@@ -82,7 +82,9 @@ def trial_row(d: Path) -> dict | None:
         "molt_exit": exit_code,
         "molt_outcome": (
             "unconfirmed" if (je or {}).get("outcome") == "verified" and ((je or {}).get("review") or {}).get("confirmed") is False
-            else "self-checked" if (je or {}).get("outcome") == "verified" and (je or {}).get("selfChecked")
+            # Builds from 2026-10-07 say who wrote the checks in job_end's `claim`.
+            else "passed-own-checks" if (je or {}).get("tier") == "passed-own-checks"
+            else "self-checked" if (je or {}).get("outcome") == "verified" and (je or {}).get("selfChecked") and not (je or {}).get("claim")
             else (je or {}).get("outcome")
         ),
         "molt_steps": (je or {}).get("steps"),

bench/local/run.py

@@ -92,6 +92,26 @@ def provider_capped(out: str, steps: int) -> bool:
     )
 
 
+def claim_of(ev: dict) -> str | None:
+    """
+    The `claim` field for one job_end event.
+
+    Builds from 2026-10-07 on carry job_end's own `claim`, which says who stood
+    behind the word: "verified (independent checks: <judge>)", "verified (your
+    checks)", or "passed own checks (<worker>), not verified" (never counted as
+    verified: it does not start with "verified"). Older builds, and outcomes
+    that are not a tier, carry the outcome as before, with " (self-checked)"
+    when every check was drafted (whoever drafted it).
+    """
+    c = ev.get("claim")
+    if isinstance(c, str) and c:
+        return c
+    o = ev.get("outcome")
+    if not isinstance(o, str):
+        return None
+    return o + (" (self-checked)" if ev.get("selfChecked") else "")
+
+
 def run_molt(d: Path, prompt: str, log: Path) -> dict:
     key = openrouter_key() if "openrouter.ai" in URL else "local"
     env = os.environ | {"MOLT_API_KEY": key, "MOLT_JUDGMENT": "0"}  # nobody rules on a benchmark run
@@ -136,12 +156,12 @@ def run_molt(d: Path, prompt: str, log: Path) -> dict:
             steps += 1
             per.append(len(ev.get("tools") or []))
         elif ev.get("kind") == "job_end":
-            outcome = ev.get("outcome") + (" (self-checked)" if ev.get("selfChecked") else "")
+            outcome = claim_of(ev)
             spend = ev.get("spend") or {}
             review = ev.get("review")
             disagree = ev.get("checksDisagree") or []
             # Only what job_end carried; absent keys stay absent (older builds).
-            extra = {k: ev[k] for k in ("revealed", "deadline", "endedBy", "retired", "build", "tier", "tierReason", "providerStall") if k in ev}
+            extra = {k: ev[k] for k in ("revealed", "deadline", "endedBy", "retired", "build", "tier", "tierReason", "providerStall", "checkAuthors") if k in ev}
     # The provider's daily cap, not the work: every later task would fail the
     # same way (2026-10-05: eleven tasks per arm "failed" in 140 s, 0 turns).
     capped = provider_capped(out, steps)

bench/local/stats.py

@@ -60,7 +60,44 @@ def no_verdict(row: dict) -> bool:
     return (not c) or c.startswith(("error", "stopped")) or bool(row.get("timed_out"))
 
 
-TIERS = ("verified", "passed-checks")
+TIERS = ("verified", "passed-checks", "passed-own-checks")
+
+
+def claim_basis(claim) -> str | None:
+    """
+    Who stood behind a claim, read from run.py's `claim` text, old or new.
+
+      "verified (independent checks: m)"            -> "independent"
+      "verified (your checks)"                      -> "person"
+      "passed own checks (m), not verified"         -> "own"
+      "verified (self-checked)"  (builds before 2026-10-07: the checks were
+                                  drafted, by the worker or a judge; the
+                                  text cannot say which)  -> "self-checked"
+      "verified"                 (old: no drafted-only flag; authorship
+                                  was never recorded)  -> "unrecorded"
+      anything else                                  -> None
+    """
+    if not isinstance(claim, str):
+        return None
+    if claim.startswith("verified (independent checks"):
+        return "independent"
+    if claim.startswith("verified (your checks)"):
+        return "person"
+    if claim.startswith("passed own checks"):
+        return "own"
+    if claim.startswith("verified (self-checked)"):
+        return "self-checked"
+    if claim.startswith("verified"):
+        return "unrecorded"
+    return None
+
+
+def claim_judge(claim) -> list[str]:
+    """The judge model(s) named by "verified (independent checks: a, b)"; [] otherwise."""
+    if not isinstance(claim, str) or not claim.startswith("verified (independent checks:"):
+        return []
+    inner = claim[len("verified (independent checks:"):].rsplit(")", 1)[0]
+    return [m.strip() for m in inner.split(",") if m.strip()]
 
 
 def tier_breakdown(rows: list[dict]) -> dict[str, tuple[int, int]]:

bench/local/tests/test_claim_labels.py

@@ -0,0 +1,55 @@
+"""run.py's `claim` field says who stood behind each "verified", and old text still parses."""
+import sys
+from pathlib import Path
+
+LOCAL = Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(LOCAL))
+import run  # noqa: E402
+import stats  # noqa: E402
+
+IND = "verified (independent checks: minimax/minimax-m3)"
+YOURS = "verified (your checks)"
+OWN = "passed own checks (qwen/qwen3-235b-a22b), not verified"
+
+
+def test_claim_of_carries_the_new_labels():
+    assert run.claim_of({"kind": "job_end", "outcome": "verified", "tier": "verified", "claim": IND, "selfChecked": True}) == IND
+    assert run.claim_of({"kind": "job_end", "outcome": "verified", "tier": "verified", "claim": YOURS}) == YOURS
+    assert run.claim_of({"kind": "job_end", "outcome": "unverified", "tier": "passed-own-checks", "claim": OWN, "selfChecked": True}) == OWN
+
+
+def test_claim_of_keeps_the_old_text_for_older_builds_and_other_outcomes():
+    assert run.claim_of({"outcome": "verified", "selfChecked": True}) == "verified (self-checked)"
+    assert run.claim_of({"outcome": "not proven", "selfChecked": True}) == "not proven (self-checked)"
+    assert run.claim_of({"outcome": "verified"}) == "verified"
+    assert run.claim_of({"outcome": "unverified", "tier": "passed-checks"}) == "unverified"
+    assert run.claim_of({}) is None
+
+
+def test_only_independent_or_person_claims_count_as_verified():
+    assert stats.is_verified(IND)
+    assert stats.is_verified(YOURS)
+    assert not stats.is_verified(OWN)
+    assert stats.is_verified("verified (self-checked)"), "old rows read as before"
+
+
+def test_claim_basis_tells_them_apart_old_and_new():
+    assert stats.claim_basis(IND) == "independent"
+    assert stats.claim_basis(YOURS) == "person"
+    assert stats.claim_basis(OWN) == "own"
+    assert stats.claim_basis("verified (self-checked)") == "self-checked"
+    assert stats.claim_basis("verified") == "unrecorded"
+    assert stats.claim_basis("not proven (self-checked)") is None
+    assert stats.claim_basis(None) is None
+
+
+def test_claim_judge_names_the_judges():
+    assert stats.claim_judge(IND) == ["minimax/minimax-m3"]
+    assert stats.claim_judge("verified (independent checks: a, b)") == ["a", "b"]
+    assert stats.claim_judge(OWN) == []
+
+
+def test_passed_own_checks_is_a_tier_of_its_own():
+    assert "passed-own-checks" in stats.TIERS
+    rows = [{"tier": "passed-own-checks", "passed": True}, {"tier": "verified", "passed": False}]
+    assert stats.tier_breakdown(rows) == {"passed-own-checks": (1, 1), "verified": (1, 0)}

src/bar.ts

@@ -10,6 +10,7 @@
  */
 import { DISPUTE_HINT } from "./dispute.js";
 import { judgePass } from "./evidence.js";
+import { REFERENCE_SENTINEL, referenceComparedAll } from "./reference.js";
 import { testsRealFor } from "./tests-real.js";
 import { runCommand, draftedShell } from "./run.js";
 import { parseLcov, coverageFor, coverageCouldSpeak, unprovenIn, type Unproven } from "./coverage.js";
@@ -2227,6 +2228,15 @@ export async function runCheck(check: Check, ctx: BarContext): Promise<CheckResu
       if (judged.broken) diagnosis = { didNotRun: true, hint: judged.why };
       output = `[molt] ${judged.why}\n\n${output}`;
     }
+    // The reference driver (src/reference.ts) ends by saying it compared
+    // every input it judged. An exit 0 without that line is something ending
+    // the driver early (the work's code, once, with os._exit(0)), not a pass.
+    if (passed && check.tags.includes("reference") && !referenceComparedAll(output)) {
+      passed = false;
+      output =
+        "[molt] the reference check exited 0 without reporting that it compared every input " +
+        `(no \`${REFERENCE_SENTINEL} n/n\` line), so nothing was shown to match\n\n${output}`;
+    }
   }
   return {
     name: check.name,

src/cli.tsx

@@ -27,7 +27,7 @@ import { buildRepoMap, DEFAULT_MAP_TOKENS } from "./repomap.js";
 import { buildBrief, DEFAULT_BRIEF_TOKENS } from "./brief.js";
 import { draftMission, missionStatus, runMission, writePlan, type MissionSummary } from "./mission.js";
 import { parseDuration } from "./session-commands.js";
-import { passedChecksWords } from "./tiers.js";
+import { jobEndWords } from "./tiers.js";
 import { judgeEffort, judgeTarget } from "./judge.js";
 import { commandsHere, draftCriteriaCritiqued, drafterInputsHash, drafterSnapshot, preflightCriteria, taskChecksFrom, type Draft, type DrafterInputs } from "./criteria.js";
 import { listProject, removeNew } from "./leftovers.js";
@@ -49,7 +49,7 @@ import {
   type StoredEndpoint,
 } from "./providers.js";
 import { Receipts } from "./receipts.js";
-import type { BarResult, Check, EngineEvent } from "./types.js";
+import type { BarResult, Check, CheckAuthor, EngineEvent } from "./types.js";
 import { stateDir } from "./statedir.js";
 import { draftReference, snapshotProject } from "./reference.js";
 import { env } from "./env.js";
@@ -178,6 +178,9 @@ options
                      A separate judge does not share the worker's misreadings;
                      it raised verified-on-correct-work on every worker tested
                      with no wrong verifieds. Same as MAAT_JUDGE_MODEL.
+                     Without one (or with the worker's own model), drafted
+                     checks are the worker's and the most a run earns is
+                     "passed own checks", never "verified".
   --judge-url <url>  where the judge runs, e.g. grok-build://subscription,
                      opencode://zen (OpenCode Zen models only, e.g.
                      opencode/big-pickle), or an OpenAI-style URL.
@@ -1327,7 +1330,11 @@ function startReference(args: Args, deadlineAt?: number): Promise<{ check: Check
 async function sealDraft(draft: Draft, args: Args, late = false): Promise<ReturnType<typeof taskChecksFrom>> {
   // Hidden: the model wrote these, and a model shown its own exam makes the
   // work equal the check. It gets the names, and the output on failure.
-  const sealed = taskChecksFrom(draft, { hidden: true });
+  // Who wrote them, recorded with the seal: the judge when one is set, else the
+  // worker itself. Whether that judge is really another model is tierOf's call.
+  const drafter = judgeTarget({ baseUrl: args.url, apiKey: args.key, model: args.model }).model;
+  const author: CheckAuthor = process.env.MAAT_JUDGE_MODEL?.trim() ? { kind: "judge", model: drafter } : { kind: "worker", model: args.model };
+  const sealed = taskChecksFrom(draft, { hidden: true, author });
   // Headless, the checks' own side effects are cleaned up (src/leftovers.ts).
   // A draft that joins at a claim (RunOptions.pendingCriteria, cut with no
   // checks ready) lands after the work began: trying it in the live folder
@@ -1441,22 +1448,7 @@ async function cmdRun(args: Args, ask = false): Promise<number> {
         // the same books.
         const sp = ev.spend;
         const cached = sp.cachedTokens > 0 ? ` (${sp.cachedTokens} cached)` : "";
-        const said =
-          ev.tier === "passed-checks"
-            ? passedChecksWords(ev.tierReason)
-            : ev.outcome === "verified" && ev.revealed?.length
-            ? "verified (checks shown after a repeat failure)"
-            : ev.outcome === "verified" && ev.review && !ev.review.confirmed
-            ? "passed its checks, unconfirmed"
-            : ev.outcome === "verified" && ev.review?.confirmed
-              ? "verified, independently reviewed"
-              : ev.outcome === "verified" && ev.unreviewed
-                ? "passed its checks, unreviewed"
-              : ev.outcome === "verified" && ev.selfChecked
-                ? "passed its own checks"
-                : ev.outcome === "unverified" && ev.checksDisagree?.length
-                  ? "unverified, its own drafted checks disagree"
-                  : ev.outcome;
+        const said = jobEndWords(ev);
         process.stdout.write(
           `· job ${said} · ${ev.steps} step(s) · ${sp.promptTokens} in${cached} · ` +
             `${sp.completionTokens} out · ${fmtDuration(ev.durationMs)}` +

src/criteria.ts

@@ -36,6 +36,7 @@ import { copyTree } from "./scratch.js";
 import { diagnoseFailure } from "./bar.js";
 import { normalizeRequirements } from "./signout.js";
 import { evidenceTags } from "./tiers.js";
+import type { CheckAuthor } from "./types.js";
 
 /**
  * `surface`: the critic read it as only looking (a file exists, a word
@@ -116,7 +117,15 @@ export function sanitizeCriteria(raw: unknown): Draft {
  */
 export function taskChecksFrom(
   raw: unknown,
-  opts: { hidden?: boolean } = {},
+  opts: {
+    hidden?: boolean;
+    /**
+     * Who wrote these checks. Defaults to a person when they are not hidden
+     * (a person approved them in the window or wrote them on the command
+     * line); hidden checks without one count as the worker's (tiers.ts).
+     */
+    author?: CheckAuthor;
+  } = {},
 ): {
   taskChecks: {
     name: string;
@@ -126,6 +135,7 @@ export function taskChecksFrom(
     expectExit: number;
     tags: string[];
     hidden?: boolean;
+    author?: CheckAuthor;
   }[];
   taskNotes: string[];
   requirements?: string[];
@@ -141,6 +151,7 @@ export function taskChecksFrom(
       // surface from the critic, value from the command itself (evidence.ts).
       tags: evidenceTags(c.run, c.surface),
       ...(opts.hidden ? { hidden: true } : {}),
+      ...(opts.author ? { author: { ...opts.author } } : opts.hidden ? {} : { author: { kind: "person" as const } }),
     })),
     taskNotes: drafted.notes,
     ...(drafted.requirements ? { requirements: drafted.requirements } : {}),