diff --git a/CHANGELOG.md b/CHANGELOG.md index 6ba80e1..431384e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,42 @@ ### Changed +- **"Verified" needs a check the worker model did not write.** Each task + check now records who wrote it (the worker, a separate judge, a person, the + reference writer). A run is `verified` only when a passing check that ran + the work and asserted a value came from another model (`--judge`, + `MAAT_JUDGE_*`, compared by provider-normalised id), or a person wrote or + approved a passing command check (done.yml, a mission, criteria approved in + the window, `--criterion`). Otherwise the tier is the new + **`passed-own-checks`**: outcome `unverified`, **exit code 3**. + **This changes headless `--criteria auto` without `--judge` (or with a judge + that is the worker model under another name): it used to exit 0 and now + exits 3.** CI jobs and bench scripts that read exit 0 as success should + pass `--judge ` or approve the checks. +- **Builtins never make a run verified.** `work-landed`, `record-intact`, + `claims-grounded`, `work-accounted`, `spec-intact` and every other builtin + or `session` check say the turn was well-formed, not that the task is + right. A bar of builtins alone now ends `passed-checks` (unverified, exit 3) + where it used to end `verified`, and receipts say these checks are Maat's, + not "a person (your check)". +- **An expected file counts only if it predates the work.** `diff out.txt + expected.txt` (or a golden/want/baseline/answer file) asserts a value only + when that file was in the project before the first step and is unchanged + at the claim. A worker that writes both files no longer earns "verified". +- **The reference check runs the deliverable out of process.** Maat's driver + used to import an import-style deliverable into its own process, so a + module that called `os._exit(0)` ended the check with exit 0 before any + comparison, and it passed. Each input now runs in a fresh child; a child + that ends without a result is a mismatch, `reference()` is removed from the + module the work's code can reach, and expected values never leave the + driver's memory. A pass also needs the driver's final `REFERENCE COMPARED + n/n` line: an exit 0 without it is a failure. +- **Claims say who stood behind them** on the terminal, the window, job_end's + `claim`, the receipt index and the receipt: `verified (independent checks: + )`, `verified (your checks)`, `passed own checks (), not + verified`. With `--review-advisory`, a review that contradicted the task or + did not run appends `, unconfirmed` / `, unreviewed`. + - **OpenCode runs OpenCode Zen models only.** The OpenCode backend (worker and judge) accepts `opencode/` (e.g. `opencode/big-pickle`) and refuses any other provider at parse time and at run time; the CLI is handed a config diff --git a/README.md b/README.md index 22be206..46f1d4f 100644 --- a/README.md +++ b/README.md @@ -200,7 +200,11 @@ molt acp # serve ACP on stdio, for an editor' Exit codes: `0` the bar was met · `1` it was not · `2` usage · `3` finished without a verdict (no bar, or `--only`/`--skip` left required checks unrun, which -is never reported as a pass). +is never reported as a pass), or passed without earning "verified": every check +that could prove the work was written by the worker model itself +(`passed-own-checks`), or nothing but builtins passed. "Verified" needs a +passing check from another model (`--judge`) or a command check you wrote or +approved; builtins never carry it. ## The bar @@ -401,9 +405,11 @@ loop, and the judge is the commands. `docs/missions.md` has the whole thing. `bench/harbor/molt_agent.py` runs molt under harbor on Terminal-Bench 2.0, installing this tree's CLI into each task container. `bench/harbor/README.md` -has the recipe. Headless runs get `--criteria auto`: the model drafts -acceptance checks for the task before the work, they are sealed, and it is -held to them. +has the recipe. Headless runs get `--criteria auto`: acceptance checks for +the task are drafted before the work, they are sealed, and the worker is held +to them. Passing checks the worker model drafted for itself are never +"verified": without `--judge ` (or checks a person approved) +the best a run earns is `passed own checks (), not verified`, exit 3. ## Docs diff --git a/bench/harbor/README.md b/bench/harbor/README.md index 1cd8e8b..915a5e7 100644 --- a/bench/harbor/README.md +++ b/bench/harbor/README.md @@ -8,6 +8,13 @@ the share of tasks whose tests pass. It is run through `molt_agent.py` is the adapter. It installs this tree's CLI into each task container and runs `molt run --yes --criteria auto` on the instruction. +The adapter passes no `--judge`, so the checks are drafted by the worker +model, and since 2026-10-07 a run that passes only its own checks ends +`passed own checks (), not verified` (job_end `tier: +passed-own-checks`, exit 3), never `verified`. Set `MAAT_JUDGE_MODEL` (and +`MAAT_JUDGE_URL` / `MAAT_JUDGE_KEY`) in the container to another model to make "verified" +reachable. The exit code never fails a trial (see below). + ## What it measures The score of **this tree**, not of a published package. The adapter installs @@ -140,7 +147,7 @@ In this order, because each is cheaper than the next: 2. `molt.jsonl` ends with `proof_exhausted` — molt refused its own claim `attempts` times. Read the last `proof_result`: the failing criterion says what the model thought done meant and could not reach. -3. `job_end` says `verified` and the test still failed — the drafted +3. `job_end` says `verified` (or `passed-own-checks`) and the test still failed — the drafted criteria did not capture the task. This is the interesting case: it is the gap between what the model checked and what the grader checked. 4. The log stops mid-step — the harness killed molt at the task's timeout. diff --git a/bench/harbor/summarize.py b/bench/harbor/summarize.py index f0d3e4d..3999c1c 100755 --- a/bench/harbor/summarize.py +++ b/bench/harbor/summarize.py @@ -82,7 +82,9 @@ def trial_row(d: Path) -> dict | None: "molt_exit": exit_code, "molt_outcome": ( "unconfirmed" if (je or {}).get("outcome") == "verified" and ((je or {}).get("review") or {}).get("confirmed") is False - else "self-checked" if (je or {}).get("outcome") == "verified" and (je or {}).get("selfChecked") + # Builds from 2026-10-07 say who wrote the checks in job_end's `claim`. + else "passed-own-checks" if (je or {}).get("tier") == "passed-own-checks" + else "self-checked" if (je or {}).get("outcome") == "verified" and (je or {}).get("selfChecked") and not (je or {}).get("claim") else (je or {}).get("outcome") ), "molt_steps": (je or {}).get("steps"), diff --git a/bench/local/run.py b/bench/local/run.py index a3e5e1b..8be41cd 100644 --- a/bench/local/run.py +++ b/bench/local/run.py @@ -92,6 +92,26 @@ def provider_capped(out: str, steps: int) -> bool: ) +def claim_of(ev: dict) -> str | None: + """ + The `claim` field for one job_end event. + + Builds from 2026-10-07 on carry job_end's own `claim`, which says who stood + behind the word: "verified (independent checks: )", "verified (your + checks)", or "passed own checks (), not verified" (never counted as + verified: it does not start with "verified"). Older builds, and outcomes + that are not a tier, carry the outcome as before, with " (self-checked)" + when every check was drafted (whoever drafted it). + """ + c = ev.get("claim") + if isinstance(c, str) and c: + return c + o = ev.get("outcome") + if not isinstance(o, str): + return None + return o + (" (self-checked)" if ev.get("selfChecked") else "") + + def run_molt(d: Path, prompt: str, log: Path) -> dict: key = openrouter_key() if "openrouter.ai" in URL else "local" env = os.environ | {"MOLT_API_KEY": key, "MOLT_JUDGMENT": "0"} # nobody rules on a benchmark run @@ -136,12 +156,12 @@ def run_molt(d: Path, prompt: str, log: Path) -> dict: steps += 1 per.append(len(ev.get("tools") or [])) elif ev.get("kind") == "job_end": - outcome = ev.get("outcome") + (" (self-checked)" if ev.get("selfChecked") else "") + outcome = claim_of(ev) spend = ev.get("spend") or {} review = ev.get("review") disagree = ev.get("checksDisagree") or [] # Only what job_end carried; absent keys stay absent (older builds). - extra = {k: ev[k] for k in ("revealed", "deadline", "endedBy", "retired", "build", "tier", "tierReason", "providerStall") if k in ev} + extra = {k: ev[k] for k in ("revealed", "deadline", "endedBy", "retired", "build", "tier", "tierReason", "providerStall", "checkAuthors") if k in ev} # The provider's daily cap, not the work: every later task would fail the # same way (2026-10-05: eleven tasks per arm "failed" in 140 s, 0 turns). capped = provider_capped(out, steps) diff --git a/bench/local/stats.py b/bench/local/stats.py index 124869c..2969f9f 100644 --- a/bench/local/stats.py +++ b/bench/local/stats.py @@ -60,7 +60,44 @@ def no_verdict(row: dict) -> bool: return (not c) or c.startswith(("error", "stopped")) or bool(row.get("timed_out")) -TIERS = ("verified", "passed-checks") +TIERS = ("verified", "passed-checks", "passed-own-checks") + + +def claim_basis(claim) -> str | None: + """ + Who stood behind a claim, read from run.py's `claim` text, old or new. + + "verified (independent checks: m)" -> "independent" + "verified (your checks)" -> "person" + "passed own checks (m), not verified" -> "own" + "verified (self-checked)" (builds before 2026-10-07: the checks were + drafted, by the worker or a judge; the + text cannot say which) -> "self-checked" + "verified" (old: no drafted-only flag; authorship + was never recorded) -> "unrecorded" + anything else -> None + """ + if not isinstance(claim, str): + return None + if claim.startswith("verified (independent checks"): + return "independent" + if claim.startswith("verified (your checks)"): + return "person" + if claim.startswith("passed own checks"): + return "own" + if claim.startswith("verified (self-checked)"): + return "self-checked" + if claim.startswith("verified"): + return "unrecorded" + return None + + +def claim_judge(claim) -> list[str]: + """The judge model(s) named by "verified (independent checks: a, b)"; [] otherwise.""" + if not isinstance(claim, str) or not claim.startswith("verified (independent checks:"): + return [] + inner = claim[len("verified (independent checks:"):].rsplit(")", 1)[0] + return [m.strip() for m in inner.split(",") if m.strip()] def tier_breakdown(rows: list[dict]) -> dict[str, tuple[int, int]]: diff --git a/bench/local/tests/test_claim_labels.py b/bench/local/tests/test_claim_labels.py new file mode 100644 index 0000000..9c0624e --- /dev/null +++ b/bench/local/tests/test_claim_labels.py @@ -0,0 +1,55 @@ +"""run.py's `claim` field says who stood behind each "verified", and old text still parses.""" +import sys +from pathlib import Path + +LOCAL = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(LOCAL)) +import run # noqa: E402 +import stats # noqa: E402 + +IND = "verified (independent checks: minimax/minimax-m3)" +YOURS = "verified (your checks)" +OWN = "passed own checks (qwen/qwen3-235b-a22b), not verified" + + +def test_claim_of_carries_the_new_labels(): + assert run.claim_of({"kind": "job_end", "outcome": "verified", "tier": "verified", "claim": IND, "selfChecked": True}) == IND + assert run.claim_of({"kind": "job_end", "outcome": "verified", "tier": "verified", "claim": YOURS}) == YOURS + assert run.claim_of({"kind": "job_end", "outcome": "unverified", "tier": "passed-own-checks", "claim": OWN, "selfChecked": True}) == OWN + + +def test_claim_of_keeps_the_old_text_for_older_builds_and_other_outcomes(): + assert run.claim_of({"outcome": "verified", "selfChecked": True}) == "verified (self-checked)" + assert run.claim_of({"outcome": "not proven", "selfChecked": True}) == "not proven (self-checked)" + assert run.claim_of({"outcome": "verified"}) == "verified" + assert run.claim_of({"outcome": "unverified", "tier": "passed-checks"}) == "unverified" + assert run.claim_of({}) is None + + +def test_only_independent_or_person_claims_count_as_verified(): + assert stats.is_verified(IND) + assert stats.is_verified(YOURS) + assert not stats.is_verified(OWN) + assert stats.is_verified("verified (self-checked)"), "old rows read as before" + + +def test_claim_basis_tells_them_apart_old_and_new(): + assert stats.claim_basis(IND) == "independent" + assert stats.claim_basis(YOURS) == "person" + assert stats.claim_basis(OWN) == "own" + assert stats.claim_basis("verified (self-checked)") == "self-checked" + assert stats.claim_basis("verified") == "unrecorded" + assert stats.claim_basis("not proven (self-checked)") is None + assert stats.claim_basis(None) is None + + +def test_claim_judge_names_the_judges(): + assert stats.claim_judge(IND) == ["minimax/minimax-m3"] + assert stats.claim_judge("verified (independent checks: a, b)") == ["a", "b"] + assert stats.claim_judge(OWN) == [] + + +def test_passed_own_checks_is_a_tier_of_its_own(): + assert "passed-own-checks" in stats.TIERS + rows = [{"tier": "passed-own-checks", "passed": True}, {"tier": "verified", "passed": False}] + assert stats.tier_breakdown(rows) == {"passed-own-checks": (1, 1), "verified": (1, 0)} diff --git a/src/bar.ts b/src/bar.ts index 8425eb1..8291e4a 100644 --- a/src/bar.ts +++ b/src/bar.ts @@ -10,6 +10,7 @@ */ import { DISPUTE_HINT } from "./dispute.js"; import { judgePass } from "./evidence.js"; +import { REFERENCE_SENTINEL, referenceComparedAll } from "./reference.js"; import { testsRealFor } from "./tests-real.js"; import { runCommand, draftedShell } from "./run.js"; import { parseLcov, coverageFor, coverageCouldSpeak, unprovenIn, type Unproven } from "./coverage.js"; @@ -2227,6 +2228,15 @@ export async function runCheck(check: Check, ctx: BarContext): Promise where the judge runs, e.g. grok-build://subscription, opencode://zen (OpenCode Zen models only, e.g. opencode/big-pickle), or an OpenAI-style URL. @@ -1327,7 +1330,11 @@ function startReference(args: Args, deadlineAt?: number): Promise<{ check: Check async function sealDraft(draft: Draft, args: Args, late = false): Promise> { // Hidden: the model wrote these, and a model shown its own exam makes the // work equal the check. It gets the names, and the output on failure. - const sealed = taskChecksFrom(draft, { hidden: true }); + // Who wrote them, recorded with the seal: the judge when one is set, else the + // worker itself. Whether that judge is really another model is tierOf's call. + const drafter = judgeTarget({ baseUrl: args.url, apiKey: args.key, model: args.model }).model; + const author: CheckAuthor = process.env.MAAT_JUDGE_MODEL?.trim() ? { kind: "judge", model: drafter } : { kind: "worker", model: args.model }; + const sealed = taskChecksFrom(draft, { hidden: true, author }); // Headless, the checks' own side effects are cleaned up (src/leftovers.ts). // A draft that joins at a claim (RunOptions.pendingCriteria, cut with no // checks ready) lands after the work began: trying it in the live folder @@ -1441,22 +1448,7 @@ async function cmdRun(args: Args, ask = false): Promise { // the same books. const sp = ev.spend; const cached = sp.cachedTokens > 0 ? ` (${sp.cachedTokens} cached)` : ""; - const said = - ev.tier === "passed-checks" - ? passedChecksWords(ev.tierReason) - : ev.outcome === "verified" && ev.revealed?.length - ? "verified (checks shown after a repeat failure)" - : ev.outcome === "verified" && ev.review && !ev.review.confirmed - ? "passed its checks, unconfirmed" - : ev.outcome === "verified" && ev.review?.confirmed - ? "verified, independently reviewed" - : ev.outcome === "verified" && ev.unreviewed - ? "passed its checks, unreviewed" - : ev.outcome === "verified" && ev.selfChecked - ? "passed its own checks" - : ev.outcome === "unverified" && ev.checksDisagree?.length - ? "unverified, its own drafted checks disagree" - : ev.outcome; + const said = jobEndWords(ev); process.stdout.write( `· job ${said} · ${ev.steps} step(s) · ${sp.promptTokens} in${cached} · ` + `${sp.completionTokens} out · ${fmtDuration(ev.durationMs)}` + diff --git a/src/criteria.ts b/src/criteria.ts index 85e8083..d352f66 100644 --- a/src/criteria.ts +++ b/src/criteria.ts @@ -36,6 +36,7 @@ import { copyTree } from "./scratch.js"; import { diagnoseFailure } from "./bar.js"; import { normalizeRequirements } from "./signout.js"; import { evidenceTags } from "./tiers.js"; +import type { CheckAuthor } from "./types.js"; /** * `surface`: the critic read it as only looking (a file exists, a word @@ -116,7 +117,15 @@ export function sanitizeCriteria(raw: unknown): Draft { */ export function taskChecksFrom( raw: unknown, - opts: { hidden?: boolean } = {}, + opts: { + hidden?: boolean; + /** + * Who wrote these checks. Defaults to a person when they are not hidden + * (a person approved them in the window or wrote them on the command + * line); hidden checks without one count as the worker's (tiers.ts). + */ + author?: CheckAuthor; + } = {}, ): { taskChecks: { name: string; @@ -126,6 +135,7 @@ export function taskChecksFrom( expectExit: number; tags: string[]; hidden?: boolean; + author?: CheckAuthor; }[]; taskNotes: string[]; requirements?: string[]; @@ -141,6 +151,7 @@ export function taskChecksFrom( // surface from the critic, value from the command itself (evidence.ts). tags: evidenceTags(c.run, c.surface), ...(opts.hidden ? { hidden: true } : {}), + ...(opts.author ? { author: { ...opts.author } } : opts.hidden ? {} : { author: { kind: "person" as const } }), })), taskNotes: drafted.notes, ...(drafted.requirements ? { requirements: drafted.requirements } : {}), diff --git a/src/engine.ts b/src/engine.ts index aa34918..92a1212 100644 --- a/src/engine.ts +++ b/src/engine.ts @@ -22,7 +22,8 @@ import { runCommand } from "./run.js"; import { describeStart, listBackground, startBackground, stopBackground } from "./background.js"; import { reviewClaim, type Review } from "./review.js"; import { judgeEffort, judgeTarget } from "./judge.js"; -import { tierOf } from "./tiers.js"; +import { authorKey, authorWords, claimLabel, tierOf, withAuthor } from "./tiers.js"; +import { recordGoldens, valueUnproven } from "./golden.js"; import { arbitrate, parseDisputes, type Ruling } from "./dispute.js"; import { createHash } from "node:crypto"; import { execFileSync } from "node:child_process"; @@ -120,6 +121,7 @@ import { estTokens, type Bar, type Check, + type CheckAuthor, type CheckResult, type BarResult, type Bom, @@ -2186,6 +2188,13 @@ export class Engine { * `pass (nothing to establish)` for it rather than presenting it as proof. */ private passedBeforeWork: ReadonlySet = new Set(); + /** + * The expected-looking operands of `diff`/`cmp` task checks, with their + * content as the turn found them (src/golden.ts). Recorded at the seal, + * before the first step; a golden file not in here, or changed since, + * proves no value. + */ + private goldensBefore: Map = new Map(); /** True only while `proveNow` runs: a bar with no turn behind it. */ private standalone = false; private barHash: string | null; @@ -3670,6 +3679,43 @@ export class Engine { return this.cfg.reviewAdvisory === true ? { reviewAdvisory: true, guards: this.passedBeforeWork } : {}; } + /** Who wrote each sealed check, keyed by the name it runs under in the bar (and its bare name). */ + private sealedAuthors(): Map { + const m = new Map(); + for (const c of this.sealedChecks) { + const a = withAuthor(c, this.cfg.model).author!; + m.set(c.name, a); + if (!c.name.startsWith("task:")) m.set(`task:${c.name}`, a); + } + return m; + } + + /** The worker under every name it ran as: the configured id and the one the backend reported. */ + private workerNames(): string[] { + return [...new Set([this.cfg.model, this.modelOfRecord()].filter((m): m is string => typeof m === "string" && m.trim().length > 0))]; + } + + /** Everything tierOf weighs beside the results: advisory mode, the worker, who wrote each check, and which golden files predate the work. */ + private tierContext(): { + reviewAdvisory?: true; + guards?: ReadonlySet; + worker: string[]; + authors: Map; + valueUnproven: Set; + } { + return { + ...this.advisoryTier(), + worker: this.workerNames(), + authors: this.sealedAuthors(), + valueUnproven: valueUnproven(this.sealedChecks.map((c) => ({ name: c.name, run: c.kind === "command" ? c.run : undefined, tags: c.tags })), this.cwd, this.goldensBefore), + }; + } + + /** The receipt's authorship map: sealed checks by bar name, as recorded. */ + private receiptAuthors(): Record { + return Object.fromEntries(this.sealedAuthors()); + } + /** See `passedBeforeWork`. Only a passing command criterion is relabelled. */ private markGuards(result: BarResult): BarResult { if (this.passedBeforeWork.size === 0) return result; @@ -4485,22 +4531,41 @@ export class Engine { // right 7/7 where everything else carrying the word was right 28/50. What // passed without earning it is reported as what it is. After the judgment // case above, which is opened on what the reviewers said. - let tier: "verified" | "passed-checks" | undefined; + let tier: "verified" | "passed-checks" | "passed-own-checks" | undefined; let tierReason: string | undefined; + let claim: string | undefined; if (outcome === "verified" && lastProof) { // Review was asked for and did not run (skipped near the deadline, or a // nudge cleared it and no re-review followed): no "verified". On v11, 6 // of the 7 "verified" claims were unreviewed this way and 4 were wrong. const unreviewed = this.cfg.review !== undefined && !review; - const t = tierOf({ results: lastProof.results, review, unreviewed, ...this.advisoryTier() }); + const t = tierOf({ results: lastProof.results, review, unreviewed, ...this.tierContext() }); tier = t.tier; if (t.tier === "passed-checks") { outcome = "unverified"; tierReason = t.reason; yield { kind: "info", text: `passed its checks (not verified: ${t.reason}).` }; + } else if (t.tier === "passed-own-checks") { + // The model that did the work wrote every check that could have + // proved it: a model never both finds and judges. Not verified, and + // the exit code is unverified's. + outcome = "unverified"; + tierReason = t.reason; + claim = claimLabel(outcome, t); + yield { kind: "info", text: `${claim}: ${t.reason}. Use --judge , or approve the checks yourself, for "verified".` }; + } else { + claim = claimLabel(outcome, t); } - if (receiptPath) this.cfg.receipts?.amendTier(basename(receiptPath), t); - this.cfg.journal?.append("note", { text: `tier: ${t.tier}`, evidence: t.evidence, ...(t.reason ? { reason: t.reason } : {}), ...(t.reviewNote ? { review: t.reviewNote } : {}) }); + if (receiptPath) this.cfg.receipts?.amendTier(basename(receiptPath), { ...t, ...(claim ? { claim } : {}) }); + this.cfg.journal?.append("note", { + text: `tier: ${t.tier}`, + evidence: t.evidence, + ...(t.basis ? { basis: t.basis } : {}), + ...(t.by?.length ? { by: t.by } : {}), + ...(claim ? { claim } : {}), + ...(t.reason ? { reason: t.reason } : {}), + ...(t.reviewNote ? { review: t.reviewNote } : {}), + }); } yield { kind: "job_end", @@ -4510,6 +4575,8 @@ export class Engine { durationMs: Date.now() - startedAt, outcome, ...(tier ? { tier, ...(tierReason ? { tierReason } : {}) } : {}), + ...(claim ? { claim } : {}), + ...(this.sealedChecks.length ? { checkAuthors: Object.fromEntries(this.sealedChecks.map((c) => [c.name.startsWith("task:") ? c.name : `task:${c.name}`, authorKey(withAuthor(c, this.cfg.model).author!)])) } : {}), ...(selfChecked ? { selfChecked: true } : {}), ...(this.reviewSkipped ? { unreviewed: true } : {}), ...(this.turnEndedBy ? { endedBy: this.turnEndedBy, ...(this.turnEndedBy === "deadline" ? { deadline: true } : {}) } : {}), @@ -5240,6 +5307,7 @@ export class Engine { let requirements = normalizeRequirements(opts.requirements); let signedOut = false; this.passedBeforeWork = new Set(); + this.goldensBefore = new Map(); const log = this.cfg.journal; /** * This turn's criteria, copied and sealed before anything runs. @@ -5276,7 +5344,9 @@ export class Engine { const self = this; /** Freeze, seal, journal and try the criteria — all before any change. */ async function* sealCriteria(checks: Check[], notes: string[]): AsyncGenerator { - taskChecks = checks.map((c) => Object.freeze({ ...c })); + // Who wrote each check is fixed here, with the checks: a check nobody + // vouched for is the worker's (src/tiers.ts withAuthor). + taskChecks = checks.map((c) => Object.freeze(withAuthor({ ...c }, self.cfg.model))); taskNotes = [...notes]; Object.freeze(taskChecks); Object.freeze(taskNotes); @@ -5291,6 +5361,7 @@ export class Engine { text: `task criteria sealed: ${taskChecks.length} check(s), ${taskNotes.length} note(s)`, seal: taskSeal, checks: taskChecks.map((c) => c.name), + authors: Object.fromEntries(taskChecks.map((c) => [c.name, authorWords(c.author)])), notes: taskNotes, }); yield { @@ -5303,6 +5374,8 @@ export class Engine { // meant to fail before the work; it is not meant to be unrunnable, and // the difference is cheap to establish here and expensive to discover // at the end of a turn. + // What each expected-looking file held before the work (src/golden.ts). + self.goldensBefore = recordGoldens(taskChecks.map((c) => ({ run: c.kind === "command" ? c.run : undefined })), self.cwd); if (taskChecks.length) { // Through `running`, so ctrl+C at the very start of a turn kills the // preflight rather than waiting it out. @@ -5537,7 +5610,9 @@ export class Engine { } referencePending = undefined; if (!got) return; - const check = Object.freeze({ ...got.check, name: got.check.name.startsWith("task:") ? got.check.name : `task:${got.check.name}` }); + const check = Object.freeze( + withAuthor({ ...got.check, name: got.check.name.startsWith("task:") ? got.check.name : `task:${got.check.name}` }, self.cfg.model), + ); if (taskChecks.some((c) => c.name === check.name)) return; taskChecks = Object.freeze([...taskChecks, check]) as Check[]; self.sealedChecks = taskChecks; @@ -5612,7 +5687,7 @@ export class Engine { } const have = new Set(taskChecks.map((c) => c.name)); const added = (got?.taskChecks ?? []) - .map((c) => Object.freeze({ ...c, name: c.name.startsWith("task:") ? c.name : `task:${c.name}` })) + .map((c) => Object.freeze(withAuthor({ ...c, name: c.name.startsWith("task:") ? c.name : `task:${c.name}` }, self.cfg.model))) .filter((c) => !have.has(c.name)); if (!added.length) { log?.append("note", { kind: "late-checks", text: "the drafter finished with no checks to join", inputsSha: draftInputs }); @@ -5770,7 +5845,8 @@ export class Engine { notes: [...taskNotes], } : undefined, - ...(verdict === "accepted" ? { tier: tierOf({ results: result.results, ...self.advisoryTier() }) } : {}), + ...(verdict === "accepted" ? { tier: tierOf({ results: result.results, ...self.tierContext() }) } : {}), + authors: self.receiptAuthors(), }); log?.append("receipt", { verdict, file: receipt.path, attempt: proofAttempts, endedBy: why }); self.bindReceipt(receipt.path, verdict); @@ -7643,7 +7719,8 @@ export class Engine { } : undefined, ...(this.turnRevealed.length ? { revealed: [...this.turnRevealed] } : {}), - ...(verdict === "accepted" ? { tier: tierOf({ results: result.results, ...this.advisoryTier() }) } : {}), + ...(verdict === "accepted" ? { tier: tierOf({ results: result.results, ...this.tierContext() }) } : {}), + authors: this.receiptAuthors(), }); log?.append("receipt", { verdict, file: receipt.path, attempt: proofAttempts }); this.bindReceipt(receipt.path, verdict); diff --git a/src/golden.ts b/src/golden.ts new file mode 100644 index 0000000..43beb32 --- /dev/null +++ b/src/golden.ts @@ -0,0 +1,70 @@ +/** + * Expected files ("golden" operands of a `diff`/`cmp` check) and whether they + * predate the work. + * + * `diff out.txt expected.txt` asserts a value only when expected.txt is the + * task's, not the worker's: a worker that writes both files makes it pass on + * anything. The seal records each expected-looking operand's content before + * the first step; at the tier, a check whose value rests only on such a file + * counts as asserting a value only when the file was there then and is + * byte-identical now. A check that joined after the work began has no record, + * so its golden file proves nothing. + */ +import { createHash } from "node:crypto"; +import { readFileSync, statSync } from "node:fs"; +import { isAbsolute, join } from "node:path"; +import { assertsValue, goldenOperands } from "./tiers.js"; + +/** Content digest of a regular file, or null when there is none. */ +export function fileDigest(abs: string): string | null { + try { + if (!statSync(abs).isFile()) return null; + return createHash("sha256").update(readFileSync(abs)).digest("hex"); + } catch { + return null; + } +} + +function resolveIn(cwd: string, operand: string): string { + return isAbsolute(operand) ? operand : join(cwd, operand); +} + +/** Each expected-looking operand of these checks, by resolved path, with its digest now (null: absent). */ +export function recordGoldens(checks: readonly { run?: string }[], cwd: string, into = new Map()): Map { + for (const c of checks) { + if (!c.run) continue; + for (const g of goldenOperands(c.run)) { + const abs = resolveIn(cwd, g); + if (!into.has(abs)) into.set(abs, fileDigest(abs)); + } + } + return into; +} + +/** + * Names of value-tagged checks whose value rests only on expected files that + * did not predate the work (absent from `before`, absent then, or changed + * since). tierOf drops their value tag. + */ +export function valueUnproven( + checks: readonly { name: string; run?: string; tags?: readonly string[] }[], + cwd: string, + before: ReadonlyMap, +): Set { + const out = new Set(); + const predates = (g: string): boolean => { + const abs = resolveIn(cwd, g); + const then = before.get(abs); + return typeof then === "string" && fileDigest(abs) === then; + }; + for (const c of checks) { + if (!c.run || !c.tags?.includes("value")) continue; + // Only a check whose value came from an expected file by name; one tagged + // value for another reason (a literal, the reference driver) is untouched. + if (assertsValue(c.run) && !assertsValue(c.run, predates)) { + out.add(c.name); + if (!c.name.startsWith("task:")) out.add(`task:${c.name}`); + } + } + return out; +} diff --git a/src/mission.ts b/src/mission.ts index bb505f6..3a17d61 100644 --- a/src/mission.ts +++ b/src/mission.ts @@ -348,6 +348,8 @@ export function assertionChecks(contract: Contract, ids: readonly string[]): Che tags: ["mission"], // The worker is told what must be true, not how it will be checked. hidden: true, + // A model may propose the contract; a person edits it and `mission run` seals what is on disk. + author: { kind: "person" }, }); } return out; diff --git a/src/receipts.ts b/src/receipts.ts index 3a33953..f3b63c2 100644 --- a/src/receipts.ts +++ b/src/receipts.ts @@ -11,7 +11,8 @@ import { appendFileSync, existsSync, mkdirSync, readFileSync, readdirSync, write import { createHash } from "node:crypto"; import { join } from "node:path"; import { MIN_SECRET_CHARS, redact } from "./redact.js"; -import type { BarResult } from "./types.js"; +import type { BarResult, CheckAuthor } from "./types.js"; +import { authorWords, claimLabel, resultAuthorWords } from "./tiers.js"; import { stateDir } from "./statedir.js"; import { WITHHELD, maskDeep, maskText } from "./withhold.js"; @@ -85,8 +86,10 @@ export type ReceiptRecord = { * (`amendTier`), so the index row is the last word and the receipt file, * hash-bound when it was written, is never rewritten. */ - tier?: "verified" | "passed-checks"; + tier?: "verified" | "passed-checks" | "passed-own-checks"; tierReason?: string; + /** The claim in words (src/tiers.ts claimLabel), e.g. "verified (independent checks: m)". */ + claim?: string; /** The strongest evidence class among the passing checks. */ evidence?: string; }; @@ -369,7 +372,13 @@ export class Receipts { /** True when that figure rests on molt's own token estimate. */ costEstimated?: boolean; /** The tier the passing checks earned, before any independent review (src/tiers.ts). */ - tier?: { tier: "verified" | "passed-checks"; reason?: string; evidence: string }; + tier?: { tier: "verified" | "passed-checks" | "passed-own-checks"; reason?: string; evidence: string; basis?: "person" | "independent" | "own"; by?: string[]; worker?: string }; + /** + * Who wrote each check, by check name, recorded at seal time: the worker + * model, a separate judge, a person, or the reference writer. A check not + * named here is printed as unrecorded. + */ + authors?: Record; /** True for a question: the bar ran advisory and could not refuse. */ ask?: boolean; /** The model was stopped (clock, provider) and the tree was judged as it stood. */ @@ -439,7 +448,10 @@ export class Receipts { if (args.verdict === "accepted" && !args.ask && args.tier) { verdictLine += args.tier.tier === "verified" ? `\n\nEvidence: ${args.tier.evidence}. A passing check of this class earns the word "verified".` - : `\n\nPassed its checks, not verified: ${args.tier.reason}. The strongest passing check is ${args.tier.evidence}.`; + : args.tier.tier === "passed-own-checks" + ? `\n\nPassed own checks, not verified: ${args.tier.reason}. A model never judges its own work; a check from a person or another model is needed for "verified".` + : `\n\nPassed its checks, not verified: ${args.tier.reason}. The strongest passing check is ${args.tier.evidence}.`; + verdictLine += `\n\nClaim: ${args.tier.tier === "passed-checks" ? "passed its checks, not verified" : claimLabel("verified", args.tier)}.`; } if (args.revealed?.length) { verdictLine += `\n\nThe command of ${args.revealed.map((n) => `\`${n}\``).join(", ")} was shown to the model after it failed the same way twice; the work was judged against a check the model had read.`; @@ -461,7 +473,11 @@ export class Receipts { ); if (task.checks.length) { asked.push("**Machine-checked.** These ran with the bar and could refuse the claim:", ""); - for (const c of task.checks) asked.push(`- \`${c}\``); + for (const c of task.checks) { + const name = c.slice(0, c.indexOf(": ") >= 0 ? c.indexOf(": ") : c.length); + const a = args.authors?.[name] ?? args.authors?.[`task:${name}`]; + asked.push(`- \`${c}\` — written by ${authorWords(a)}`); + } asked.push(""); } if (task.notes.length) { @@ -596,6 +612,7 @@ export class Receipts { "", `check: ${r.name}`, `kind: ${r.kind}`, + `written by: ${resultAuthorWords(r, args.authors?.[r.name])}`, `command: ${r.detail}`, ...(r.ranInPlace ? [`ran in place: no throwaway copy of the tree — ${r.ranInPlace}`] : []), `exit: ${r.exitCode ?? "n/a"}`, @@ -691,7 +708,12 @@ export class Receipts { ...(args.revealed?.length ? { revealed: [...args.revealed] } : {}), ...(args.endedBy ? { endedBy: args.endedBy } : {}), ...(args.verdict === "accepted" && !args.ask && args.tier - ? { tier: args.tier.tier, evidence: args.tier.evidence, ...(args.tier.reason ? { tierReason: args.tier.reason } : {}) } + ? { + tier: args.tier.tier, + evidence: args.tier.evidence, + ...(args.tier.reason ? { tierReason: args.tier.reason } : {}), + ...(args.tier.tier !== "passed-checks" ? { claim: claimLabel("verified", args.tier) } : {}), + } : {}), // Only when the caller said what changed. A row with no count is // unknown, and unknown is not zero — it is counted as a change, which is @@ -718,7 +740,7 @@ export class Receipts { * is hash-bound and stays byte for byte as written. A row that is not * there, or an index that cannot be read, is left alone. */ - amendTier(file: string, tier: { tier: "verified" | "passed-checks"; reason?: string }): boolean { + amendTier(file: string, tier: { tier: "verified" | "passed-checks" | "passed-own-checks"; reason?: string; claim?: string }): boolean { try { if (!existsSync(this.indexPath)) return false; let hit = false; @@ -730,8 +752,13 @@ export class Receipts { const r = JSON.parse(l) as ReceiptRecord; if (r.file !== file) return l; hit = true; - const { tierReason: _drop, ...rest } = r; - return JSON.stringify({ ...rest, tier: tier.tier, ...(tier.reason ? { tierReason: redact(tier.reason, this.secrets) } : {}) }); + const { tierReason: _drop, claim: _was, ...rest } = r; + return JSON.stringify({ + ...rest, + tier: tier.tier, + ...(tier.reason ? { tierReason: redact(tier.reason, this.secrets) } : {}), + ...(tier.claim ? { claim: tier.claim } : {}), + }); } catch { return l; } diff --git a/src/reference.ts b/src/reference.ts index 8d55042..579989d 100644 --- a/src/reference.ts +++ b/src/reference.ts @@ -167,9 +167,24 @@ export const SECOND_REFERENCE_SYSTEM = [ * every input before the deliverable runs; judges the deliverable only where * they agree; and exits 3 — the check's own failure, never the work's — when * a reference fails, the two mostly disagree, or run_deliverable raises. + * + * The deliverable never runs in this process. Each input goes to a fresh + * child (RUNNER), which loads check.py, drops reference() and the inputs from + * it, calls run_deliverable and sends back repr() of the result on its own + * pipe. An import-style deliverable used to be imported here, so a module + * with `os._exit(0)` at top level ended the driver with exit 0 before any + * comparison, and the check passed on wrong work. Now a child that ends + * without sending a result (os._exit, sys.exit, a crash) is a result: "the + * deliverable ended the process", which matches nothing. Expected values + * live only in this process's memory: never on disk, never in a child. + * + * The last line is `REFERENCE COMPARED n/n`. Maat requires it for a pass + * (src/bar.ts referenceComparedAll): an exit 0 without it is a failure, so + * nothing that ends the driver early can pass for it. */ +export const REFERENCE_SENTINEL = "REFERENCE COMPARED"; export const DRIVER = [ - "import importlib.util, json, os, sys, traceback", + "import ast, importlib.util, json, os, pickle, signal, subprocess, sys, threading, traceback", "HERE = os.path.dirname(os.path.abspath(__file__))", "def load(name):", " spec = importlib.util.spec_from_file_location(name, os.path.join(HERE, name + '.py'))", @@ -179,6 +194,40 @@ export const DRIVER = [ " except Exception: return repr(v)", "def own(msg):", " sys.stdout.flush(); sys.stderr.write('REFERENCE ERROR: ' + msg + '\\n'); sys.exit(3)", + "def deliver(x):", + " # One child per input: the deliverable runs there, never in this process.", + " r, w = os.pipe()", + " got = []", + " def drain():", + " with os.fdopen(r, 'rb') as f: got.append(f.read(16 * 1024 * 1024))", + " t = threading.Thread(target=drain, daemon=True); t.start()", + " try:", + " p = subprocess.Popen([sys.executable, os.path.join(HERE, 'runner.py'), os.path.join(HERE, 'check.py'), str(w)],", + " stdin=subprocess.PIPE, stdout=subprocess.DEVNULL, stderr=subprocess.PIPE, pass_fds=(w,), start_new_session=True)", + " finally:", + " os.close(w)", + " try:", + " _, err = p.communicate(pickle.dumps(x), timeout=90)", + " except subprocess.TimeoutExpired:", + " try: os.killpg(p.pid, signal.SIGKILL)", + " except Exception: pass", + " p.communicate(); t.join(5)", + " return ('value', '')", + " try: os.killpg(p.pid, signal.SIGKILL)", + " except Exception: pass", + " t.join(10)", + " raw = got[0] if got else b''", + " try: msg = json.loads(raw.decode('utf-8')) if raw else None", + " except Exception: msg = None", + " if not isinstance(msg, dict):", + " return ('value', '' % p.returncode)", + " if 'raised' in msg:", + " sys.stderr.write(str(msg['raised'])[-4000:]); return ('raised', None)", + " if 'exited' in msg:", + " return ('value', '' % (msg['exited'],))", + " text = str(msg.get('repr', ''))", + " try: return ('value', ast.literal_eval(text))", + " except Exception: return ('value', text)", "try:", " a = load('check'); b = load('second'); inputs = list(a.INPUTS)[:200]", "except SystemExit: raise", @@ -197,17 +246,63 @@ export const DRIVER = [ " x, ea, eb = disagreed[0]", " own('the two independent references agree on only %d of %d inputs; first disagreement, input %r: %r vs %r' % (len(agreed), len(inputs), x, ea, eb))", "same = getattr(a, 'same', None)", + "compared = 0", "for x, e, _ in agreed:", - " try: got = a.run_deliverable(x)", - " except BaseException: traceback.print_exc(); own('run_deliverable raised on input %r' % (x,))", + " how, got = deliver(x)", + " if how == 'raised': own('run_deliverable raised on input %r' % (x,))", " try: ok = same(e, got) if same else key(e) == key(got)", " except BaseException: traceback.print_exc(); own('the comparison failed on input %r' % (x,))", " if not ok:", " print('input: %r\\nexpected: %r\\nactual: %r\\n(two independent references agree on the expected result)' % (x, e, got)); sys.exit(1)", - "print('%d inputs match two independent references%s' % (len(agreed), '; %d where they disagreed were not judged' % len(disagreed) if disagreed else ''))", + " compared += 1", + "print('%d inputs match two independent references%s' % (compared, '; %d where they disagreed were not judged' % len(disagreed) if disagreed else ''))", + "print('" + "REFERENCE COMPARED" + " %d/%d' % (compared, len(agreed)))", "", ].join("\n"); +/** + * The child the driver runs the deliverable in, one per input. It sends one + * JSON message on the pipe it is handed: {"repr": repr(result)}, {"raised": + * traceback} when run_deliverable raised, or {"exited": code} when the + * deliverable called sys.exit. A child that sends nothing ended early. + * reference() and INPUTS are removed from the module before the deliverable + * runs, so the work's code cannot simply call them. + */ +export const RUNNER = [ + "import importlib.util, json, os, pickle, sys, traceback", + "out = os.fdopen(int(sys.argv[2]), 'w')", + "def send(d):", + " try: out.write(json.dumps(d)); out.flush(); out.close()", + " except Exception: pass", + "x = pickle.loads(sys.stdin.buffer.read())", + "try:", + " spec = importlib.util.spec_from_file_location('check', sys.argv[1])", + " m = importlib.util.module_from_spec(spec); spec.loader.exec_module(m)", + " for n in ('reference', 'INPUTS'):", + " if hasattr(m, n): delattr(m, n)", + " sys.modules.pop('check', None)", + " got = m.run_deliverable(x)", + "except SystemExit as e:", + " send({'exited': repr(e.code)}); os._exit(0)", + "except BaseException:", + " send({'raised': traceback.format_exc()}); os._exit(0)", + "try: text = repr(got)", + "except BaseException: text = ''", + "send({'repr': text})", + "os._exit(0)", + "", +].join("\n"); + +/** + * Did the reference driver say it compared every input it judged? A pass + * needs its last `REFERENCE COMPARED n/n` line with n > 0: an exit 0 without + * it (the driver ended early, or something else answered) is not a pass. + */ +export function referenceComparedAll(output: string): boolean { + const m = [...output.matchAll(/^REFERENCE COMPARED (\d+)\/(\d+)\s*$/gm)].at(-1); + return !!m && m[1] === m[2] && Number(m[1]) > 0; +} + /** What the second reviewer is told about the inputs: their format and a few of them, never an answer. */ export const PROBE = [ "import importlib.util, json, os, sys", @@ -347,6 +442,7 @@ async function writeAndTry(opts: Ask, prompt: string): Promise boolean): boolean { const ops = /(==|!=|(?<=\s)-eq(?=\s)|(?<=\s)-ne(?=\s)|(?<=\s)=(?=\s))/g; for (const m of run.matchAll(ops)) { const at = m.index ?? 0; @@ -77,7 +102,13 @@ export function assertsValue(run: string): boolean { for (const m of run.matchAll(/\b(?:diff|cmp)\b([^\n|;&]*)/g)) { const args = m[1] ?? ""; if (/<<|<\(/.test(args)) return true; - if (/(?:^|[\s/'"])[\w.-]*(?:expect|golden|want|answer|baseline|correct)/i.test(args)) return true; + if (GOLDEN_NAME.test(args)) { + // Without a pre-work record (the drafter tagging a check), the name is + // taken at its word; the tier asks again with one (`goldenPredates`). + if (!goldenPredates) return true; + const named = goldenOperands(m[0]); + if (named.length && named.every((g) => goldenPredates(g))) return true; + } } for (const m of run.matchAll(/\bgrep\b([^\n|;&]*)/g)) { const words: string[] = (m[1] ?? "").match(/"(?:[^"\\]|\\.)*"|'[^']*'|\S+/g) ?? []; @@ -98,7 +129,15 @@ export function evidenceTags(run: string, surface: boolean | undefined): string[ return ["task", ...(surface ? ["surface"] : []), ...(assertsValue(run) ? ["value"] : [])]; } -export type Tier = "verified" | "passed-checks"; +/** + * "passed-own-checks": every passing check that ran the work and asserted a + * value was written by the worker model itself (or by a judge that is the + * same model). Never verified: Maat's rule is that the model that did the + * work is never the one that judges it. On the 2026-10-06/07 container + * bench, self-judged arms were right in 125 of 174 "verified" claims (72%), + * separate-judge arms in 48 of 55 (87%). + */ +export type Tier = "verified" | "passed-checks" | "passed-own-checks"; export type TierVerdict = { tier: Tier; reason?: string; @@ -106,8 +145,175 @@ export type TierVerdict = { evidence: "person" | "runs+value" | "runs" | "surface" | "none"; /** Advisory review mode only: what the review said, recorded on the receipt instead of gating. */ reviewNote?: string; + /** + * Advisory review mode only: the review contradicted the task, or was asked + * for and did not run. The tier still stands (the review gates nothing in + * that mode), but every label says so: ", unconfirmed" / ", unreviewed". + */ + reviewGap?: "unconfirmed" | "unreviewed"; + /** + * Who stood behind the word, when the passing runs+value (or person) checks + * decided it: "person", "independent" (another model), or "own" (the worker). + */ + basis?: "person" | "independent" | "own"; + /** The models that wrote the independent passing runs+value checks ("independent" basis). */ + by?: string[]; + /** The worker model the checks were weighed against. */ + worker?: string; }; +/** + * A model id with its provider's spelling taken off, so the same model + * reached two ways compares equal: `minimax/minimax-m3:free` on OpenRouter + * and `MiniMax-M3` direct are one model, and one model judging its own work + * is not independent whichever door it came in by. + */ +export function normalizeModelId(id: string): string { + let s = id.trim().toLowerCase(); + // Provider routing: `openrouter/qwen/qwen3-235b` -> `qwen3-235b`. + s = s.slice(s.lastIndexOf("/") + 1); + // Variant suffixes: `:free`, `:nitro`, Bedrock's `:0`. + s = s.replace(/:.*$/, ""); + // Bedrock / Vertex vendor prefixes: `us.anthropic.claude-...`. + s = s.replace(/^(?:[a-z]{2}\.)?(?:anthropic|meta|amazon|mistral|cohere|ai21|deepseek|qwen|google|openai|xai|minimax)\./, ""); + s = s.replace(/[._\s]+/g, "-"); + // Dated and alias releases of the same model: `-20250805`, `-2025-08-05`, `-latest`, + // Bedrock's `-20240620-v1`. A bare `-v3` is a different model (deepseek-v3), never stripped. + s = s.replace(/-(?:\d{8}(?:-v\d+)?|\d{4}-\d{2}-\d{2}|latest)$/, ""); + // Tuning names one provider spells and another drops: `qwen3-235b-a22b-instruct-2507` + // on one is `qwen3-235b-a22b-2507` on another; `gemma-3-27b-it`. Erring toward + // "same model" fails closed: it can only cost a judge its independence. + s = s.replace(/-(?:instruct|chat|it)(?=-|$)/g, ""); + return s; +} + +/** Same model, provider spelling aside. An empty id matches nothing. */ +export function sameModel(a: string | undefined, b: string | undefined): boolean { + if (!a?.trim() || !b?.trim()) return false; + return normalizeModelId(a) === normalizeModelId(b); +} + +/** + * Whether a check by this author can judge the worker's work. A person can. + * A model can only when it is known and is not the worker model under any + * of the worker's names. An author nobody recorded is the worker: unknown + * authorship never earns the word. + */ +export function independentOf(author: CheckAuthor | undefined, worker: string | readonly string[] | undefined): boolean { + if (!author) return false; + if (author.kind === "person") return true; + if (author.kind === "worker") return false; + if (!author.model?.trim()) return false; + const names = (typeof worker === "string" ? [worker] : [...(worker ?? [])]).filter((w) => w.trim()); + if (!names.length) return false; + return !names.some((w) => sameModel(author.model, w)); +} + +/** + * Can this result stand for a person's judgment of the task? Only a command + * check (`kind: "command"`, never a builtin) that is not one of the session + * checks. A result with no recorded kind cannot: every real result carries + * one, so a missing kind is a caller that did not say, and that fails closed. + */ +export function personCheck(r: { kind?: CheckResult["kind"]; tags?: readonly string[] }): boolean { + return r.kind === "command" && !r.tags?.includes("session"); +} + +/** One line for a receipt: who wrote this result's check. Builtins and session checks are Maat's, whatever done.yml says. */ +export function resultAuthorWords(r: { kind?: CheckResult["kind"]; tags?: readonly string[]; hidden?: boolean }, a: CheckAuthor | undefined): string { + if (r.kind === "builtin" || r.tags?.includes("session")) return "Maat (a session check: how the turn was done, never whether the task is right; does not count toward verified)"; + return authorWords(a ?? (r.hidden ? undefined : { kind: "person" })); +} + +/** One line for a receipt: who wrote this check. */ +export function authorWords(a: CheckAuthor | undefined): string { + if (!a) return "unrecorded (treated as the worker model)"; + switch (a.kind) { + case "person": + return "a person (your check)"; + case "judge": + return `the judge model${a.model ? ` ${a.model}` : ""}`; + case "reference": + return `the reference writer${a.model ? ` ${a.model}` : ""}`; + default: + return `the worker model${a.model ? ` ${a.model}` : ""}`; + } +} + +/** A check's author as one short machine-readable string: "person", "worker ", "judge ", "reference ". */ +export function authorKey(a: CheckAuthor): string { + return a.kind === "person" ? "person" : `${a.kind}${a.model ? ` ${a.model}` : ""}`; +} + +/** A check's author, filled in where nobody recorded one: a hidden check is the worker's, a visible one a person's. */ +export function withAuthor(c: T, worker: string): T { + if (c.author) return c; + const person = c.hidden !== true || c.tags?.includes("mission") === true; + return { ...c, author: person ? { kind: "person" } : { kind: "worker", model: worker } }; +} + +/** + * The claim as every surface prints it, and as the bench's `claim` field + * carries it, so an analysis can tell who stood behind each "verified": + * verified (independent checks: ) + * verified (your checks) + * passed own checks (), not verified + * Anything else is the outcome word unchanged. + */ +export function claimLabel(outcome: string, tier: Pick | undefined): string { + const gap = tier?.reviewGap ? `, ${tier.reviewGap}` : ""; + if (tier?.tier === "passed-own-checks") return `passed own checks (${tier.worker || "the worker model"}), not verified${gap}`; + if (outcome === "verified" && tier?.tier === "verified") { + if (tier.basis === "person") return `verified (your checks)${gap}`; + if (tier.basis === "independent") return `verified (independent checks: ${(tier.by ?? []).join(", ") || "another model"})${gap}`; + } + return outcome; +} + +/** The parts of a job_end event the verdict words are made from. */ +export type JobEndWords = { + outcome?: unknown; + tier?: unknown; + tierReason?: unknown; + claim?: unknown; + revealed?: unknown; + review?: { confirmed?: boolean } | null; + unreviewed?: unknown; + selfChecked?: unknown; + checksDisagree?: unknown; +}; + +/** + * How a job ended, in the words the terminal and the window both print. + * + * Order matters: a review that contradicted the task, or one that was asked + * for and did not run, qualifies every "verified", including one that carries + * a claim. In review-advisory mode the tier stays verified with a review note, + * and a claim branch tested first used to print "verified (independent checks: + * j)" with no qualifier. The qualifier is now in the claim itself (claimLabel's + * `reviewGap`); a claim from a build that did not put it there gets it added. + */ +export function jobEndWords(ev: JobEndWords): string { + const claim = typeof ev.claim === "string" && ev.claim ? ev.claim : undefined; + const verified = ev.outcome === "verified"; + const contradicted = verified && !!ev.review && ev.review.confirmed === false; + const unreviewed = verified && !ev.review && !!ev.unreviewed; + if (ev.tier === "passed-checks") return passedChecksWords(typeof ev.tierReason === "string" ? ev.tierReason : undefined); + if (ev.tier === "passed-own-checks" && claim) return claim; + if (verified && Array.isArray(ev.revealed) && ev.revealed.length) return "verified (checks shown after a repeat failure)"; + if (verified && claim) { + if (contradicted) return /, unconfirmed$/.test(claim) ? claim : `${claim}, unconfirmed`; + if (unreviewed) return /, unreviewed$/.test(claim) ? claim : `${claim}, unreviewed`; + return `${claim}${ev.review?.confirmed ? ", independently reviewed" : ""}`; + } + if (contradicted) return "passed its checks, unconfirmed"; + if (verified && ev.review?.confirmed) return "verified, independently reviewed"; + if (unreviewed) return "passed its checks, unreviewed"; + if (verified && ev.selfChecked) return "passed its own checks"; + if (ev.outcome === "unverified" && Array.isArray(ev.checksDisagree) && ev.checksDisagree.length) return "unverified, its own drafted checks disagree"; + return String(ev.outcome); +} + /** * The verdict words for a turn that passed its checks without earning * "verified". One phrase on every surface, so the same turn reads the same @@ -127,15 +333,24 @@ export function contradictions(review: { votes: string; violations: unknown[] } /** * The tier a passing turn has earned. "verified" needs (a) a passing check a * person wrote, or a passing drafted check that runs the deliverable (not - * surface) AND asserts a value, and (b) an independent review, when one ran, - * that found no contradiction. (c) — the turn not ended by the clock or the - * provider — is decided before this, by `passedAtEnd`. + * surface) AND asserts a value AND was written by someone other than the + * worker model, and (b) an independent review, when one ran, that found no + * contradiction. (c) — the turn not ended by the clock or the provider — is + * decided before this, by `passedAtEnd`. * - * A person's check is not Maat's to second-guess: a passing check from the - * project's done.yml (not hidden) carries "verified" as it always did. + * A person's check is not Maat's to second-guess: a passing command check + * from the project's done.yml (not hidden), or one a person approved, carries + * "verified". Builtins and `session` checks never do (`personCheck`): they + * are in every `maat init` bar, nobody wrote them for this task, and any + * worker that changes a file passes them. + * + * Never let a model both find and judge: when every passing runs+value check + * was written by the worker model (no judge, or a judge that is the same + * model reached another way), the best the turn earns is "passed-own-checks". + * A check whose author was never recorded counts as the worker's. */ export function tierOf(args: { - results: readonly (Pick & { name?: string })[]; + results: readonly (Pick & { name?: string; kind?: CheckResult["kind"] })[]; review?: { votes: string; violations: unknown[] } | null; /** An independent review was asked for and did not run: it cannot have found nothing. */ unreviewed?: boolean; @@ -150,16 +365,43 @@ export function tierOf(args: { reviewAdvisory?: boolean; /** Names (as in `results`) of checks that already passed before the work. */ guards?: ReadonlySet; + /** The worker model, under every name it ran as (configured id, the id the backend reported). */ + worker?: string | readonly string[]; + /** Who wrote each check, by result name, recorded at seal time. */ + authors?: ReadonlyMap; + /** + * Checks whose `value` tag rests only on a `diff`/`cmp` against an + * expected-looking file that was not there before the work, or changed + * since (src/golden.ts). Their value tag does not count. + */ + valueUnproven?: ReadonlySet; }): TierVerdict { - const passing = args.results.filter((r) => r.ok && !r.advisory && !r.skipped); + const passing = args.results + .filter((r) => r.ok && !r.advisory && !r.skipped) + .map((r) => (args.valueUnproven?.has(r.name ?? "") && r.tags?.includes("value") ? { ...r, tags: r.tags.filter((t) => t !== "value") } : r)); + const workerNames = (typeof args.worker === "string" ? [args.worker] : [...(args.worker ?? [])]).filter((w) => w.trim()); + const worker = workerNames[0]; // A mission's assertions are its contract, written before the work and // sealed with it (mission.ts): they are the person's bar, not a model's // guess at one, so they carry the word as a done.yml check does. - const person = passing.some((r) => r.hidden !== true || r.tags?.includes("mission")); - const drafted = passing.filter((r) => r.hidden === true && r.tags?.includes("task")); - const strong = drafted.some( + const authorOf = (r: (typeof passing)[number]): CheckAuthor => + args.authors?.get(r.name ?? "") ?? + (r.hidden !== true || r.tags?.includes("mission") ? { kind: "person" } : { kind: "worker", ...(worker ? { model: worker } : {}) }); + // Only a command a person wrote or approved can carry the word for them. + // Maat's builtins (work-landed, record-intact, claims-grounded, ...) and the + // session checks every `maat init` bar ships say the turn was well-formed, + // never that the task is right: nobody wrote them for this task, and a + // worker that changes any file passes them. + const person = passing.some((r) => personCheck(r) && authorOf(r).kind === "person"); + const drafted = passing.filter((r) => r.hidden === true && r.tags?.includes("task") && authorOf(r).kind !== "person"); + const strongAll = drafted.filter( (r) => !r.tags?.includes("surface") && r.tags?.includes("value") && !(args.reviewAdvisory && args.guards?.has(r.name ?? "")), ); + const strongIndependent = strongAll.filter((r) => independentOf(authorOf(r), workerNames)); + const strong = strongAll.length > 0; + const by = [...new Set(strongIndependent.map((r) => authorOf(r).model ?? "another model"))]; + const basis: TierVerdict["basis"] = person ? "person" : strongIndependent.length ? "independent" : strong ? "own" : undefined; + const who = { ...(basis ? { basis } : {}), ...(by.length && !person ? { by } : {}), ...(worker ? { worker } : {}) }; const evidence: TierVerdict["evidence"] = person ? "person" : strong @@ -169,13 +411,18 @@ export function tierOf(args: { : drafted.length ? "surface" : "none"; + const ownReason = `every passing check that ran the work and asserted a value was written by the worker model${worker ? ` (${worker})` : ""}`; + const earned = person || strongIndependent.length > 0; if (args.reviewAdvisory) { const n = contradictions(args.review); const reviewNote = n > 0 ? `advisory: the independent review found ${args.review!.votes} contradicting the task` : args.unreviewed ? "advisory: the independent review did not run" : undefined; - if (person || strong) return { tier: "verified", evidence, ...(reviewNote ? { reviewNote } : {}) }; + const gap: Pick = n > 0 ? { reviewGap: "unconfirmed" } : args.unreviewed ? { reviewGap: "unreviewed" } : {}; + if (earned) return { tier: "verified", evidence, ...who, ...(reviewNote ? { reviewNote } : {}), ...gap }; + if (strong) return { tier: "passed-own-checks", evidence, ...who, reason: ownReason, ...(reviewNote ? { reviewNote } : {}), ...gap }; return { tier: "passed-checks", evidence, + ...who, reason: drafted.some((r) => !r.tags?.includes("surface") && r.tags?.includes("value")) ? "the only passing value check also passed before the work began" : evidence === "runs" @@ -185,13 +432,15 @@ export function tierOf(args: { }; } if (contradictions(args.review) > 0) { - return { tier: "passed-checks", evidence, reason: `the independent review found ${args.review!.votes} contradicting the task` }; + return { tier: "passed-checks", evidence, ...who, reason: `the independent review found ${args.review!.votes} contradicting the task` }; } - if (args.unreviewed && !person) return { tier: "passed-checks", evidence, reason: "the independent review did not run" }; - if (person || strong) return { tier: "verified", evidence }; + if (args.unreviewed && !person) return { tier: "passed-checks", evidence, ...who, reason: "the independent review did not run" }; + if (earned) return { tier: "verified", evidence, ...who }; + if (strong) return { tier: "passed-own-checks", evidence, ...who, reason: ownReason }; return { tier: "passed-checks", evidence, + ...who, reason: evidence === "runs" ? "no check that ran the work asserted an expected value" diff --git a/src/types.ts b/src/types.ts index 5283af5..674acc5 100644 --- a/src/types.ts +++ b/src/types.ts @@ -115,8 +115,25 @@ export type Advisory = { * back on failure; only the text of the command is withheld. */ hidden?: boolean; + /** + * Who wrote this check, recorded when it is sealed. "verified" needs a + * passing check that ran the work and asserted a value from someone other + * than the worker model (src/tiers.ts). Absent: a visible check is a + * person's, a hidden one the worker's. + */ + author?: CheckAuthor; }; +/** + * The author of a check. `worker`: the model doing the work drafted it. + * `judge`: a separate judge model (`--judge`, MAAT_JUDGE_MODEL) drafted it. + * `person`: done.yml, a mission contract, or criteria a person approved. + * `reference`: the reference check (src/reference.ts), written by `model`. + * Independence is decided by comparing `model` to the worker's, never by kind + * alone: a judge that is the worker model is not independent. + */ +export type CheckAuthor = { kind: "worker" | "judge" | "person" | "reference"; model?: string }; + export type Check = Advisory & ( | { @@ -663,8 +680,17 @@ export type EngineEvent = * check) and no reviewer contradicted it. "passed-checks" is the outcome * "unverified" with `tierReason` saying why the word was not earned. */ - tier?: "verified" | "passed-checks"; + tier?: "verified" | "passed-checks" | "passed-own-checks"; tierReason?: string; + /** + * The claim in words, the same on every surface and in the bench's + * `claim` field: "verified (independent checks: )", "verified + * (your checks)", "passed own checks (), not verified", or the + * outcome word. + */ + claim?: string; + /** Who wrote each sealed check, by name: "worker ", "judge ", "person", "reference ". */ + checkAuthors?: Record; } | { kind: "step_summary"; diff --git a/test/ended-with-verdict.test.ts b/test/ended-with-verdict.test.ts index 9714e1f..e2113aa 100644 --- a/test/ended-with-verdict.test.ts +++ b/test/ended-with-verdict.test.ts @@ -15,6 +15,8 @@ import { allowAll, drain, scriptedProvider, workspace } from "./helpers.js"; const greet = (hidden = true): Check => ({ name: "greeting", kind: "command", run: "grep -qx hello out.txt", timeoutMs: 5_000, expectExit: 0, tags: ["task", "value"], hidden, + // Drafted by a separate judge model, not the worker "m". + author: { kind: "judge", model: "judge-j" }, }); function end(events: Awaited>) { @@ -128,7 +130,7 @@ describe("a provider that gives up after work happened", () => { }); describe("a check that exceeds its timeout", () => { - const hang: Check = { name: "hangs", kind: "command", run: "sleep 5", timeoutMs: 300, expectExit: 0, tags: ["task"], hidden: true }; + const hang: Check = { name: "hangs", kind: "command", run: "sleep 5", timeoutMs: 300, expectExit: 0, tags: ["task"], hidden: true, author: { kind: "judge", model: "judge-j" } }; it("is retired as a timeout, and the others judge", async () => { const ws = workspace(); diff --git a/test/exit-codes.test.ts b/test/exit-codes.test.ts index 68f1946..e8842c4 100644 --- a/test/exit-codes.test.ts +++ b/test/exit-codes.test.ts @@ -149,13 +149,25 @@ describe("molt run's exit code", () => { it("is 0 when checks drafted while the model read were met (--criteria auto)", async () => { // Drafted criteria arrive while the model reads, so none are passed in up // front. The exit code read that as "no criteria" and a verified turn - // exited 3. It is decided by the turn's own outcome now. + // exited 3. It is decided by the turn's own outcome now. Drafted by a + // separate judge: checks the worker drafts for itself never verify. const url = await draftingProvider(); const code = await molt(["run", "write a.txt", "--url", url, "--model", "m", "--key", "k", - "--cwd", project(), "--no-stream", "--yes", "--criteria", "auto"]); + "--cwd", project(), "--no-stream", "--yes", "--criteria", "auto", "--judge", "j"]); assert.equal(code, 0); }); + it("is 3 when the only checks met were drafted by the worker itself (passed own checks)", async () => { + const url = await draftingProvider(); + const code = await molt(["run", "write a.txt", "--url", url, "--model", "m", "--key", "k", + "--cwd", project(), "--no-stream", "--yes", "--criteria", "auto"]); + assert.equal(code, 3); + // A judge that is the worker under a provider's spelling is the worker. + const same = await molt(["run", "write a.txt", "--url", url, "--model", "m", "--key", "k", + "--cwd", project(), "--no-stream", "--yes", "--criteria", "auto", "--judge", "openrouter/m:free"]); + assert.equal(same, 3); + }); + it("is 3 when required checks were left out, not 1 as though something failed", async () => { assert.equal(await run(project(BAR), "--skip", "slow"), 3); }); diff --git a/test/harness.test.ts b/test/harness.test.ts index da1266f..8116bb8 100644 --- a/test/harness.test.ts +++ b/test/harness.test.ts @@ -487,7 +487,7 @@ describe("self-checked work", () => { it("is marked on job_end when every judging check was drafted by the model, and not otherwise", async () => { const ws = workspace(); try { - const hidden = { name: "own", kind: "command" as const, run: "grep -qx x x.txt", timeoutMs: 5_000, expectExit: 0, tags: ["task", "value"], hidden: true }; + const hidden = { name: "own", kind: "command" as const, run: "grep -qx x x.txt", timeoutMs: 5_000, expectExit: 0, tags: ["task", "value"], hidden: true, author: { kind: "judge" as const, model: "judge-j" } }; const shown = { ...hidden, name: "chosen", hidden: undefined }; const a = engineFor(ws.dir, [{ calls: [{ name: "write_file", args: { path: "x.txt", content: "x" } }] }, { text: "done" }]); const ea = await drain(a.engine.run("make x", allowAll, { taskChecks: [hidden] })); diff --git a/test/hidden-checks-withheld.test.ts b/test/hidden-checks-withheld.test.ts index 9658c6b..651062d 100644 --- a/test/hidden-checks-withheld.test.ts +++ b/test/hidden-checks-withheld.test.ts @@ -32,7 +32,7 @@ const TASK = "Write out.txt containing the greeting the release notes ask for."; const SECRET_A = "grep -q 'zebra-quantum-42' out.txt"; const SECRET_B = "test \"$(wc -l < out.txt)\" -ge 1 && grep -c 'zebra' out.txt"; const checks = (hidden: boolean): Check[] => [ - { name: "task:has-greeting", kind: "command", run: SECRET_A, timeoutMs: 5_000, expectExit: 0, tags: ["task", "value"], hidden }, + { name: "task:has-greeting", kind: "command", run: SECRET_A, timeoutMs: 5_000, expectExit: 0, tags: ["task", "value"], hidden, author: { kind: "judge", model: "judge-j" } }, { name: "task:non-empty", kind: "command", run: SECRET_B, timeoutMs: 5_000, expectExit: 0, tags: ["task"], hidden }, ]; @@ -331,7 +331,7 @@ describe("checks cannot change the work", () => { try { const before = treeHash(ws.dir); const listed = treeListing(ws.dir); - const check: Check = { name: "task:merged", kind: "command", run: MUTATOR, timeoutMs: 20_000, expectExit: 0, tags: ["task"], hidden: true }; + const check: Check = { name: "task:merged", kind: "command", run: MUTATOR, timeoutMs: 20_000, expectExit: 0, tags: ["task"], hidden: true, author: { kind: "judge", model: "judge-j" } }; const ctx = { cwd: ws.dir, record: [], ledger: [], archivedBatches: 0 } as unknown as BarContext; const r = await runCheck(check, ctx); assert.equal(r.ok, true, r.output); diff --git a/test/independent-checks.test.ts b/test/independent-checks.test.ts new file mode 100644 index 0000000..6a71efa --- /dev/null +++ b/test/independent-checks.test.ts @@ -0,0 +1,293 @@ +/** + * "verified" never rests on checks written by the model that did the work. + * + * Who wrote each check is recorded when it is sealed: the worker, a separate + * judge, a person, or the reference writer. A run is verified only when a + * passing check that ran the work and asserted a value came from someone + * other than the worker model (compared by provider-normalized id), or a + * person wrote or approved a passing check. Otherwise the best it earns is + * "passed own checks", reported unverified. + */ +import assert from "node:assert/strict"; +import { readFileSync, readdirSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; +import { describe, it } from "node:test"; +import { taskChecksFrom } from "../src/criteria.js"; +import { Engine } from "../src/engine.js"; +import { Receipts } from "../src/receipts.js"; +import { assertsValue, claimLabel, goldenOperands, independentOf, jobEndWords, normalizeModelId, sameModel, tierOf } from "../src/tiers.js"; +import { parseBar } from "../src/bar.js"; +import { proposeBar } from "../src/detect.js"; +import type { Check, CheckAuthor } from "../src/types.js"; +import { allowAll, drain, scriptedProvider, workspace, type ScriptedTurn } from "./helpers.js"; + +describe("model ids, provider spelling aside", () => { + it("the same model through another provider is the same model", () => { + assert.ok(sameModel("minimax/minimax-m3", "MiniMax-M3")); + assert.ok(sameModel("qwen/qwen3-235b-a22b:free", "qwen3-235b-a22b")); + assert.ok(sameModel("anthropic/claude-opus-4.1", "claude-opus-4-1-20250805")); + assert.ok(sameModel("us.anthropic.claude-3-5-sonnet-20240620-v1:0", "claude-3.5-sonnet")); + assert.ok(sameModel("openrouter/nvidia/nemotron-3-ultra-550b-a55b:free", "nemotron-3-ultra-550b-a55b")); + assert.equal(normalizeModelId("x-ai/Grok-4.6"), "grok-4-6"); + assert.ok(sameModel("qwen/qwen3-235b-a22b-2507", "Qwen/Qwen3-235B-A22B-Instruct-2507"), "-instruct is a spelling, not a model"); + assert.ok(sameModel("google/gemma-3-27b-it", "gemma-3-27b")); + assert.ok(sameModel("deepseek-chat", "deepseek")); + }); + it("different models stay different", () => { + assert.ok(!sameModel("deepseek-v3", "deepseek-v2"), "a version is not a date suffix"); + assert.ok(!sameModel("qwen3-235b", "qwen3-32b")); + assert.ok(!sameModel("minimax-m3", "minimax-m2.5")); + assert.ok(!sameModel("", ""), "an empty id matches nothing"); + }); +}); + +describe("independentOf", () => { + it("a person is independent; the worker never is; a model only when it is another model", () => { + assert.ok(independentOf({ kind: "person" }, "m")); + assert.ok(!independentOf({ kind: "worker", model: "m" }, "m")); + assert.ok(!independentOf({ kind: "worker", model: "other" }, "m"), "the worker kind is never independent"); + assert.ok(independentOf({ kind: "judge", model: "j" }, "m")); + assert.ok(!independentOf({ kind: "judge", model: "m" }, "m")); + assert.ok(!independentOf({ kind: "judge", model: "openrouter/m:free" }, ["m"])); + assert.ok(!independentOf({ kind: "judge", model: "j" }, ["m", "j"]), "the id the backend reported counts too"); + assert.ok(!independentOf({ kind: "judge" }, "m"), "a judge with no recorded model proves nothing"); + assert.ok(!independentOf(undefined, "m")); + assert.ok(!independentOf({ kind: "judge", model: "j" }, undefined), "no worker to compare with: not independent"); + assert.ok(independentOf({ kind: "reference", model: "r" }, "m")); + assert.ok(!independentOf({ kind: "reference", model: "m" }, "m")); + }); +}); + +describe("tierOf: who wrote the checks", () => { + const value = (name: string) => ({ name, ok: true, hidden: true as const, kind: "command" as const, tags: ["task", "value"] }); + const by = (a: CheckAuthor, name = "task:v") => new Map([[name, a]]); + + it("worker-authored checks only: not verified", () => { + const t = tierOf({ results: [value("task:v")], worker: "m", authors: by({ kind: "worker", model: "m" }) }); + assert.equal(t.tier, "passed-own-checks"); + assert.equal(t.basis, "own"); + assert.match(t.reason!, /written by the worker model \(m\)/); + assert.equal(claimLabel("unverified", t), "passed own checks (m), not verified"); + }); + + it("an author nobody recorded on a hidden check counts as the worker's", () => { + assert.equal(tierOf({ results: [value("task:v")], worker: "m" }).tier, "passed-own-checks"); + assert.equal(tierOf({ results: [value("task:v")] }).tier, "passed-own-checks"); + }); + + it("judge-authored: verified, and the label names the judge", () => { + const t = tierOf({ results: [value("task:v")], worker: "m", authors: by({ kind: "judge", model: "judge-j" }) }); + assert.deepEqual([t.tier, t.basis, t.by], ["verified", "independent", ["judge-j"]]); + assert.equal(claimLabel("verified", t), "verified (independent checks: judge-j)"); + }); + + it("same model id as judge: not independent", () => { + for (const judge of ["m", "openrouter/m:free", "M"]) { + const t = tierOf({ results: [value("task:v")], worker: "m", authors: by({ kind: "judge", model: judge }) }); + assert.equal(t.tier, "passed-own-checks", judge); + assert.equal(claimLabel("unverified", t), "passed own checks (m), not verified"); + } + }); + + it("person-approved: verified, labelled as your checks", () => { + // A visible check (done.yml, or criteria approved in the window) is a person's. + const visible = tierOf({ results: [{ name: "task:v", ok: true, kind: "command" as const, tags: ["task"] }], worker: "m" }); + assert.deepEqual([visible.tier, visible.basis], ["verified", "person"]); + assert.equal(claimLabel("verified", visible), "verified (your checks)"); + // So is a hidden one a person signed off (a mission contract). + const signed = tierOf({ results: [value("task:v")], worker: "m", authors: by({ kind: "person" }) }); + assert.equal(claimLabel("verified", signed), "verified (your checks)"); + }); + + it("one independent runs+value check is enough beside the worker's own", () => { + const authors = new Map([ + ["task:own", { kind: "worker", model: "m" }], + ["task:judged", { kind: "judge", model: "j" }], + ]); + const t = tierOf({ results: [value("task:own"), value("task:judged")], worker: "m", authors }); + assert.equal(claimLabel("verified", t), "verified (independent checks: j)"); + // ...but an independent check that asserted no value does not carry it. + const weak = tierOf({ + results: [value("task:own"), { name: "task:judged", ok: true, hidden: true, tags: ["task"] }], + worker: "m", + authors, + }); + assert.equal(weak.tier, "passed-own-checks"); + }); + + it("Maat's builtins and session checks are nobody's judgment of the task", () => { + // The default `maat init` bar: work-landed, record-intact, claims-grounded, ... (src/detect.ts). + const builtin = (name: string) => ({ name, ok: true, kind: "builtin" as const, tags: ["session"] }); + const own = by({ kind: "worker", model: "m" }, "task:out"); + const t = tierOf({ results: [builtin("work-landed"), value("task:out")], worker: ["m"], authors: own }); + assert.deepEqual([t.tier, t.basis], ["passed-own-checks", "own"], "a passing work-landed made a self-judged run 'verified (your checks)'"); + const surface = { name: "task:look", ok: true, hidden: true as const, kind: "command" as const, tags: ["task", "surface"] }; + assert.equal(tierOf({ results: [builtin("work-landed"), surface], worker: "m" }).tier, "passed-checks"); + for (const n of ["work-landed", "record-intact", "claims-grounded", "work-accounted", "spec-intact"]) { + assert.equal(tierOf({ results: [builtin(n)], worker: "m" }).tier, "passed-checks", `${n} alone is not verified`); + } + // A builtin without the session tag (work-complete: imports-tracked) is no different. + assert.equal(tierOf({ results: [{ name: "work-complete", ok: true, kind: "builtin" as const, tags: [] }], worker: "m" }).tier, "passed-checks"); + // A session-tagged command check is Maat's too. + assert.equal(tierOf({ results: [{ name: "s", ok: true, kind: "command" as const, tags: ["session"] }], worker: "m" }).tier, "passed-checks"); + // Even recorded as a person's, a builtin is not a person's judgment. + assert.equal(tierOf({ results: [builtin("work-landed")], worker: "m", authors: by({ kind: "person" }, "work-landed") }).tier, "passed-checks"); + // A result with no recorded kind fails closed. + assert.equal(tierOf({ results: [{ name: "x", ok: true, tags: [] }], worker: "m" }).tier, "passed-checks"); + // A person's own command still carries it. + const mine = tierOf({ results: [builtin("work-landed"), { name: "tests", ok: true, kind: "command" as const, tags: [] }], worker: "m" }); + assert.deepEqual([mine.tier, mine.basis], ["verified", "person"]); + }); + + it("review-advisory: a contradicted or missing review qualifies every label", () => { + const judged = by({ kind: "judge", model: "j" }); + const contradicted = tierOf({ results: [value("task:v")], worker: "m", authors: judged, review: { votes: "2/3", violations: [] }, reviewAdvisory: true }); + assert.deepEqual([contradicted.tier, contradicted.reviewGap], ["verified", "unconfirmed"]); + assert.equal(claimLabel("verified", contradicted), "verified (independent checks: j), unconfirmed"); + const missing = tierOf({ results: [value("task:v")], worker: "m", authors: judged, unreviewed: true, reviewAdvisory: true }); + assert.equal(claimLabel("verified", missing), "verified (independent checks: j), unreviewed"); + const fine = tierOf({ results: [value("task:v")], worker: "m", authors: judged, review: { votes: "0/3", violations: [] }, reviewAdvisory: true }); + assert.equal(claimLabel("verified", fine), "verified (independent checks: j)"); + const own = tierOf({ results: [value("task:v")], worker: "m", review: { votes: "2/3", violations: [] }, reviewAdvisory: true }); + assert.equal(claimLabel("unverified", own), "passed own checks (m), not verified, unconfirmed"); + }); + + it("the terminal and the window print the qualifier before any claim", () => { + const claim = "verified (independent checks: j)"; + // The engine's own job_end (the claim already carries it) ... + assert.equal(jobEndWords({ outcome: "verified", tier: "verified", claim: `${claim}, unconfirmed`, review: { confirmed: false } }), `${claim}, unconfirmed`); + // ... and a claim from a build that did not put it there. + assert.equal(jobEndWords({ outcome: "verified", tier: "verified", claim, review: { confirmed: false } }), `${claim}, unconfirmed`); + assert.equal(jobEndWords({ outcome: "verified", tier: "verified", claim, unreviewed: true }), `${claim}, unreviewed`); + assert.equal(jobEndWords({ outcome: "verified", tier: "verified", claim, review: { confirmed: true } }), `${claim}, independently reviewed`); + assert.equal(jobEndWords({ outcome: "verified", review: { confirmed: false } }), "passed its checks, unconfirmed"); + assert.equal(jobEndWords({ outcome: "verified", unreviewed: true }), "passed its checks, unreviewed"); + assert.equal(jobEndWords({ outcome: "unverified", tier: "passed-checks", tierReason: "r" }), "passed its checks (not verified: r)"); + }); + + it("a reviewer contradiction still outranks authorship", () => { + const t = tierOf({ results: [value("task:v")], worker: "m", authors: by({ kind: "judge", model: "j" }), review: { votes: "2/3", violations: [] } }); + assert.equal(t.tier, "passed-checks"); + assert.equal(claimLabel("unverified", t), "unverified"); + }); +}); + +describe("authorship at the seal, in a turn", () => { + const TASK = "Write out.txt containing exactly the word hello."; + const work: ScriptedTurn[] = [ + { calls: [{ name: "write_file", args: { path: "out.txt", content: "hello\n" } }] }, + { text: "Done." }, + ]; + const value = (extra: Partial = {}): Check => + ({ name: "made", kind: "command", run: "grep -qx hello out.txt", timeoutMs: 5_000, expectExit: 0, tags: ["task", "value"], hidden: true, ...extra }) as Check; + + async function run(checks: Check[], opts: { initBar?: boolean } = {}) { + const ws = workspace(); + try { + const provider = scriptedProvider(work); + const engine = new Engine({ + baseUrl: "http://provider.test/v1", model: "m", cwd: ws.dir, fetchFn: provider.fetchFn, + bar: opts.initBar ? parseBar(proposeBar(ws.dir).yaml) : null, + receipts: new Receipts(ws.dir), stream: false, autonomy: "high", + }); + const events = await drain(engine.run(TASK, allowAll, { taskChecks: checks })); + const end = events.find((e) => e.kind === "job_end"); + assert.ok(end && end.kind === "job_end"); + const dir = join(ws.dir, ".maat", "receipts"); + const rows = readFileSync(join(dir, "index.jsonl"), "utf8").trim().split("\n").map((l) => JSON.parse(l)); + const receipt = readdirSync(dir).filter((f) => f.endsWith(".md")).sort().map((f) => readFileSync(join(dir, f), "utf8")).join("\n"); + return { end, events, rows, receipt }; + } finally { + ws.cleanup(); + } + } + + it("checks the worker drafted: passed own checks, unverified, and the receipt says who wrote them", async () => { + const { end, events, rows, receipt } = await run([value()]); + assert.deepEqual([end.outcome, end.tier, end.claim], ["unverified", "passed-own-checks", "passed own checks (m), not verified"]); + assert.deepEqual(end.checkAuthors, { "task:made": "worker m" }); + assert.ok(events.some((e) => e.kind === "info" && /passed own checks \(m\), not verified/.test(e.text))); + assert.deepEqual([rows.at(-1).tier, rows.at(-1).claim], ["passed-own-checks", "passed own checks (m), not verified"]); + assert.match(receipt, /written by: the worker model m/); + assert.match(receipt, /Claim: passed own checks \(m\), not verified\./); + }); + + it("the default `maat init` bar beside the worker's own checks: still not verified", async () => { + const { end, receipt } = await run([value()], { initBar: true }); + assert.deepEqual([end.outcome, end.tier, end.claim], ["unverified", "passed-own-checks", "passed own checks (m), not verified"]); + assert.match(receipt, /check: work-landed[\s\S]*?written by: Maat \(a session check/); + assert.doesNotMatch(receipt, /check: work-landed\nkind: builtin\nwritten by: a person/); + }); + + it("checks a separate judge drafted: verified, naming the judge", async () => { + const { end, rows, receipt } = await run([value({ author: { kind: "judge", model: "judge-j" } })]); + assert.deepEqual([end.outcome, end.tier, end.claim], ["verified", "verified", "verified (independent checks: judge-j)"]); + assert.deepEqual(end.checkAuthors, { "task:made": "judge judge-j" }); + assert.equal(rows.at(-1).claim, "verified (independent checks: judge-j)"); + assert.match(receipt, /written by: the judge model judge-j/); + }); + + it("a judge that is the worker model under another provider's name: not verified", async () => { + const { end } = await run([value({ author: { kind: "judge", model: "openrouter/m:free" } })]); + assert.deepEqual([end.outcome, end.tier, end.claim], ["unverified", "passed-own-checks", "passed own checks (m), not verified"]); + }); + + it("checks a person approved (the window, --criterion): verified (your checks), unchanged", async () => { + const approved = taskChecksFrom({ checks: [{ name: "made", run: "test -s out.txt" }] }).taskChecks; + assert.deepEqual(approved[0]!.author, { kind: "person" }, "approved criteria are recorded as a person's"); + const { end, receipt } = await run(approved); + assert.deepEqual([end.outcome, end.tier, end.claim], ["verified", "verified", "verified (your checks)"]); + assert.match(receipt, /written by: a person \(your check\)/); + }); +}); + +describe("a golden file only stands for a value when it predates the work", () => { + it("goldenOperands and assertsValue with a pre-work record", () => { + assert.deepEqual(goldenOperands("diff out.txt expected.txt"), ["expected.txt"]); + assert.deepEqual(goldenOperands("cmp got.bin 'baseline.bin' && diff -u a golden/x.json"), ["baseline.bin", "golden/x.json"]); + assert.ok(assertsValue("diff out.txt expected.txt"), "by name alone, as the drafter tags it"); + assert.ok(!assertsValue("diff out.txt expected.txt", () => false), "not there before the work: no value"); + assert.ok(assertsValue("diff out.txt expected.txt", (g) => g === "expected.txt")); + assert.ok(assertsValue("diff out.txt expected.txt && [ \"$(wc -l < out.txt)\" -eq 3 ]", () => false), "a literal comparison still counts"); + assert.ok(assertsValue("diff out.txt <(printf 'a\\n')", () => false), "a heredoc or <(...) is written in the command"); + }); + + const TASK = "Write out.txt with the line hello."; + const diffCheck: Check = { + name: "matches", kind: "command", run: "diff out.txt expected.txt", timeoutMs: 5_000, expectExit: 0, + tags: ["task", "value"], hidden: true, author: { kind: "judge", model: "judge-j" }, + } as Check; + async function turn(writes: Record, before: Record = {}) { + const ws = workspace(); + try { + for (const [f, c] of Object.entries(before)) writeFileSync(join(ws.dir, f), c); + const provider = scriptedProvider([ + { calls: Object.entries(writes).map(([path, content]) => ({ name: "write_file", args: { path, content } })) }, + { text: "Done." }, + ]); + const engine = new Engine({ baseUrl: "http://provider.test/v1", model: "m", cwd: ws.dir, fetchFn: provider.fetchFn, bar: null, stream: false, autonomy: "high" }); + const events = await drain(engine.run(TASK, allowAll, { taskChecks: [diffCheck] })); + const end = events.find((e) => e.kind === "job_end"); + assert.ok(end && end.kind === "job_end"); + return end; + } finally { + ws.cleanup(); + } + } + + it("the worker writes both files: the diff passes and is not a value, so not verified", async () => { + const end = await turn({ "out.txt": "wrong\n", "expected.txt": "wrong\n" }); + assert.deepEqual([end.outcome, end.tier], ["unverified", "passed-checks"]); + }); + + it("the task shipped expected.txt and the worker left it alone: verified", async () => { + const end = await turn({ "out.txt": "hello\n" }, { "expected.txt": "hello\n" }); + assert.deepEqual([end.outcome, end.tier, end.claim], ["verified", "verified", "verified (independent checks: judge-j)"]); + }); + + it("the worker rewrote the shipped expected.txt: not verified", async () => { + const end = await turn({ "out.txt": "wrong\n", "expected.txt": "wrong\n" }, { "expected.txt": "hello\n" }); + assert.deepEqual([end.outcome, end.tier], ["unverified", "passed-checks"]); + }); +}); + diff --git a/test/judgment.test.ts b/test/judgment.test.ts index 02751e5..4c4d898 100644 --- a/test/judgment.test.ts +++ b/test/judgment.test.ts @@ -21,7 +21,7 @@ const check = (name: string, run: string, hidden: boolean): Check => ({ timeoutMs: 5_000, expectExit: 0, tags: ["task", "value"], - ...(hidden ? { hidden: true } : {}), + ...(hidden ? { hidden: true, author: { kind: "judge", model: "judge-j" } } : {}), }); const writesThenClaims = () => [ diff --git a/test/late-criteria.test.ts b/test/late-criteria.test.ts index 1713fb0..d232181 100644 --- a/test/late-criteria.test.ts +++ b/test/late-criteria.test.ts @@ -15,7 +15,7 @@ import type { Check } from "../src/types.js"; import { allowAll, drain, scriptedProvider, workspace } from "./helpers.js"; // Value-asserting checks (src/tiers.ts): a check that only looks cannot earn "verified". -const mk = (name: string, run: string): Check => ({ name, kind: "command", run, timeoutMs: 5_000, expectExit: 0, tags: ["task", "value"], hidden: true }); +const mk = (name: string, run: string): Check => ({ name, kind: "command", run, timeoutMs: 5_000, expectExit: 0, tags: ["task", "value"], hidden: true, author: { kind: "judge", model: "judge-j" } }); const later = (v: T, ms: number) => new Promise((r) => setTimeout(() => r(v), ms)); const nothing = async () => ({ taskChecks: [] as Check[], taskNotes: [] as string[] }); const TASK = "write hello to out.txt"; diff --git a/test/lazy-seal.test.ts b/test/lazy-seal.test.ts index 3a73406..75a0a16 100644 --- a/test/lazy-seal.test.ts +++ b/test/lazy-seal.test.ts @@ -9,7 +9,7 @@ import { changesSomething, criteriaWaitMs, Engine } from "../src/engine.js"; import type { Check, EngineEvent } from "../src/types.js"; import { allowAll, drain, scriptedProvider, workspace } from "./helpers.js"; -const check: Check = { name: "made", kind: "command", run: "grep -qx y out.txt", timeoutMs: 5_000, expectExit: 0, tags: ["task", "value"], hidden: true }; +const check: Check = { name: "made", kind: "command", run: "grep -qx y out.txt", timeoutMs: 5_000, expectExit: 0, tags: ["task", "value"], hidden: true, author: { kind: "judge", model: "judge-j" } }; const later = (v: T, ms: number) => new Promise((r) => setTimeout(() => r(v), ms)); const sealedAt = (ev: EngineEvent[]) => ev.findIndex((e) => e.kind === "info" && /sealed for this turn/.test(e.text)); const toolAt = (ev: EngineEvent[], name: string) => ev.findIndex((e) => e.kind === "tool" && e.name === name); diff --git a/test/mission.test.ts b/test/mission.test.ts index 8e49b10..e5e2c96 100644 --- a/test/mission.test.ts +++ b/test/mission.test.ts @@ -120,7 +120,7 @@ describe("the contract", () => { const c = parseContract(CONTRACT); const checks = assertionChecks(c, ["VAL-A-001", "VAL-UI-001", "nope"]); assert.deepEqual(checks, [ - { name: "VAL-A-001", kind: "command", run: "test -f a.txt", timeoutMs: 120_000, expectExit: 0, tags: ["mission"], hidden: true }, + { name: "VAL-A-001", kind: "command", run: "test -f a.txt", timeoutMs: 120_000, expectExit: 0, tags: ["mission"], hidden: true, author: { kind: "person" } }, ]); assert.deepEqual(assertionNotes(c, ["VAL-A-001", "VAL-UI-001"]), ["VAL-UI-001: a person judges the layout"]); }); diff --git a/test/record-scope.test.ts b/test/record-scope.test.ts index b276333..5e32413 100644 --- a/test/record-scope.test.ts +++ b/test/record-scope.test.ts @@ -56,7 +56,9 @@ describe("a completion claim is judged on the turn that made it", () => { maxProofAttempts: 1, }); const first = await drain(engine.run("add a file", allowAll)); - assert.equal((first.at(-1) as { outcome: string }).outcome, "verified"); + // The bar passed; a builtin alone never makes it "verified" (src/tiers.ts personCheck). + const firstEnd = first.at(-1) as { outcome: string; tier?: string }; + assert.deepEqual([firstEnd.outcome, firstEnd.tier], ["unverified", "passed-checks"]); const second = await drain(engine.run("refactor the auth module", allowAll)); const end = second.at(-1) as { kind: string; outcome: string }; diff --git a/test/reference.test.ts b/test/reference.test.ts index 5c9bf4c..4892321 100644 --- a/test/reference.test.ts +++ b/test/reference.test.ts @@ -13,7 +13,7 @@ import { existsSync, mkdirSync, writeFileSync } from "node:fs"; import { join } from "node:path"; import { describe, it } from "node:test"; import { Engine } from "../src/engine.js"; -import { draftReference, fencedPython, parseReferenceReply, REFERENCE_MAX_SNAPSHOT_FILES, snapshotProject } from "../src/reference.js"; +import { draftReference, fencedPython, parseReferenceReply, referenceComparedAll, REFERENCE_MAX_SNAPSHOT_FILES, snapshotProject } from "../src/reference.js"; import type { Check } from "../src/types.js"; import { allowAll, drain, scriptedProvider, workspace } from "./helpers.js"; @@ -87,6 +87,16 @@ describe("snapshotProject", () => { }); }); +describe("referenceComparedAll", () => { + it("needs the driver's last line to say every judged input was compared", () => { + assert.ok(referenceComparedAll("5 inputs match two independent references\nREFERENCE COMPARED 5/5\n")); + assert.ok(!referenceComparedAll(""), "an exit 0 with no output is not a pass"); + assert.ok(!referenceComparedAll("REFERENCE COMPARED 3/5")); + assert.ok(!referenceComparedAll("REFERENCE COMPARED 0/0")); + assert.ok(!referenceComparedAll("REFERENCE COMPARED 5/5\nREFERENCE COMPARED 2/5"), "the last one counts"); + }); +}); + describe("parsing the writers' replies", () => { it("reads the module from its own fenced block, braces and all", () => { const program = 'd = {"a": 1}\nprint("x\\ny")'; @@ -105,7 +115,7 @@ describe("draftReference", () => { const r = await draftReference({ ...ask, task: "write double.py", snapshot, fetchFn: p.fetchFn }); assert.ok(r.ok, r.ok ? "" : r.why); assert.equal(r.check.hidden, true); - for (const f of ["check.py", "second.py", "driver.py"]) assert.ok(existsSync(join(snapshot.dir, f))); + for (const f of ["check.py", "second.py", "driver.py", "runner.py"]) assert.ok(existsSync(join(snapshot.dir, f))); assert.equal(existsSync(join(w.dir, "check.py")), false, "nothing is written into the project"); // The second reviewer is told the input format and inputs — never the first's code or answers. assert.match(p.secondPrompts[0]!, /ONE INPUT IS: an integer n/); @@ -178,11 +188,12 @@ describe("draftReference", () => { }); describe("a reference check, in a turn", () => { - async function turn(content: string, opts: { second?: string; first?: string; waitMs?: number; extra?: Check[] } = {}) { + async function turn(content: string, opts: { second?: string; first?: string; waitMs?: number; extra?: Check[]; refModel?: string } = {}) { const w = workspace(); const snapshot = snapshotProject(w.dir)!; const p = writers([opts.first ?? fenced(moduleA())], [opts.second ?? secondModule("str(n + n)")]); - const ref = await draftReference({ ...ask, task: "write double.py", snapshot, fetchFn: p.fetchFn }); + // Written by a model other than the worker ("m"), unless a test says otherwise. + const ref = await draftReference({ ...ask, model: opts.refModel ?? "ref-model", task: "write double.py", snapshot, fetchFn: p.fetchFn }); assert.ok(ref.ok, ref.ok ? "" : ref.why); const provider = scriptedProvider([ { calls: [{ name: "write_file", args: { path: "double.py", content } }] }, @@ -202,9 +213,17 @@ describe("a reference check, in a turn", () => { ); w.cleanup(); const end = events.find((e) => e.kind === "job_end"); - return { events, outcome: end && end.kind === "job_end" ? end.outcome : undefined }; + return { events, outcome: end && end.kind === "job_end" ? end.outcome : undefined, end: end && end.kind === "job_end" ? end : undefined }; } + it("records who wrote it, and a reference by the worker model is not independent", async () => { + const other = await turn("import sys\nprint(2 * int(sys.argv[1]))\n"); + assert.equal(other.end?.checkAuthors?.["task:reference"], "reference ref-model"); + assert.equal(other.end?.claim, "verified (independent checks: ref-model)"); + const own = await turn("import sys\nprint(2 * int(sys.argv[1]))\n", { refModel: "m" }); + assert.deepEqual([own.outcome, own.end?.tier, own.end?.claim], ["unverified", "passed-own-checks", "passed own checks (m), not verified"]); + }); + it("joins at the claim and refuses work both references disagree with, showing the input", async () => { const { events, outcome } = await turn("import sys\nn = int(sys.argv[1])\nprint(n + n if n < 4 else n * 3)\n"); assert.ok(events.some((e) => e.kind === "info" && /independent reference check joined/.test(e.text))); @@ -225,7 +244,7 @@ describe("a reference check, in a turn", () => { it("is retired, not obeyed, when its own code fails at the claim; the other checks judge", async () => { const raises = " if __import__('os').path.exists('double.py'):\n raise KeyError('oops')\n return 'missing'"; - const made: Check = { name: "made", kind: "command", run: "[ \"$(python3 double.py)\" = \"1\" ]", timeoutMs: 5_000, expectExit: 0, tags: ["task", "value"], hidden: true }; + const made: Check = { name: "made", kind: "command", run: "[ \"$(python3 double.py)\" = \"1\" ]", timeoutMs: 5_000, expectExit: 0, tags: ["task", "value"], hidden: true, author: { kind: "judge", model: "judge-j" } }; const { events, outcome } = await turn("print(1)\n", { first: fenced(moduleA("str(2 * n)", raises)), extra: [made] }); assert.ok(events.some((e) => e.kind === "info" && /failed in its own code, not on the work/.test(e.text))); assert.equal(outcome, "verified", "judged by the check that remained"); @@ -238,6 +257,44 @@ describe("a reference check, in a turn", () => { assert.notEqual(outcome, "verified"); }); + // An import-style deliverable: run_deliverable imports the work's module. + const importing = [ + " try:", + " import importlib, os, sys", + " sys.path.insert(0, os.getcwd())", + " return str(importlib.import_module('double').f(n))", + " except Exception as e:", + " return 'failed: %s' % e", + ].join("\n"); + + it("a deliverable that ends the process on import (os._exit(0)) is not a pass", async () => { + // The driver used to import the work's module in its own process: `os._exit(0)` at top + // level ended it with exit 0 before any comparison, and the check passed on wrong work. + const { events, outcome } = await turn("import os\nos._exit(0)\ndef f(n):\n return n * 999\n", { first: fenced(moduleA("str(2 * n)", importing)) }); + assert.notEqual(outcome, "verified"); + const refused = events.find((e) => e.kind === "proof_refused" || e.kind === "proof_exhausted"); + assert.match(JSON.stringify(refused), /the deliverable ended the process/); + }); + + it("a deliverable that calls sys.exit(0) on import, or exits during the call, is not a pass", async () => { + for (const content of ["import sys\nsys.exit(0)\n", "import os\ndef f(n):\n os._exit(0)\n"]) { + const { outcome } = await turn(content, { first: fenced(moduleA("str(2 * n)", importing)) }); + assert.notEqual(outcome, "verified", content); + } + }); + + it("an import-style deliverable that is right still verifies, judged out of process", async () => { + const { outcome } = await turn("def f(n):\n return 2 * n\n", { first: fenced(moduleA("str(2 * n)", importing)) }); + assert.equal(outcome, "verified"); + }); + + it("the work's code cannot call reference() through the module it was handed", async () => { + // A deliverable that reaches into the check module for the expected value gets nothing. + const cheat = "import sys\ndef f(n):\n m = [v for k, v in sys.modules.items() if hasattr(v, 'reference')]\n return int(m[0].reference(n)) if m else -1\n"; + const { outcome } = await turn(cheat, { first: fenced(moduleA("str(2 * n)", importing)) }); + assert.notEqual(outcome, "verified"); + }); + it("is not waited for past its time: the claim is judged without it", async () => { const { events } = await turn("import sys\nprint(0)\n", { waitMs: 0 }); assert.ok(events.some((e) => e.kind === "info" && /still being written; judging this claim without it/.test(e.text))); diff --git a/test/reveal-default.test.ts b/test/reveal-default.test.ts index 09a9ff2..a515a37 100644 --- a/test/reveal-default.test.ts +++ b/test/reveal-default.test.ts @@ -20,8 +20,8 @@ const TASK = "Write out.txt containing a greeting. Valid for 365 days."; const TASK_WITH_INPUT = "Read data.csv and write out.txt containing a greeting. Valid for 365 days."; const hiddenChecks: Check[] = [ - { name: "made", kind: "command", run: "test -f out.txt", timeoutMs: 5_000, expectExit: 0, tags: ["task"], hidden: true }, - { name: "too-strict", kind: "command", run: "grep -q 366 out.txt", timeoutMs: 5_000, expectExit: 0, tags: ["task", "value"], hidden: true }, + { name: "made", kind: "command", run: "test -f out.txt", timeoutMs: 5_000, expectExit: 0, tags: ["task"], hidden: true, author: { kind: "judge", model: "judge-j" } }, + { name: "too-strict", kind: "command", run: "grep -q 366 out.txt", timeoutMs: 5_000, expectExit: 0, tags: ["task", "value"], hidden: true, author: { kind: "judge", model: "judge-j" } }, ]; const personChecks: Check[] = hiddenChecks.map((c) => ({ ...c, hidden: false })); diff --git a/test/review-advisory.test.ts b/test/review-advisory.test.ts index f1e4202..dfa0fa1 100644 --- a/test/review-advisory.test.ts +++ b/test/review-advisory.test.ts @@ -12,34 +12,36 @@ import { tierOf } from "../src/tiers.js"; describe("tierOf, review advisory", () => { const ck = (name: string, tags: string[]) => ({ name, ok: true, hidden: true as const, tags }); const strong = ck("task:counts", ["task", "value"]); + // Drafted by a separate judge; tierOf's authorship rule has its own tests (independent-checks.test.ts). + const J = { worker: "worker-m", authors: new Map(["task:counts", "task:other", "task:a"].map((n) => [n, { kind: "judge" as const, model: "judge-j" }])) }; it("the review is recorded and does not gate", () => { const review = { votes: "2/3", violations: [] }; - assert.equal(tierOf({ results: [strong], review }).tier, "passed-checks", "default: a contradiction demotes"); - const r = tierOf({ results: [strong], review, reviewAdvisory: true }); + assert.equal(tierOf({ ...J, results: [strong], review }).tier, "passed-checks", "default: a contradiction demotes"); + const r = tierOf({ ...J, results: [strong], review, reviewAdvisory: true }); assert.equal(r.tier, "verified"); assert.match(r.reviewNote!, /2\/3 contradicting/); - assert.equal(tierOf({ results: [strong], unreviewed: true, reviewAdvisory: true }).tier, "verified"); - assert.equal(tierOf({ results: [strong], unreviewed: true }).tier, "passed-checks", "default unchanged"); + assert.equal(tierOf({ ...J, results: [strong], unreviewed: true, reviewAdvisory: true }).tier, "verified"); + assert.equal(tierOf({ ...J, results: [strong], unreviewed: true }).tier, "passed-checks", "default unchanged"); }); it("needs a value check that did not already pass before the work", () => { const guards = new Set(["task:counts"]); - assert.equal(tierOf({ results: [strong], guards }).tier, "verified", "off by default: guards are ignored"); - const r = tierOf({ results: [strong], guards, reviewAdvisory: true }); + assert.equal(tierOf({ ...J, results: [strong], guards }).tier, "verified", "off by default: guards are ignored"); + const r = tierOf({ ...J, results: [strong], guards, reviewAdvisory: true }); assert.equal(r.tier, "passed-checks"); assert.match(r.reason!, /also passed before the work/); - const two = tierOf({ results: [strong, ck("task:other", ["task", "value"])], guards, reviewAdvisory: true }); + const two = tierOf({ ...J, results: [strong, ck("task:other", ["task", "value"])], guards, reviewAdvisory: true }); assert.equal(two.tier, "verified", "another value check that failed on the pristine tree carries it"); }); it("a surface-only or value-less pass is still not verified", () => { - assert.equal(tierOf({ results: [ck("task:a", ["task"])], reviewAdvisory: true }).tier, "passed-checks"); - assert.equal(tierOf({ results: [ck("task:a", ["task", "surface", "value"])], reviewAdvisory: true }).tier, "passed-checks"); + assert.equal(tierOf({ ...J, results: [ck("task:a", ["task"])], reviewAdvisory: true }).tier, "passed-checks"); + assert.equal(tierOf({ ...J, results: [ck("task:a", ["task", "surface", "value"])], reviewAdvisory: true }).tier, "passed-checks"); }); it("a person's passing check still verifies", () => { - assert.equal(tierOf({ results: [{ ok: true, tags: [] }], reviewAdvisory: true, review: { votes: "3/3", violations: [] } }).tier, "verified"); + assert.equal(tierOf({ ...J, results: [{ ok: true, kind: "command" as const, tags: [] }], reviewAdvisory: true, review: { votes: "3/3", violations: [] } }).tier, "verified"); }); }); diff --git a/test/sharper-checks.test.ts b/test/sharper-checks.test.ts index 93634e2..4840297 100644 --- a/test/sharper-checks.test.ts +++ b/test/sharper-checks.test.ts @@ -67,8 +67,8 @@ describe("a disputed check, in a turn", () => { // An arbiter other than the worker: a same-model arbiter is skipped. const ARBITER = { model: "arbiter" }; const checks: Check[] = [ - { name: "made", kind: "command", run: "head -c 5 out.txt | grep -qx hello", timeoutMs: 5_000, expectExit: 0, tags: ["task", "value"], hidden: true }, - { name: "too-strict", kind: "command", run: "grep -q 366 out.txt", timeoutMs: 5_000, expectExit: 0, tags: ["task"], hidden: true }, + { name: "made", kind: "command", run: "head -c 5 out.txt | grep -qx hello", timeoutMs: 5_000, expectExit: 0, tags: ["task", "value"], hidden: true, author: { kind: "judge", model: "judge-j" } }, + { name: "too-strict", kind: "command", run: "grep -q 366 out.txt", timeoutMs: 5_000, expectExit: 0, tags: ["task"], hidden: true, author: { kind: "judge", model: "judge-j" } }, ]; it("an upheld dispute retires the check; the rest still judge the claim", async () => { @@ -284,7 +284,7 @@ describe("what the checks leave behind", () => { else process.env.MAAT_CHECK_COPY = was; } }; - const leaves: Check = { name: "builds", kind: "command", run: "mkdir -p bin && touch bin/cmain made.o && test -f keep.txt", timeoutMs: 5_000, expectExit: 0, tags: ["task"], hidden: true }; + const leaves: Check = { name: "builds", kind: "command", run: "mkdir -p bin && touch bin/cmain made.o && test -f keep.txt", timeoutMs: 5_000, expectExit: 0, tags: ["task"], hidden: true, author: { kind: "judge", model: "judge-j" } }; const turn = () => [{ calls: [{ name: "write_file", args: { path: "keep.txt", content: "k\n" } }] }, { text: "Done." }]; it("is removed in an unattended run, and nothing that existed before is touched", async () => { @@ -601,7 +601,7 @@ describe("a drafted check that fails in its own code", () => { const ws = workspace(); try { const checks: Check[] = [ - { name: "made", kind: "command", run: "head -c 5 out.txt | grep -qx hello", timeoutMs: 5_000, expectExit: 0, tags: ["task", "value"], hidden: true }, + { name: "made", kind: "command", run: "head -c 5 out.txt | grep -qx hello", timeoutMs: 5_000, expectExit: 0, tags: ["task", "value"], hidden: true, author: { kind: "judge", model: "judge-j" } }, // Fails on the missing file before the work, so preflight keeps it; its own bug shows only after. { name: "broken-after", kind: "command", run: "test -f out.txt && python3 -c \"print(f'x {1')\"", timeoutMs: 5_000, expectExit: 0, tags: ["task"], hidden: true }, ]; diff --git a/test/tiers.test.ts b/test/tiers.test.ts index ebba153..72070ed 100644 --- a/test/tiers.test.ts +++ b/test/tiers.test.ts @@ -76,20 +76,22 @@ describe("the value tag", () => { }); describe("tierOf", () => { - const ok = (tags: string[], hidden = true) => ({ ok: true, hidden: hidden || undefined, tags }); + const ok = (tags: string[], hidden = true) => ({ name: "c", ok: true, hidden: hidden || undefined, kind: "command" as const, tags }); + // Drafted by a separate judge model: the authorship rule is pinned in independent-checks.test.ts. + const J = { worker: "worker-m", authors: new Map([["c", { kind: "judge" as const, model: "judge-j" }]]) }; it("needs a passing check that runs the work and asserts a value", () => { - assert.equal(tierOf({ results: [ok(["task", "value"])] }).tier, "verified"); - assert.equal(tierOf({ results: [ok(["task"]), ok(["task", "value"])] }).tier, "verified"); - assert.equal(tierOf({ results: [ok(["task"])] }).tier, "passed-checks"); - assert.equal(tierOf({ results: [ok(["task", "surface", "value"])] }).tier, "passed-checks", "a check that only looks cannot carry it"); - assert.equal(tierOf({ results: [{ ok: false, hidden: true, tags: ["task", "value"] }] }).tier, "passed-checks", "a failing check is no evidence"); + assert.equal(tierOf({ ...J, results: [ok(["task", "value"])] }).tier, "verified"); + assert.equal(tierOf({ ...J, results: [ok(["task"]), ok(["task", "value"])] }).tier, "verified"); + assert.equal(tierOf({ ...J, results: [ok(["task"])] }).tier, "passed-checks"); + assert.equal(tierOf({ ...J, results: [ok(["task", "surface", "value"])] }).tier, "passed-checks", "a check that only looks cannot carry it"); + assert.equal(tierOf({ ...J, results: [{ ok: false, hidden: true, tags: ["task", "value"] }] }).tier, "passed-checks", "a failing check is no evidence"); }); it("lets a person's passing check verify, and a contradiction undo any of it", () => { - assert.equal(tierOf({ results: [ok([], false)] }).tier, "verified"); - const r = tierOf({ results: [ok(["task", "value"])], review: { votes: "1/3", violations: [] } }); + assert.equal(tierOf({ ...J, results: [{ ...ok([], false), name: "project" }] }).tier, "verified"); + const r = tierOf({ ...J, results: [ok(["task", "value"])], review: { votes: "1/3", violations: [] } }); assert.equal(r.tier, "passed-checks"); assert.match(r.reason!, /1\/3/); - assert.equal(tierOf({ results: [ok(["task", "value"])], review: { votes: "0/3", violations: [] } }).tier, "verified"); + assert.equal(tierOf({ ...J, results: [ok(["task", "value"])], review: { votes: "0/3", violations: [] } }).tier, "verified"); }); it("says it in one phrase", () => { assert.equal(passedChecksWords("no check asserted an expected value"), "passed its checks (not verified: no check asserted an expected value)"); @@ -99,7 +101,11 @@ describe("tierOf", () => { describe("the tier in a turn", () => { const TASK = "Write out.txt containing exactly the word hello."; const make = (tags: string[], run: string, hidden = true): Check => - ({ name: "made", kind: "command", run, timeoutMs: 5_000, expectExit: 0, tags, ...(hidden ? { hidden: true } : {}) }) as Check; + ({ + name: "made", kind: "command", run, timeoutMs: 5_000, expectExit: 0, tags, + // Hidden ones as a separate judge model (not the worker "m") drafted them. + ...(hidden ? { hidden: true, author: { kind: "judge", model: "judge-j" } } : {}), + }) as Check; const SURFACE = make(["task", "surface"], "test -f out.txt"); const RUNS = make(["task"], "test -s out.txt"); const VALUE = make(["task", "value"], `grep -qx hello out.txt`); diff --git a/ui/app.ts b/ui/app.ts index 2b0c643..dbb764d 100644 --- a/ui/app.ts +++ b/ui/app.ts @@ -16,6 +16,7 @@ import { renderMarkdown } from "./markdown.js"; import { nextWaitWord } from "./wait-words.js"; import { playMaatSplash } from "./maat-splash.js"; import { fmtCost, spendLine, stepDid } from "../src/format.js"; +import { jobEndWords, type JobEndWords } from "../src/tiers.js"; import { expandEndpointShorthand, typedEndpointProblem } from "../src/endpoint.js"; import { matchCommands } from "../src/commands.js"; import { JOURNAL_RENDER_CAP, STREAM_CAP, contextCap, contextFill, newest, trimOldest } from "./bounds.js"; @@ -1139,17 +1140,7 @@ molt.onEvent((ev) => { case "job_end": { // The same words the terminal uses for the verdict (cli.tsx). - const review = ev.review as { confirmed?: boolean } | undefined; - const said = - ev.outcome === "verified" && review && review.confirmed === false - ? "passed its checks, unconfirmed" - : ev.outcome === "verified" && review?.confirmed - ? "verified, independently reviewed" - : ev.outcome === "verified" && ev.selfChecked - ? "passed its own checks" - : ev.outcome === "unverified" && Array.isArray(ev.checksDisagree) && ev.checksDisagree.length - ? "unverified, its own drafted checks disagree" - : String(ev.outcome); + const said = jobEndWords(ev as JobEndWords); say( "", `job ${said} · ${ev.steps} step(s) · ${spendLine(ev.spend)} · ${fmtMs(Number(ev.durationMs) || 0)}`,