[solvyxtech]

Why

On the 2026-10-07 mmhd lane (MiniMax M3 worker, Haiku 5.5 judge, #43 build), cron-next was labelled verified and the grader failed it. For */15 * * * * from 2025-01-01 00:07 the work printed 00:45, 01:00, 01:45 instead of 00:15, 00:30, 00:45. The judge's four checks all tested properties: 8 lines, each minute %15 == 0, sorted, unique; "every date is the 13th or a Friday"; two runs agree; "starts with 2024-06-1". Wrong output with the right shape passes every one of them.

What

• Tier rule (src/tiers.ts): assertsExact tags a check exact, next to value. tierOf grants "verified" only on a passing independent exact check. Independent value checks that all test properties now get passed-checks with the reason "passed checks that test properties only".

  • Exact: equality with a literal or with a table of literal cases; diff against literal text or a pre-work golden file; grep -x of a literal line; is None on literal input; a specific non-zero exit or HTTP status; a worked-out value found in a program's output on literal input; an oracle built from the input files.
  • Property: counts and sizes, types, modulo, prefixes and slices, membership in a file, format regexes, the same constant checked across every item (all(), find -exec, loops), two unknowns compared, checks that cannot fail, and regex patterns tried on the check's own lines.

• Post-work audit: the new V-property-only gate runs after V-no-value.

• Prompts (src/criteria.ts drafter and critic, src/post-audit.ts audit):

  • Each functional requirement gets at least one check that compares the output on a concrete input with an EXACT expected output worked out from the task text.
  • Each check carries a new expect field that quotes or derives that expected output.
  • The drafter is asked for edge and near-miss inputs: boundaries, empty input, 1.2.3.4.5 for an IP rule, both sides of an either/or rule.
  • Removed: the old line telling drafters to test properties instead of working values out.
  • The critic sees expect and is told that a derivation is not a guess.

• Tests: test/exact-values.test.ts covers:

  • the cron-next checks verbatim: property only, so not verified;
  • positive and negative exact examples;
  • the audit gate;
  • the prompt builders.

  Test fixtures that expect "verified" now carry the exact tag.

Replay gate (bench/local/exact_replay.py)

Lanes last written 2026-10-07, with regraded verdicts where present. A = every recorded verified. B = the verifieds the base rule still grants (an independent author recorded on the receipt). B is the gate.

	A right	A wrong	B right	B wrong
recorded verified	113	8	79	2
--literal-only kept	53 (47%)	0	48 (61%)	0
shipped rule kept	61 (54%)	0	56 (71%)	0

• The literal-only rule failed the two-thirds bar. It was narrowed by two extra forms (containment of a worked-out value in a program's output; an oracle), and the narrowed rule passes in B.
• In A, 11 right runs rest only on commands that also carried a wrong run (identical find -exec, || echo fail and print(len(header) == 5) checks). No command-level rule can keep those.
• Every grader-wrong verified is removed in both populations.

npm run check passes with ~/.local/bin off PATH: 2179 tests.

[solvyxtech]

Verdict: changes requested (3 blocking, 4 non-blocking). Review only, nothing merged. Posted as a comment review because GitHub won't let this account request changes on its own PR.

The direction is right. The cron-next miss is exactly the case a value tag can't catch, and gating "verified" on an exact comparison is the right fix. Running the replay over both populations, with --literal-only as a baseline, is the right way to decide it. The branch goes claude/exact-values into claude/value-classifier, there's no subscription backend and nothing touches fine-tune data. One pairing problem is listed as blocking 3 below.

I tested against c5dd6d4. npm run typecheck is clean, and the full suite passes 2179/2179 with ~/.local/bin off PATH. The runner still exited 1 afterwards because of the same coverage-reporter error on this box (Cannot read properties of undefined (reading 'line')), which is a local issue and not this PR. CI did not run: typecheck-and-test wasn't started because of the GitHub Actions billing or spending-limit block on the account, so the red check isn't a code failure.

Every probe below was run against the built assertsValue / assertsExact / evidenceTags at c5dd6d4.

Blocking

1. The containment form (rule 5) accepts an expected value that is already in the literal input, so a program that echoes its input or crashes on it earns "exact". Confidence: high (reproduced on the function and with a real script).

• Where: src/tiers.ts:1033 (containsValue), the literal-input test at src/tiers.ts:1050.
• Scenario: python3 tool.py '2024-06-13' 2>&1 | grep -q '2024-06-13' is tagged ["task","value","exact"]. So are node cli.js 'abc123' 2>&1 | grep -q abc123 and python3 nextrun.py '0 9 * * *' '2023-01-01 09:00' 1 | grep -q '2023-01-01 09:00'. Before the work, tool.py is missing, the error doesn't contain the date, and the check fails, so it counts as discriminating. A worker that writes datetime.strptime(sys.argv[1], "%Y/%m/%d") then crashes with ValueError: time data '2024-06-13' does not match format, grep finds the date, and the run becomes "verified". I ran exactly this: before=1, after=0. A program that just prints its arguments passes too.
• Why it matters: the base rule already tags this check value, so it isn't new exposure compared with the parent branch. But exact is meant to be the stricter gate, and this form was added to get over the two-thirds bar. As written, a crash passes the gate that's supposed to require a worked-out value.
• Fix: refuse when the pattern body (or any 4+ character run of it containing a digit) appears in the upstream stage's literal arguments or redirect target. Also refuse when stderr is merged (2>&1, |&), since a traceback or usage error is never the expected output. Add the three probes above to the property list in test/exact-values.test.ts.

2. A name bound to one literal gets past the rule that a constant compared across every item is a property. Because value and exact are judged over the whole command, a count check plus that loop gets both tags. Confidence: high (reproduced).

• Where: src/tiers.ts:924 (if (quantified(run, at) && !names.has(expected.trim())) continue;) together with exactNames at src/tiers.ts:673, which puts scalar constants (ok = True) in the same set as loop variables over a case table. Also src/tiers.ts:1071, where the two tags are computed independently.
• Scenario: out=cron.next('*/15 * * * *','2025-01-01 00:07',8); assert len(out) == 8; ok=True; assert all(cron.valid(t) == ok for t in out) is tagged value + exact. len(out) == 8 supplies value (the base rule), and all(... == ok ...) supplies exact, because ok is in names. Spelled with a literal (== True), the same all() is correctly refused. That is cron-next again: a count plus "every item passes the validator" holds for output that skips every other occurrence.
• Why it matters: the docstring says "the same constant compared across every item ... is still a property", and the only thing that changes the answer here is giving the constant a name.
• Fix: in exactNames, keep loop variables that come from unpacking a literal table separately from names bound to one scalar or string. At line 924, exempt only the loop-variable kind (the for inp, want in CASES shape). Longer term, require the value and the exact verdict to come from the same assertion, not from anywhere in the command.

3. A competitor name and model ID in the test file. Confidence: high on the rule.

• Where: test/exact-values.test.ts:5 ("MiniMax M3 worker, Haiku 5.5 judge") and test/exact-values.test.ts:97 (model: "claude-haiku-5-5"). Line 98 also has "minimax-m3".
• Why it matters: Tyler's decision on #33 was that no competitor name appears anywhere in the repo, not just in product copy, and the base branch currently has none in src/ or test/. This PR adds the first ones. The bench helper's --base default, origin/claude/value-classifier, is a branch ref and is fine.
• Fix: use generic IDs in the fixture (judge-small, worker-a) and describe the lane generically in the comment ("a small judge model on the 2026-10-07 lane"). The PR body and commit message say the same thing. The commit message can't change without a history rewrite, which is Tyler's call, but the body can be edited.

Non-blocking

4. One exact check whose expected value is trivial can carry "verified" on a stub. Confidence: medium (reproduced on the function. How often it happens is unmeasured).

• Where: exactEquality / statusExpected (src/tiers.ts:714). Nothing weighs which literal is expected.
• Scenario: assert tool.parse('') == [], assert ip.valid('1.2.3.4.5') == False and python3 tool.py --bad; [ $? -eq 1 ] are all value + exact. def parse(s): return [], def valid(a): return False and sys.exit(1) pass them. Each fails before the work, because the module is missing, so each counts as discriminating. The new drafter prompt asks specifically for empty input and near-miss inputs (src/criteria.ts), so a lane where the only exact check is the edge case becomes more likely.
• Fix: don't let a check whose only expected literal is empty, zero, False or None, or a non-zero exit, be the only exact check behind "verified". Require at least one exact check with a non-trivial expected value alongside it.

5. The prompt reversal isn't measured by the replay. Confidence: medium.

• Where: src/criteria.ts removes "Never write an expected value you worked out yourself ... Your own arithmetic is where checks go wrong". src/criteria.ts:868 tells the critic that a derivation in expect "is not a guess".
• Scenario: the replay reclassifies checks drafted under the OLD prompt, so it shows the tag is safe. It says nothing about how often the drafter now writes a wrong worked-out value. A wrong derivation fails correct work, which gives a false refusal or A-fails-on-work, and the critic has now been told to let derivations through. The old rule was there because judge arithmetic went wrong.
• Suggestion: before this goes to main, rerun one lane with the new prompts and report how many grader-right runs are refused because a sealed check's expected value was wrong. Consider having the critic re-derive any expect with arithmetic in it, rather than accepting it as given.

6. The oracle form (rule 6) treats any file the check opens as the task's input, including files the work wrote. Confidence: medium (reproduced on assertsExact. It only matters when another clause in the same command supplies value).

• Where: READS_INPUT at src/tiers.ts:933, used by oracleEquality at src/tiers.ts:944.
• Scenario: rows=json.load(open("out.json")); exp=[...rows]; assert open("out.txt").read().splitlines()==exp and exp=open("out.txt").read().splitlines(); assert open("out.txt").read().splitlines()==exp both return assertsExact true. That is two of the work's own outputs, or one output compared with itself, which is the "two unknowns" property. Separately, the PR's own positive oracle example in test/exact-values.test.ts:51 returns assertsValue false, so evidenceTags never tags it exact and the audit drops it at V-no-value first. The test passes only because it calls assertsExact directly.
• Fix: require that the opened file existed before the work with the same digest (the goldensBefore / valueUnproven mechanism already does this for expected files), or drop the form if the replay shows it isn't needed in population B. Add an evidenceTags assertion for each positive example so a dead form shows up.

7. Small things.

• tierOf in advisory mode: the new property-only passed-checks return drops ...gap, unlike the passed-untested and passed-own-checks returns next to it. That matches the existing fallback, but it means an "unconfirmed" review isn't recorded on a property-only run. Low.
• Sealed criteria carry tags from draft time, so a session sealed before this lands has no exact tag and can't earn "verified" afterwards, and its reason will say "tests properties only" even for assert f('x') == 'y'. Recompute exact from run at tier time, or note it in the CHANGELOG. Low-medium.
• The positive example bash -c 'git rev-parse --abbrev-ref HEAD | grep -qx master' (test line 49) counts as exact. That's reasonable for a branch task, but it shows that rule 3 doesn't check whether the deliverable ran at all. That's worth a sentence in the docstring.

Tests I'd add

• The three containment probes in finding 1, plus a 2>&1 traceback case, as property.
• The named-constant all() / for cases in finding 2, with and without a len() value clause, as property. Keep for inp, want in CASES exact.
• evidenceTags (not just assertsExact) for every positive example.
• A tierOf case where the only independent exact check expects [] or False.

[solvyxtech]

Live A/B (Haiku 5.5 worker, Sonnet 5.5 judge, 20 tasks, same bench): baseline build (#44+#45) 20/20 pass, 12 verified (60%), 0 wrong; this PR (with the cut-off retry fix, 87d…) 20/20 pass, 7 verified (35%), 0 wrong. The exact rule plus the longer drafting prompt cost recall with no precision gain on this sample. Decision: keep the exact rule opt-in (it still removes the cron-next class in replay), keep the cut-off retry in ask.ts as a general fix; do not make the rule default. Next: confirm the 60% baseline at charter scale.

[solvyxtech]

Verdict: CHANGES REQUESTED

Prior items: 0 of 3 blockers fixed, and non-blocking items 4–7 are still open. The only code change since my last review (5450990992, at c5dd6d4) is f213246, which retries a cut-off reply in src/ask.ts. src/tiers.ts, src/criteria.ts, src/post-audit.ts and test/exact-values.test.ts are byte-identical to what I reviewed. This pass adds one new blocker and two new non-blocking items.

Re-review from Grok. Comment only, nothing merged. Tested at f213246: npm run typecheck is clean. The full suite passed 2179 of 2180. The one failure was acp-deadline "ends the job on time…", which timed out because I was running three suites at once on this box. Run on its own, that file passes 13 of 13. CI did not run because of the account-wide GitHub Actions billing or spending-limit block, so a red or missing check here isn't a code finding.

Blocking

1. NEW: The code makes the exact rule the default, but the PR's own decision says it should be opt-in. Confidence: high (code read; there's no flag anywhere).

• Where: src/tiers.ts:1413-1433. exactIndependent is always required for earned, and nothing in tiers.ts, engine.ts or post-audit.ts checks an env var or option. The audit gate at src/post-audit.ts:519 (V-property-only) and the drafting-prompt rewrite at src/criteria.ts:536-547 are also unconditional.
• Scenario: your A/B comment from 9:13 PM PT (20 tasks, same bench) found that the baseline (#44+#45) verified 12 of 20 with 0 wrong, and this PR verified 7 of 20 with 0 wrong. It concludes: "keep the exact rule opt-in … do not make the rule default." Merging this head does the opposite. Every run drops to the A/B's 35% verified rate, with no precision gain measured on that sample, one day before an npm publish.
• Why it matters: the change and the decision recorded on the PR disagree. Anyone who merges based on the comment ships the measured recall loss.
• Fix: put the exact requirement (tier, audit gate, and the drafting prompt that goes with it) behind an explicit option such as --require-exact or MAAT_REQUIRE_EXACT=1, defaulting to off. Add a tierOf test for each setting. Or, if the plan has changed, update the PR body and say so. If the cut-off retry is the part you want to land now, it could go in its own small PR.

2. STILL OPEN: The containment form (rule 5) passes a crash whose error message repeats the input. Confidence: high (re-checked at this head). src/tiers.ts:1033 (containsValue). python3 tool.py '2024-06-13' 2>&1 | grep -q '2024-06-13' is still tagged ["task","value","exact"]. Details and the fix are in the previous review.

3. STILL OPEN: A named constant gets past the same-constant-across-items rule. Confidence: high (re-checked). src/tiers.ts:924 and exactNames at :673. The len(out) == 8; ok=True; assert all(cron.valid(t) == ok for t in out) probe is still tagged value + exact. That's the cron-next false "verified" this PR exists to remove.

4. STILL OPEN: Competitor names are in the repo. Confidence: high on the rule. They're at test/exact-values.test.ts:5 ("MiniMax M3 worker, Haiku 5.5 judge"), :97 (claude-haiku-5-5) and :98 (minimax-m3). The fix is still generic IDs. The commit message f7d5bb2 names them too, and changing that means rewriting history, which is Tyler's call. The A/B comment on this PR also names the models. That isn't repo content, but please keep it out of any squash message.

Non-blocking

5. NEW: The cut-off retry multiplies the caller's own ceiling with no cap, and if the retry fails, the clear "cut off" result is lost. Confidence: high on the mechanism (reproduced with a scripted provider), medium on how often it happens.

• Where: src/ask.ts:119-120.
• Scenario A (hosted): writeReference asks with REFERENCE_MAX_TOKENS = 16_000 (src/reference.ts:434). A cut-off reply now triggers a second request with max_tokens: 64000, and providers whose output cap is under that answer 400. With a scripted provider I got ceilings [16000, 64000] and the result {ok:false, error:"HTTP 400 writing a reference check"}. Before this change, the caller got cutOff: true and reported "cut off at the token limit". The mission planner works the same way, going from 6000 to 24000 (src/mission.ts:866).
• Scenario B (self-hosted): askTimeoutMs at default local speeds gives a 64,000-token retry an allowance of about 220 minutes, and a 24,000-token one about 87 minutes. The retry holds the server's only turn (takeTurn) the whole time, so with no deadlineAt (interactive drafting) every other ask to that box waits behind it. Before this change, a reply that had text in it was never retried.
• Fix: cap the retry ceiling (for example Math.min(bigger, ASK_MAX_TOKENS * ASK_RETRY_FACTOR), or skip the retry when the caller passed its own maxTokens). If the second ask fails, return the first cut-off result so the caller's "cut off at the token limit" message survives. Add a test for "second ask returns 400" alongside the two new ones.

6. NEW (exists on the base too, but this PR's gate is supposed to be the strict one): Python assertions that can never fail are tagged exact. Confidence: high (reproduced with real python3).

• Where: assertsExact in src/tiers.ts, and cannotFail in src/checklint.ts:403, which misses all four forms below.
• Each of these gets ["task","value","exact"] from evidenceTags, and cannotFail, swallowsExit and runnerPipedAway all return null:
  • python3 -c "import tool; assert (tool.f('a') == 'x', 'msg')". A non-empty tuple is always truthy, and Python itself warns "assertion is always true".
  • python3 -c "import tool; assert tool.f('a') == 'x' or True"
  • python3 -c "import tool; assert tool.f('a') == 'x' if False else True"
  • python3 -O -c "import tool; assert tool.f('a') == 'x'" (and PYTHONOPTIMIZE=1), where the assert is stripped.
• Scenario: before the work, import tool fails, so each check exits 1 and counts as discriminating. With def f(a): return 'WRONG' they exit 0. I ran the first three and got pre=1, post=0 for all three. A judge that writes the parenthesized-assert mistake, which is a classic one, earns "verified" for wrong work. The if [ … = "7" ]; then echo ok; fi shell form is correctly caught by cannotFail. These Python forms aren't.
• Fix: in cannotFail, flag assert ( followed by a top-level comma, or True / or 1 at the top level of an assert, a conditional expression as the assert's test, and -O / PYTHONOPTIMIZE. Cheaper still: run Python checks with -W error::SyntaxWarning at seal time. Add all four to the property list in test/exact-values.test.ts.

7. STILL OPEN from the last review: 4 (one trivial exact literal, like [], False or exit 1, can carry "verified" on a stub), 5 (the prompt reversal isn't measured by the replay; your A/B now suggests it costs recall), 6 (the oracle form trusts files the work wrote), and 7 (the advisory passed-checks return drops gap, sessions sealed before this lands have no exact tag, and rule 3 doesn't check that the deliverable ran).

Stack note

#45 and #46 both branch from claude/value-classifier and both edit src/ask.ts. A trial merge of the two is clean. Note that with both in, the cut-off retry here calls askOnce, and #45's temperature retry still recurses inside askOnce while holding the local turn. So #45's deadlock is reachable from this path too, until it's fixed there.

[solvyxtech]

Round-3 review, item by item. Fixed in claude/integration-all@3d1ca16 (#49); this branch is superseded by integration.

• 1 (opt-in): fixed. The exact rule is off unless MAAT_REQUIRE_EXACT=1 (exactRuleOn(); tierOf({requireExact}) overrides it). Off means the tier is decided as before Exact tag: verified needs a check that compares the output with an exact value #46, the drafter and auditor get the earlier prompts (draftSystem() / auditSystem(); DRAFT_SYSTEM_EXACT / AUDIT_SYSTEM_EXACT keep this PR's), and V-property-only does not run. Tests cover both settings: tier, prompts and audit gate.
• 2 (containsValue passing a crash that repeats its input): fixed. Containment does not count a value that appears in the program's own arguments, or when stderr is merged in (2>&1, |&, &>).
• 3 (named constant across items): fixed. Inside a quantifier, only loop variables over a literal table vary. ok = True bound once is still the same constant.
• 4 (model names in tests): fixed. The tests now say "a worker model checked by a judge model", judge-model and worker-model. Commit messages were not rewritten.
• 5 (cut-off retry): fixed. The retry ceiling is capped at ASK_RETRY_MAX_TOKENS (16k), a caller already at the cap is not retried, and if the retry fails outright the first cut-off reply is returned. There are tests for each.
• 6 (Python asserts that cannot fail): fixed. pythonAssertNeverFails covers a tuple assert, or True, a conditional with a true branch, and python -O / PYTHONOPTIMIZE. These are neither exact nor sealable (L16 cannotFail).
• 7 (one trivial literal carrying verified): fixed. [], {}, '', True/False, None, 0/1 and exit status 1 do not count as exact on their own.