Value tag: count behavioural assertions on literal input

[solvyxtech]

assertsValue decides the value tag on sealed checks and the post-work audit's V-no-value gate. It rejected real behavioural assertions. "No check that ran the work asserted an expected value" is the biggest remaining reason good-judge runs go unverified.

What changes (src/tiers.ts, rule 5 plus a rule 1 fix)

• is None / is not None on a call into the work with a literal argument, or a literal in / not in its answer. The result has to be asserted, through assert, sys.exit(0 if … else 1) or if …: sys.exit(1).
• A specific exception, where only the path that did not raise exits non-zero: try/except KeyError, pytest.raises, assertRaises, assert.throws. except Exception does not count, and neither does a handler that only prints.
• grep (plain, -q or -x, never -v, -c or -l) of a literal at the end of a pipeline that runs a program. head -1 out.csv | grep and git log | grep still don't count.
• Rule 1 fix: a ; or newline inside [ "$(python3 -c "a; b")" = "2" ] used to hide the bracket. These brackets are now read shell-aware, and the operands are compared as whole shell words. $'…' now counts as a literal. Type probes (type(x).__name__, isinstance) still don't count.
• Not widened: a pattern the work wrote, matched against strings the check chose (re.search(p, '…')). With that form included, 4 of the 7 runs it alone would have verified were grader failures. All 4 were regex runs whose judge examples missed x 1.2.3.4.5 on 2024-01-05.
• Structural checks (existence, isinstance, hasattr, len > 0), the truthiness of an arbitrary call, and timing bounds still get no tag.

Replay (bench/local/value_replay.py, run over the 2026-10-07 lanes; 722 runs, 1395 distinct commands)

• The old rule rejects 677 commands. The new rule tags 77 of them, and every command the old rule tagged keeps its tag.
• Sealed checks: 118 runs were blocked only by the value tag. 8 flip to verified: 8 passed the grader, 0 failed. Counting only checks that failed before the work, 7 flip and 7 passed.
• Audit (--audit-gates, gates run in the bench container): 71 checks dropped with V-no-value are now tagged. 7 runs flip to verified-audit: 7 passed, 0 failed.
• Recorded "verified" on these lanes: 92/100 right before, 100/108 after.
• Caveat: the critic's surface tag isn't journalled. 6 of the 8 sealed flips have several passing checks, so those counts can only be too high.

Tests: every real example from the audit journals is in test/tiers.test.ts, both positive and negative. npm run check is green with ~/.local/bin off PATH.

🤖 Generated with Claude Code

https://claude.ai/code/session_01D63hP8FwZTxcamaLRq4jRt

[solvyxtech]

Verdict: changes requested (2 blocking, 4 non-blocking). Review only, nothing merged. Posted as a comment review because GitHub won't let this account request changes on its own PR.

The direction is right, and the replay is the right way to decide it. Withdrawing the "pattern the work wrote" form after the 4-of-7 grader failures was a good call, and the shell-aware bracket read fixes a real miss. Pairing rules are clean: no competitor names in the added lines (the only claude/ strings are the default --base branch ref in bench/local/value_replay.py), no subscription backend, nothing touching fine-tune data, and the branch goes claude/value-classifier into claude/post-work-audit.

Tested against 2ed2838: npm run typecheck is clean, test/tiers.test.ts passes 76/76, and the full suite passes 2132/2132 with ~/.local/bin off PATH. (On this box, Node 20.19.2, the runner still exited 1 afterwards because the coverage reporter threw Cannot read properties of undefined (reading 'line'). That looks local to the box rather than caused by this PR.) CI did not run: typecheck-and-test was never started because of the GitHub Actions billing or spending-limit block on the account, so the red check isn't a code failure.

Every probe below was run against the built assertsValue on this branch and on claude/post-work-audit. On the base branch all of them return false, and on this branch they all return true.

Blocking

1. A pipeline negated with ! gets the value tag, so "this output does not contain X" counts as asserting a value. Confidence: high (reproduced on the function).

• Where: src/tiers.ts:243 (grepsRunOutput) and src/tiers.ts:252. The -v exclusion only looks at grep's own flags. The leading ! is stripped by stageProgram to find the runner, but nothing checks whether it negates the whole pipeline.
• Scenario: ! python3 tool.py 2>&1 | grep -q Traceback is tagged value. cannotFail, swallowsExit and runnerPipedAway all return null for it, so it gets through L16 too. Take a task like "fix the crash in tool.py". Before the work the tool crashes, so the check fails and counts as discriminating. A worker that wraps main() in try/except: pass, or makes the tool print nothing, then passes it, and the run becomes "verified" on a check that only proves the tool printed no traceback.
• Why it matters: this is exactly the -v case the PR says it excludes, just spelled differently. Before this PR the same run stopped at "no check asserted an expected value".
• Fix: refuse rule 5c when the top-level command starts with !, or when it is followed by && exit 1, && false, or then exit 1 (the "grep finds it, so fail" shapes). Add ! python3 tool.py | grep -q Traceback to notBehaviour.

2. is not None and a literal not in the result are weaker than truthiness, but they count while truthiness doesn't. Confidence: medium (reproduced on the function. The replay found no grader failure from these, but the sample of flips is 8 runs).

• Where: src/tiers.ts:411 (member, which allows not in) and src/tiers.ts:413 (none, which allows is not None).
• Scenario: assert pricing.best_coupon('item1', 1) is not None and assert '!' not in slug.slugify('Hello!') are both tagged. Before the work the function doesn't exist, so both fail and count as discriminating. A wrong implementation that returns "", 0, False or [] passes both. Meanwhile assert tool.ok('x'), which at least rules out False and "", is correctly refused as "truthiness of an arbitrary call". The rule's own principle ("a check counted wrongly costs the word its meaning") points the other way here.
• Fix: keep is None and positive in, which name a specific value. Count is not None and not in only when they sit in an and with a conjunct that already asserts a value. Otherwise drop them. Move the existing is not None example in behaviour to notBehaviour, or pair it with a value.

Non-blocking

3. A runner that only reads a file counts, which goes against the stated rule that head -1 out.csv | grep doesn't. Confidence: medium (reproduced on the function).

• Where: src/tiers.ts:240 (RUNNER) and src/tiers.ts:258.
• Scenario: python3 -c "print(open('out.csv').read())" | grep -q 'id,name', bash -c 'head -1 out.csv' | grep -q 'id,name', and cat out.csv | python3 -c "import sys; print(sys.stdin.read())" | grep -q 'id,name' are all tagged. Each fails before the work when out.csv is missing, so a worker that writes only a header row earns "verified". The same goes for python3 -c "print('untitled')" | grep -q untitled, where the check prints its own literal. When there's a pre-work copy, P1 drops that last one, but with no copy it would be sealed with the value tag.
• Fix: when the runner stage is python3 -c, node -e, bash -c or sh -c, require the inline code to contain a deliverableCall, or require the runner to be a script by path (./x, x.py). Also refuse a pipeline whose first stage is cat <file>.

4. A keyword argument makes any method call look like "the deliverable on literal input". Confidence: medium-high (reproduced).

• Where: src/tiers.ts:264 (NOT_DELIVERABLE) and src/tiers.ts:366-368 (literalInput strips name=, so encoding='utf-8' reads as a literal).
• Scenario: assert 'id,name' in Path('out.csv').read_text(encoding='utf-8') is tagged. Path is opaque, but read_text(encoding='utf-8') then counts as a call into the work with literal input. It's the same file-shape read as finding 3.
• Fix: don't count keyword arguments as literal input, and add read_text, read_bytes, readlines, fetchall, decode and similar to NOT_DELIVERABLE.

5. Exception handlers that catch the check's own failing exit, or everything. Confidence: medium on the mechanism, low on how often it happens (reproduced on the function).

• Where: src/tiers.ts:496-499 reads only the first name in except (...). src/tiers.ts:483 doesn't require with.
• Scenarios:
  • try: p.f('x'); sys.exit(1) followed by except SystemExit: pass is tagged. The handler catches the sys.exit(1), so the check passes whatever f does.
  • except (KeyError, Exception): is tagged, because only KeyError is read.
  • pytest.raises(KeyError) used as a statement without with, followed by p.f('x'), is tagged. Here the check passes exactly when no exception is raised.
• Fix: refuse when any name in the except tuple is SystemExit, BaseException or Exception, and require with before pytest.raises(E) unless the call form pytest.raises(E, f, ...) is used.

6. Very short or alternated grep patterns. Confidence: low-medium.

• Where: src/tiers.ts:257, which needs only 2 word characters.
• Scenario: python3 t.py | grep -qi 'er' matches nearly any error text, and python3 t.py | grep -E 'ok|fail' passes on either outcome. Both are tagged.
• Fix: require about 4 or more word characters, and refuse | alternation under -E/egrep (or count only when every branch is a value).

No changes needed in bench/local/value_replay.py for the pairing rules. The docker gate mounts the tree and bench read-only, and nothing in it is tied to a provider. Once 1 and 2 are fixed, it would be worth rerunning the replay to confirm the 8 sealed and 7 audit flips still hold.

[solvyxtech]

Verdict: CHANGES REQUESTED

Prior items: 0 of 2 blockers fixed. ! cmd | grep -q X still gets the value tag and passes the can't-fail screen, and is not None and literal not in still count as value assertions. Non-blocking items 3–6 are still open. The head is still 2ed2838, the commit I reviewed in 5450739903, and the base hasn't moved.

Fresh pass from Grok, comment only. I re-ran it at 2ed2838: npm run typecheck is clean, and the suite passed 2132 of 2132. CI hasn't run because of the account-wide GitHub Actions billing block. One new note, non-blocking:

Comparisons that are never asserted, or can never fail, still get the value tag. This is pre-existing on claude/post-work-audit, so it isn't a regression from this PR, but this PR owns assertsValue. Confidence: high (reproduced with real python3).

• python3 -c "import tool; tool.f('a') == 'x'" gets ["task","value"] even though the comparison is a bare expression statement that nothing asserts. cannotFail (src/checklint.ts:403) and swallowsExit both return null.
• The same happens with assert (tool.f('a') == 'x', 'msg') (always-true tuple), assert … or True, assert … if False else True, and python3 -O / PYTHONOPTIMIZE=1.
• Scenario: before the work, import tool fails, so the check exits 1 and counts as discriminating. With any tool.f, including return 'WRONG', it exits 0. I ran it and got pre=1, post=0. A weak judge writes one of these (the parenthesized assert is a classic mistake, and Python itself warns "assertion is always true"), and the run is labeled "verified" on wrong work.
• Fix: require the comparison to be asserted (under assert, if not …: exit/raise, sys.exit(0 if … else 1), or a test bracket), the way assertsExact does in #46. Teach cannotFail the four always-pass forms. Add them to notBehaviour in test/tiers.test.ts. #46 gets the same note, because its exact tag accepts all four.

[solvyxtech]

Round-3 review, item by item. Fixed in claude/integration-all@3d1ca16 (#49); this branch is superseded by integration.

• 1 (!-negated pipeline tagged as value): fixed. grepsRunOutput refuses a segment that starts with !, one followed by && exit N/&& false, and if … | grep …; then exit N. Those three are in notBehaviour, and so is the earlier md2html ! make | grep example.
• 2 (is not None / not in): fixed. Only a positive 'x' in f(...) and is None count. not in, not 'x' in and is not None do not. The best_coupon … is not None example moved to notBehaviour, and slugify not in cases were added.
• 6 (short/alternated grep patterns): fixed. A pattern needs 4 word characters, and an -E/egrep alternation does not count.
• 3, 4, 5: not done in this pass.