fix: check-claim accepts a numeric claim only when every number in it is supported

[moonweave]

check-claim accepted "a service lifetime of 61 years at 80 °C" from an abstract that says "16 years at 80 °C". A false ACCEPT is the verdict users trust, so this fixes the engine and adds two frozen claim sets to measure it.

Cause

• False ACCEPT. The numeric path compares only the first quantity its unit list recognises. "years" is not on that list, so the 61-year claim was judged on "80 °C" alone, and the subject words (service, lifetime, year) all appear in the evidence clause. The same blind spot dropped signs (−25 °C vs 25 °C), every percentage after the first, secondary units (42 ms vs 42 s), and direction words that are not the number's own comparator ("comparatively lower than ... 11.85%" was accepted from "higher than").
• Verbatim claim stays WARN. "1.28 MPa tensile strength with 573% elongation" sits inside parentheses in a ;-separated list. The percentage path scans the rest of the sentence for scope words and hits "at" in a later item ("retaining flexibility at −25 °C"). The word-for-word text path was switched off for any claim with a percentage.

Fix

• claim_quantities_supported(claim, evidence) in numeric_claim.py: every number in the claim, with its unit, prefix case (mPa/MPa), sign, and comparator, must be entailed by a number in the same evidence clause (numeric path) or sentence (percentage and text paths). Direction words ("higher", "decreased", ...) outside a number's own comparator must appear there too. An approximate figure (~, ≈, "about") supports only an approximate claim. It reads scientific notation, thousands with commas or thin spaces, ranges and lists ("0–6 kPa", "50 and 60 nm"), "± x", and spaced exponents ("kPa −1"). It skips numbers inside names (Fe2C, KIT-6, COVID-19, cm−1) and citation markers.
• Every ACCEPT path now also has to pass that check. The numeric path keeps looking at other clauses when it fails.
• A value inside parentheses is scoped to its own ;-separated list item for the scope-word check (prefix and suffix). Outside parentheses nothing changes, so "Actuation strain exceeded 117%; before treatment." stays WARN.
• Non-strain percentage claims can now be accepted word for word. Strain claims keep their pre-strain and actuation checks.

Measured

Two sets on 30 new papers (none in any other set, the E2E sets, or the fixtures), each frozen in its own commit before the engine was run on it. Runs used a fresh cache and no REF_VERIFY_MAILTO. Before = 6c44979 / 7d6a148 (engine unchanged), after = 2768bca. No abstract changed between runs.

Set	Run	False ACCEPT on unsupported	ACCEPT on supported	ACCEPT precision
claims-copy-v1 (copied, one edit)	before	4/30 (5–30%)	14/30 (30–64%)	14/18 (55–91%)
claims-copy-v1	after	0/30 (0–11%)	16/30 (36–70%)	16/16 (81–100%)
claims-v1 (paraphrased)	before	0/30 (0–11%)	0/30 (0–11%)	no ACCEPT
claims-v1	after	0/30 (0–11%)	0/30 (0–11%)	no ACCEPT

95% Wilson intervals. False ACCEPT by error type on the copy set, before → after: wrong number 1/6 → 0/6, wrong unit 1/6 → 0/6, wrong direction 1/6 → 0/6, multi-quantity 1/6 → 0/6, wrong subject 0/6 → 0/6.

• claims-v1 was the set asked for: 30 true and 30 false paraphrased claims, six per error type. The unfixed engine accepted none of them, true or false, so it could not show the bug. claims-copy-v1 was added after that run. Its supported claim copies an abstract span, and its unsupported claim changes exactly one thing in the span. That is how a sentence drafted from an abstract goes wrong.
• In-sample caveat. The direction-word rule was added after the before run showed cc1-10 ("higher than" → "lower than") passing. The copy set's wrong_direction row is partly in-sample.

Named E2E cases (benchmarks/e2e/v2/truth.json, live): v2m3_crystal_latex#03 (61 years) ACCEPT → WARN. v2m2_ionic_en#04 (1.28 MPa / 573%) WARN → ACCEPT. Across all 30 claim items: FLAG items with ACCEPT 1/22 → 0/22, CLAIM_OK items with ACCEPT 0/8 → 1/8. The other seven CLAIM_OK items were not tuned on. Five are paraphrases of the abstract ("4 mm/s at 7 kV and 2 Hz" vs "4 mm s −1 at 7 kV and 2 Hz"; "89.9% in electrospun fibres compared with 78.9%" vs "fibers (89.9%) compared to ... (78.9%)"). Two are Korean claims checked against English abstracts.

Risks

Accepts less (observed).

• A true claim whose numbers sit in two clauses of one sentence, when the clause the numeric path matched lacks the second number and the word-for-word path then hits a scope word later in the sentence. In the copy set, cc1-27 went ACCEPT → WARN: "(1.0510 S/m) and high sensitivity (4.7271 MPa−1)", followed by "whereas ... (from 0.000486 to 1.17 MPa)".
• An exact claim against approximate evidence ("5000 cycles" vs "~5000 cycles"): ACCEPT before, WARN now.
• An exact claim against "exceeded": "the device lifetime was 5000 cycles" from "The device lifetime exceeded 5000 cycles." was ACCEPT, now WARN.

Accepts less (possible, not seen in these sets). The unit is the token after the number, so a secondary unit written differently from the abstract (mAh cm−2 vs mAh/cm2, W/mK vs W/(m K)) blocks ACCEPT. Direction words must match exactly, so "rose" does not match "increased". In every case I tried, the old engine already returned WARN, because those words were required as subject words.

Accepts more.

• Word-for-word claims carrying a percentage (copy set: cc1-25, cc1-45, cc1-51, all labelled true).
• Inside parentheses, scope words from other ;-separated list items are ignored. A qualifier in the same item, inside or after the parentheses, still blocks ("(1.2 MPa with 500% elongation) at 80 °C;" stays WARN). A qualifier at the start of the sentence still blocks.
• Not covered by the new check: two numbers swapped inside one sentence when every number and word is still present ("ETL and HTL ... at 60 and 50 nm"). claim_quantities_supported cannot tell the order apart. Here the existing subject binding and the word-for-word path rejected all six wrong-subject claims, four of which swap two numbers.

Checks

• PYTHONPATH=src python -m unittest discover -s tests: 392 tests, OK. New: ClaimQuantitiesTests (5), 5 claim-check tests that use the two real abstract sentences, and 6 fixture rows in numeric_claim_eval.jsonl (one wrong number among several, all right, wrong unit prefix in one quantity, reversed direction in one quantity, numbers split across two sentences, and one sentence that has both numbers).
• evals/run_cli_regression.py (live): SAFETY 12/12.
• python -m py_compile src/ref_verify/*.py tests/*.py scripts/*.py: OK.
• Not merged and not released.

🤖 Generated with Claude Code