fix: engine-first, decisive pre-submission sweep in SKILL.md

[moonweave]

This PR changes SKILL.md (and CHANGELOG.md) and adds measurement data under benchmarks/e2e/v2/. It makes no engine, AGENT_USAGE.md, or test changes. SKILL.md grows from 413 to 432 lines.

Why

In the v2 end-to-end runs, the skill's answers hedged ("can't say for sure it's fabricated") in 6 of 16 Opus A answers and 1 of 8 Sonnet A answers. Three causes:

• Semantic Scholar 429s. SKILL.md told agents to hand-search Semantic Scholar, which rate-limited them, and then to "note single-source limitation". The engine never hit a rate limit itself.
• Hand-written claim checks. Agents checked claims with hand-written curl and never ran check-file.
• Oversized engine output. Engine JSON exceeded the tool-result limit in 11 of 24 runs.

What changed in SKILL.md

Removed:

• The duplicate ref-verify … command blocks for verify-doi, check-bib, and check-claim. The bundled form stays, and one sentence says how to use an installed engine instead.
• The 5-line check-claim source-order paragraph, which repeated the CLI intro. It is now one line on --source.
• --json from the check-bib example.
• Semantic Scholar as a default Layer 1 existence source.
• "fetch from CrossRef or S2" in the anti-hallucination rules.
• The edge case "S2 rate limiting: … note single-source limitation".

Added:

• Pre-submission sweep (the mode decision now points to it):
  1. Run check-bib on the reference list only. Read the text output, or save --json to a file and read it in parts.
  2. Run check-file on the numeric claims, copied exactly as written. The example is \SI{172.3}{\joule\per\mole}, which means J/mol; never correct a claim while copying it.
  3. Judge each claim from the abstract sentence the engine returns, never from memory or the label alone:
     • A different value, unit, or direction is wrong, even under ACCEPT.
     • The same value, unit, and direction is not a problem, and the label stays WARN.
     • If the sentence covers only part of the claim, or something else, read the whole abstract before saying the abstract does not state it.
  4. Look things up by hand only where the engine is undecided. For a citation without a title, search the author, the journal, and the topic words of the citing sentence. Layer 3b and Layers 4–5 still apply.
• Layer 1: OpenAlex is the second existence source; Semantic Scholar is only a last resort.
• Layer 3: fetch the abstract with the engine first. The manual list applies only when the engine is unavailable or found no abstract.
• Rate limits: a 429 is no evidence. Mention it only if it leaves an item undecided, and never soften a verdict because of it.
• Output format for the sweep: lead with decisions (each problem, its evidence, and the fix), then a short "Could not be confirmed automatically — look these up yourself" list. No commentary on tools or rate limits.

Kept unchanged:

• WARN, PARTIAL, REJECT, and UNVERIFIABLE are never acceptance.
• CLI ACCEPT is abstract-level evidence only.
• Mechanism claims require full text (Layer 3b), and ABSTRACT-ONLY is never upgraded.
• REFERENCE_UNMATCHED is never called fabricated from that result alone, and DOI_NOT_IN_CROSSREF is never called dead.
• Two-source existence, and Layers 4 and 5.

Measurement (condition A, v2 set, one run per manuscript per round)

	Opus before (2×)	Opus final (r3+r4)	Sonnet before (1×)	Sonnet final (r3+r4)
All planted errors	104/104	104/104	51/52	104/104
CLAIM_DIRECTION	16/16	16/16	7/8	16/16
False alarms (refs / unindexed / claims)	0/0/0	0/0/1	0/0/0	0/0/0
Runs using check-file	0/16	16/16	0/8	14/16
Answers mentioning a rate limit	11/16	0/16	2/8	0/16
…that softened a verdict	6/16	0	1/8	0
Engine output over the tool-result limit	6/16	0/16	5/8	1/16
Median time	75 s	64–66 s	56 s	53 s

Earlier rounds had regressions, now fixed. Each earlier round had one Sonnet miss caused by the new workflow:

• r1, title-less volume/page: missed because the search left out the citing sentence's topic words. Fixed by the hint.
• r2, unit: missed because the agent copied the claim already corrected to kJ/mol. Fixed by the exact-copy rule.

r3 and r4 are the same final text. Per-round tables, misses, and wrong statements are in benchmarks/e2e/v2/README.md, section "SKILL.md fix". All adjudication files are in benchmarks/e2e/v2/skill-fix/.

Engine finding for a separate PR. Run on the 30 v2 claims, check-file did two wrong things:

• It returned ACCEPT for "a service lifetime of 61 years at 80 °C" when the evidence says 16 years.
• It returned WARN for 8 of 8 correct claims, including verbatim ones.

🤖 Generated with Claude Code