[Yigtwxx]

What this changes

• scripts/eval_averitec.py gains --browser, --judge and --fresh. Every source now records the ladder step that read it and, under --judge, the judge's opinion when its verdict was in the escalation band. The report adds two numbers beside the product's:

  • the accuracy with the browser's pages dropped (a lower bound on the pages a run without the browser reads, not on its accuracy);
  • a hypothetical column in which the judge decides the low band, labelled as something the product never does (spec §11.1).

  Each run setup gets its own results file and report, so a --resume never mixes setups.

• verify.escalates(verdict) is public, so the eval measures the exact band the product sends to the judge.

• Three runs on 2026-10-05 (AVeriTeC dev, 100 claims, 0.708 majority baseline):

  Run	3-way	Without the browser
  Gold URLs, browser, judge	0.371	0.348
  Search, browser, sentence queries	0.371	0.348
  Search, browser, judge queries	0.483	0.483

• README "Measured", the evidence-search section, the site's Measured data, CHANGELOG, and OPEN-ITEMS (20.10 done; new 20.13, 20.14 and 20.15).

Why

OPEN-ITEMS 20.10: every published AVeriTeC number had been measured without the browser and without the judge. The results:

• Coverage is no longer the main loss. 165 of 200 gold URLs were read, and the browser is worth about two claims, which is within the noise.
• The reading is the loss. 36 of 89 claims had a readable source and came back NEI.
• The judge's queries are the one gain that cleared the noise: 14 claims fixed, 4 broken, exact McNemar p = 0.031.
• As shipped, the judge never sees an NEI (no passage, rule 1), so its hypothetical column equals the product (20.13).
• Search calls far more false claims SUPPORTED than the cited sources do: 20 to 22 of 63, against 5 (20.15). The cause is marked as an unmeasured hypothesis.

The harness change had an independent review and a re-review, and every number in the prose was fact-checked against the raw results (3 rounds).

Checklist

• ruff check and ruff format --check are clean (and mypy --strict, plus mypy scripts/eval_averitec.py)
• pytest passes
• Type annotations on every new function signature
• No network access in unit tests
• If a new reported state was added, spec section 15 is updated too (none added)
• This change does not make the tool sound more confident than its evidence

🤖 Generated with Claude Code

https://claude.ai/code/session_01K46EEsxMvuAuM8wjvPNWLM

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
proofpath	Ready	Preview	Oct 5, 2026 12:19pm UTC