Yigtwxx · Oct 5, 2026 · Oct 5, 2026 · Oct 5, 2026 · Oct 5, 2026
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -7,6 +7,15 @@ All notable changes to this project are documented here. The format follows
 ## [Unreleased]
 
 ### Added
+- **AVeriTeC eval: `--browser`, `--judge` and `--fresh`** (`scripts/eval_averitec.py`).
+  Every source now records the ladder step that read it and, under `--judge`, the
+  judge's opinion when its verdict was in the escalation band. The report adds two
+  numbers beside the product's: the accuracy with the browser's pages dropped, and a
+  hypothetical one in which the judge decides the low band. The product never does the
+  second, and the report says so. Each run setup gets its own results file and report.
+  Measured on 2026-10-05: gold URLs 0.371; search with sentence queries 0.371; search
+  with the judge writing the queries 0.483 (14 claims fixed, 4 broken, p = 0.031). The
+  majority baseline is 0.708.
 - **`proofpath update`** installs the latest release from PyPI, and
   **`proofpath update --check`** only says whether there is one. In the TUI they are
   `/update` and `/update --check`. Running the command is the consent, so nothing

diff --git a/README.md b/README.md
@@ -410,9 +410,21 @@ if it is not in English) before searching. If the judge is rate-limited, the loc
 fallback below takes over; only if that fails too is the plain sentence searched, and
 the report says so.
 
-**This is experimental.** On AVeriTeC's search scenario, proofpath scores
-[**0.404**](docs/eval/2026-09-29-averitec-search.md) against a 0.708 majority baseline, and every report from a
-run that searched prints that line at the top.
+**This is experimental.** On AVeriTeC's search scenario (2026-10-05, browser allowed),
+proofpath scores [**0.371**](docs/eval/2026-10-05-averitec-search-browser-fresh.md) with
+sentence queries and [**0.483**](docs/eval/2026-10-05-averitec-search-browser-judge-fresh.md)
+when the judge writes the queries, against a 0.708 majority baseline. The judge's queries
+are the one gain that cleared the noise: on the same 89 claims they fixed 14 and broke 4
+(paired exact McNemar p = 0.031). The share of pages from fact-check sites did not change
+(22.0 % against 22.5 %).
+
+**Search also asserts the wrong way far more often than the cited sources do.** With the
+judge's queries, 25 of the 89 claims came back the wrong way, and 20 of those were false
+claims called SUPPORTED. With sentence queries the figure was 29, against 11 when the gold
+URLs are read. ("Wrong way" counts Supported read as REFUTED and Refuted read as SUPPORTED.)
+One likely cause, not yet measured: a page found by searching for a claim often repeats it,
+and the models may read the repetition as support. Both scores are still below the
+baseline, and every report from a run that searched prints that line at the top.
 
 ## Optional LLM judge (v0.3)
 
@@ -480,16 +492,24 @@ switch to, the run says so and tells you how to add one, on the same channels.
 | Source access | 50 DOIs | 72 % full text, 18 % abstract only, 10 % nothing ([details](docs/eval/2026-09-11-coverage.md)) — a real biomedical paper in the live runs reached 33 % full text |
 | Citation pairing, numeric | 61 hand-built passages | 0.99 ([details](docs/eval/2026-09-11-pairing.md)) |
 | Citation pairing, author-year | 55 hand-built passages, 83 expectations | 0.940 ([details](docs/eval/2026-09-12-pairing-author-year.md)) |
-| **End to end on real web claims** | AVeriTeC dev, 100 claims | **0.270 3-way accuracy against a 0.708 majority baseline — worse than always guessing "refuted"** ([details](docs/eval/2026-09-16-averitec.md)) |
-
-**The AVeriTeC row is the one to read before trusting this tool on a news claim.** A third
-of those claims had no readable source at all: 32 of the source URLs needed the browser
-step, 29 were unreachable, 14 were refused by `robots.txt`. On the claims that *did* have a
-readable source the score is 0.361 — still below the baseline, and every one of the 19
-`Supported` claims was missed. The retrieval and entailment models were calibrated on
-scientific abstracts, and a fact-check page is a different object: long, discursive, and
-usually quoting the claim it debunks. Nothing was tuned after that measurement, and no
-blocked URL was dropped from it.
+| **End to end on real web claims** | AVeriTeC dev, 100 claims | **0.371 3-way accuracy from the gold source URLs (browser allowed) against a 0.708 majority baseline — still worse than always guessing "refuted"** ([details](docs/eval/2026-10-05-averitec-browser-judge-fresh.md)). With evidence search and a judge writing the queries: 0.483 ([details](docs/eval/2026-10-05-averitec-search-browser-judge-fresh.md)). The first measurement, without the browser, was 0.270 ([2026-09-16](docs/eval/2026-09-16-averitec.md)) |
+
+**The AVeriTeC row is the one to read before trusting this tool on a news claim.** Coverage
+is no longer the main loss. On 2026-10-05, 165 of the 200 gold source URLs were read,
+and only 10 of those needed the browser step, worth about two claims, which is within the
+noise of 89. The loss is in the reading:
+
+- 36 of the 89 scorable claims had a readable source and still came back NEI where the
+  gold label was Supported or Refuted;
+- 11 were called the wrong way;
+- 7 had nothing readable.
+
+The likely reason, not separately measured: the retrieval and entailment models were
+calibrated on scientific abstracts, and a fact-check page is a different object (long,
+discursive, and usually quoting the claim it debunks).
+As shipped, the judge cannot help with those NEIs. An NEI verdict carries no passage, so it
+is never escalated (rule 1); only 1 of the 165 read sources reached the judge. Nothing was
+tuned after these measurements, and no blocked URL was dropped from them.
 
 What proofpath is good at is the academic path the other rows measure: finding out whether a
 cited paper exists, whether it was retracted, and whether its text says what the sentence

diff --git a/docs/eval/2026-10-05-averitec-browser-judge-fresh.md b/docs/eval/2026-10-05-averitec-browser-judge-fresh.md
@@ -0,0 +1,86 @@
+# AVeriTeC dev — 2026-10-05
+
+- claims: 100  (limit=100)
+- measured path: gold source URLs, no search
+- browser step: allowed
+- fetch: live, cache bypassed
+- judge: groq openai/gpt-oss-120b (its opinions are the hypothetical column below; the verdicts are the models')
+- dataset: `https://raw.githubusercontent.com/MichSchli/AVeriTeC/main/data/dev.json`  (sha256 499793726b4a…)
+- one run of the whole product: real claims, real source pages, real fetch ladder.
+
+## Headline
+
+3-way accuracy **0.371** vs majority baseline 0.708, over 89 of 100 claims.
+
+The 3-way number excludes the Conflicting Evidence/Cherrypicking rows, which proofpath has no verdict for; counting them as wrong gives a 4-way accuracy of 0.330.
+
+## Per label
+
+| label | n | correct | accuracy |
+|---|---|---|---|
+| Supported | 19 | 0 | 0.000 |
+| Refuted | 63 | 28 | 0.444 |
+| Not Enough Evidence | 7 | 5 | 0.714 |
+| Conflicting Evidence/Cherrypicking | 11 | 0 | 0.000 |
+
+## Source coverage
+
+| state | count |
+|---|---|
+| ok | 165 |
+| UNVERIFIED (blocked, robots.txt) | 17 |
+| UNVERIFIED (provider unavailable) | 10 |
+| not a url | 9 |
+| UNVERIFIED (blocked) | 4 |
+| UNVERIFIED (reached, no text extracted) | 4 |
+
+76 of 200 source URLs are web.archive.org snapshots (38.0 %).
+
+## Without the browser
+
+10 of 165 read sources were read by the browser step. Treating them as unread gives a 3-way accuracy of **0.348**.
+
+Those pages never reached the Wayback step, which a run without the browser tries next, so this is a lower bound on the pages such a run reads. It is not a bound on its accuracy: dropping a page can lose a right verdict or a wrong one.
+
+## Judge (hypothetical)
+
+The product never lets the judge change a verdict (spec section 11.1). This column is what the accuracy would be if it did: a medium or high model verdict still wins; otherwise the judge's opinions on the escalated sources decide, by majority of the asserting votes (an NEI opinion abstains), a tie being NEI. An escalation the judge did not answer votes with the models' label.
+
+3-way accuracy **0.371**, against 0.371 for the product. 1 sources were escalated; 0 got no opinion back.
+
+| label | n | correct | accuracy |
+|---|---|---|---|
+| Supported | 19 | 0 | 0.000 |
+| Refuted | 63 | 28 | 0.444 |
+| Not Enough Evidence | 7 | 5 | 0.714 |
+
+| judge model | opinions |
+|---|---|
+| groq openai/gpt-oss-120b | 1 |
+
+| decided by | n | correct | accuracy |
+|---|---|---|---|
+| models (medium or high tier) | 41 | 28 | 0.683 |
+| nothing escalated | 48 | 5 | 0.104 |
+
+## Notes
+
+- Same 100 dev claims as `2026-09-16-averitec.md` (0.270), but not the same web. That
+  run used `--no-browser` and read 120 sources; this one read 165 of 200, the browser
+  step 10 of them. Pages that were unreachable then answer now, so the two numbers
+  measure different snapshots of the same claims.
+- **The browser is worth about two claims here.** Dropping the 10 pages it read gives
+  0.348, against 0.371. At 89 claims the standard error is about ±0.05, so this is
+  within the noise.
+- **The judge saw one source.** `pipeline.decide` never gives an NEI verdict a
+  passage, and `verify.escalates` sends nothing without one (rule 1). 105 of the 165
+  read sources are NEI. The hypothetical column therefore equals the product. The judge
+  never switched to its local fallback.
+- **Where the 56 wrong claims are.** 36 had a readable source and came back NEI where
+  the gold label was Supported or Refuted. 11 were called the wrong way: Refuted read as
+  Supported 5 times, Supported as Refuted 6 times. 7 had nothing readable, and 2 gold
+  NEI claims got a verdict. Coverage is no longer the main loss; the reading is.
+- Of the 33 claims scored right, one is a gold-NEI claim with nothing readable, which
+  the score counts as NEI. 32 are verdicts or NEIs on sources that were read.
+- Measured with the default NLI profile (pinned), `--fresh`, `--sleep 1.0`.
+
diff --git a/docs/eval/2026-10-05-averitec-search-browser-fresh.md b/docs/eval/2026-10-05-averitec-search-browser-fresh.md
@@ -0,0 +1,56 @@
+# AVeriTeC dev — 2026-10-05
+
+- claims: 100  (limit=100)
+- measured path: sentence queries, no judge
+- browser step: allowed
+- fetch: live, cache bypassed
+- dataset: `https://raw.githubusercontent.com/MichSchli/AVeriTeC/main/data/dev.json`  (sha256 499793726b4a…)
+- one run of the whole product: real claims, real source pages, real fetch ladder.
+
+## Headline
+
+3-way accuracy **0.371** vs majority baseline 0.708, over 89 of 100 claims.
+
+The 3-way number excludes the Conflicting Evidence/Cherrypicking rows, which proofpath has no verdict for; counting them as wrong gives a 4-way accuracy of 0.330.
+
+## Per label
+
+| label | n | correct | accuracy |
+|---|---|---|---|
+| Supported | 19 | 6 | 0.316 |
+| Refuted | 63 | 26 | 0.413 |
+| Not Enough Evidence | 7 | 1 | 0.143 |
+| Conflicting Evidence/Cherrypicking | 11 | 0 | 0.000 |
+
+## Source coverage
+
+| state | count |
+|---|---|
+| ok | 313 |
+| UNVERIFIED (blocked, robots.txt) | 6 |
+| UNVERIFIED (blocked) | 5 |
+| UNVERIFIED (provider unavailable) | 1 |
+
+0 of 325 source URLs are web.archive.org snapshots (0.0 %).
+
+## Without the browser
+
+18 of 313 read sources were read by the browser step. Treating them as unread gives a 3-way accuracy of **0.348**.
+
+Those pages never reached the Wayback step, which a run without the browser tries next, so this is a lower bound on the pages such a run reads. It is not a bound on its accuracy: dropping a page can lose a right verdict or a wrong one.
+
+## Notes
+
+- Evidence search with sentence queries (no judge), browser allowed, every page fetched
+  live. The 2026-09-29 run of the same path, without the browser, scored 0.404. The
+  three-claim gap sits inside the ±0.05 standard error at 89 claims, and the search
+  results themselves differ between the days.
+- The browser read 18 of the 313 pages read. Dropping them gives 0.348.
+- Coverage is close to complete: 313 of 325 pages read, 11 blocked (6 by `robots.txt`).
+- **Wrong-way verdicts:** 29 of 89. 22 false claims were called SUPPORTED and 7 true
+  ones REFUTED, against 11 when the gold URLs are read. The count leaves out verdicts
+  given to gold-NEI claims. One likely cause, not yet measured: a page found by
+  searching for a claim often repeats it.
+- This run is the baseline for `2026-10-05-averitec-search-browser-judge-fresh.md`,
+  which differs only in who writes the queries.
+
diff --git a/docs/eval/2026-10-05-averitec-search-browser-judge-fresh.md b/docs/eval/2026-10-05-averitec-search-browser-judge-fresh.md
@@ -0,0 +1,87 @@
+# AVeriTeC dev — 2026-10-05
+
+- claims: 100  (limit=100)
+- measured path: judge queries (groq openai/gpt-oss-120b)
+- browser step: allowed
+- fetch: live, cache bypassed
+- judge: groq openai/gpt-oss-120b (writes the search queries; its opinions are the hypothetical column below)
+- dataset: `https://raw.githubusercontent.com/MichSchli/AVeriTeC/main/data/dev.json`  (sha256 499793726b4a…)
+- one run of the whole product: real claims, real source pages, real fetch ladder.
+
+## Headline
+
+3-way accuracy **0.483** vs majority baseline 0.708, over 89 of 100 claims.
+
+The 3-way number excludes the Conflicting Evidence/Cherrypicking rows, which proofpath has no verdict for; counting them as wrong gives a 4-way accuracy of 0.430.
+
+## Per label
+
+| label | n | correct | accuracy |
+|---|---|---|---|
+| Supported | 19 | 10 | 0.526 |
+| Refuted | 63 | 31 | 0.492 |
+| Not Enough Evidence | 7 | 2 | 0.286 |
+| Conflicting Evidence/Cherrypicking | 11 | 0 | 0.000 |
+
+## Source coverage
+
+| state | count |
+|---|---|
+| ok | 319 |
+| UNVERIFIED (blocked, robots.txt) | 9 |
+| UNVERIFIED (blocked) | 4 |
+
+0 of 332 source URLs are web.archive.org snapshots (0.0 %).
+
+## Without the browser
+
+17 of 319 read sources were read by the browser step. Treating them as unread gives a 3-way accuracy of **0.483**.
+
+Those pages never reached the Wayback step, which a run without the browser tries next, so this is a lower bound on the pages such a run reads. It is not a bound on its accuracy: dropping a page can lose a right verdict or a wrong one.
+
+## Judge (hypothetical)
+
+The product never lets the judge change a verdict (spec section 11.1). This column is what the accuracy would be if it did: a medium or high model verdict still wins; otherwise the judge's opinions on the escalated sources decide, by majority of the asserting votes (an NEI opinion abstains), a tie being NEI. An escalation the judge did not answer votes with the models' label.
+
+3-way accuracy **0.483**, against 0.483 for the product. 1 sources were escalated; 0 got no opinion back.
+
+| label | n | correct | accuracy |
+|---|---|---|---|
+| Supported | 19 | 10 | 0.526 |
+| Refuted | 63 | 31 | 0.492 |
+| Not Enough Evidence | 7 | 2 | 0.286 |
+
+| judge model | opinions |
+|---|---|
+| groq openai/gpt-oss-120b | 1 |
+
+| decided by | n | correct | accuracy |
+|---|---|---|---|
+| models (medium or high tier) | 71 | 41 | 0.577 |
+| nothing escalated | 18 | 2 | 0.111 |
+
+## Notes
+
+- **The one gain that cleared the noise.** The only difference from
+  `2026-10-05-averitec-search-browser-fresh.md` (0.371) is who writes the search
+  queries: here the judge, `groq openai/gpt-oss-120b`. Paired over the same 89 claims,
+  the judge's queries were right where the sentence queries were wrong 14 times, and
+  the reverse 4 times (exact McNemar p = 0.031). `Supported` went from 6/19 to 10/19.
+- **The share of fact-check pages did not change.** By a host/URL heuristic, they were
+  22.0 % of the pages here and 22.5 % with sentence queries. That says the mix did not
+  shift, not that no better fact-check page was found. The checker's own site is
+  excluded in both, as in every search run.
+- **Many claims still come back the wrong way.** 25 of the 89 claims came back the wrong
+  way: 20 false claims were called SUPPORTED, and 5 true ones REFUTED. Reading the gold
+  URLs gives 11. The count leaves out verdicts given to gold-NEI claims. One likely
+  cause, not yet measured: a page found by searching for a claim often repeats it.
+- **One risk this run cannot rule out.** A model writing the queries may carry what it
+  remembers about a claim into them. The verdicts still come from the pages read, each
+  with its passage, but the choice of pages may not be blind.
+- All queries were written by Groq. The judge never switched to its local fallback;
+  the run printed no switch notice.
+- The browser read 17 of the 319 pages read; dropping them leaves the score at 0.483.
+  The hypothetical judge column equals the product: one source was escalated (see the
+  gold-URL run's note on NEI verdicts).
+- Still below the 0.708 majority baseline, so the search stays experimental.
+