feat(judge): check the judge's quote, and measure it on the NEIs (20.9, 20.13)

CHANGELOG.md

@@ -6,6 +6,34 @@ All notable changes to this project are documented here. The format follows
 
 ## [Unreleased]
 
+### Changed
+- **The judge's quote is checked, and shown.** A `SUPPORTED` or `REFUTED` opinion now
+  has to return the exact words from the passage that decide it, in a new `quote`
+  field. An opinion is dropped, with a note saying why, when its quote has no words,
+  when a part of it is shorter than two words, or when it is not in the passage it was
+  shown at word boundaries ("5 million" is not in "15 million"). The match folds case,
+  curly quotes, dashes and the minus sign, spaces, the fraction slash and invisible
+  characters such as the soft hyphen, and nothing else: a superscript stays a
+  superscript, and a sign or a percent is never trimmed. `...` may elide the middle of
+  a quote as long as every part is in the passage, in order. The verified quote is
+  printed in the judge line, `judge (<model>): SUPPORTED — "<quote>" — <rationale>`,
+  and JSON gains a `quote` field. An `NEI` opinion asserts nothing and may quote
+  nothing. Cached opinions are now keyed by the passage they were checked against, and
+  those cached before this change are not served again: the next run asks.
+
+### Added
+- **AVeriTeC eval: `--judge-nei` and `--no-fallback`** (`scripts/eval_averitec.py`).
+  `--judge-nei` (requires `--judge`) also sends every read source whose verdict is NEI
+  to the judge, with the passage the models came closest to deciding on, and the
+  hypothetical column gains a second number in which those opinions vote too, with its
+  per-label counts, its wrong-way count beside the product's, and an exact McNemar test
+  against the product claim by claim. The report also counts opinions by model and kind
+  of escalation, and the ones the quote check dropped by reason. `--no-fallback`
+  (requires `--judge`) turns the judge's local fallback off for the run, so every
+  opinion is from the configured model. Both
+  flags get their own results file and report name. Measurement only: the product
+  still never sends an NEI to the judge.
+
 ## [0.4.11] - 2026-10-05
 
 `proofpath update` keeps proofpath current, and it keeps the extras you installed. A

README.md

@@ -442,9 +442,13 @@ were least sure about — the `low` tier, never a numeric mismatch and never a c
 without a quoted passage — go to the model in batches of up to 20 (about 7k tokens),
 each with its claim and the passage it was checked against. The model answers from the
 passage alone, and its label and one-sentence rationale are printed **beside** the local
-verdict: `= judge (groq openai/gpt-oss-120b): NEI — …`. The local verdict, the finding
-kind and the report's states never change. Opinions are cached with the verdict, so a
-re-run asks nothing.
+verdict: `= judge (groq openai/gpt-oss-120b): NEI — …`. A `SUPPORTED` or `REFUTED`
+opinion must also quote the words that decide it: at least two words, found in the
+passage at word boundaries (after folding case, curly quotes, dashes and spacing; `...`
+may elide a part). An opinion whose quote is not there is dropped, and the run says so
+in a note; a verified quote is printed in the judge line before the rationale. The local
+verdict, the finding kind and the report's states never change. Opinions are cached
+with the verdict, so a re-run asks nothing.
 
 **What it cannot do.** It never sees a source document, so it cannot introduce a claim
 or an evidence passage of its own; it cannot turn `NEI` into `SUPPORTED`; it cannot
@@ -511,7 +515,11 @@ The likely reason, not separately measured: the retrieval and entailment models
 calibrated on scientific abstracts, and a fact-check page is a different object (long,
 discursive, and usually quoting the claim it debunks).
 As shipped, the judge cannot help with those NEIs. An NEI verdict carries no passage, so it
-is never escalated (rule 1); only 1 of the 165 read sources reached the judge. Nothing was
+is never escalated (rule 1); only 1 of the 165 read sources reached the judge. Shown each
+NEI's closest passage in a measurement-only run, the judge answered NEI on 97 of 105, and
+its 8 other opinions fixed 5 claims and broke none (0.427, exact McNemar p = 0.062). That
+is just short of the bar set beforehand, so it is not in the product
+([details](docs/eval/2026-10-05-averitec-browser-judge-nei-nofallback.md)). Nothing was
 tuned after these measurements, and no blocked URL was dropped from them.
 
 What proofpath is good at is the academic path the other rows measure: finding out whether a

docs/eval/2026-10-05-averitec-browser-judge-nei-nofallback.md

@@ -0,0 +1,124 @@
+# AVeriTeC dev — 2026-10-05
+
+- claims: 100  (limit=100)
+- measured path: gold source URLs, no search
+- browser step: allowed
+- fetch: cache allowed
+- judge: groq openai/gpt-oss-120b (its opinions are the hypothetical column below; the verdicts are the models')
+- NEI probes: on (every read NEI source was also sent to the judge, with its closest passage)
+- judge fallback: off (every opinion is from the configured model)
+- dataset: `https://raw.githubusercontent.com/MichSchli/AVeriTeC/main/data/dev.json`  (sha256 499793726b4a…)
+- one run of the whole product: real claims, real source pages, real fetch ladder.
+
+## Headline
+
+3-way accuracy **0.371** vs majority baseline 0.708, over 89 of 100 claims.
+
+The 3-way number excludes the Conflicting Evidence/Cherrypicking rows, which proofpath has no verdict for; counting them as wrong gives a 4-way accuracy of 0.330.
+
+## Per label
+
+| label | n | correct | accuracy |
+|---|---|---|---|
+| Supported | 19 | 0 | 0.000 |
+| Refuted | 63 | 28 | 0.444 |
+| Not Enough Evidence | 7 | 5 | 0.714 |
+| Conflicting Evidence/Cherrypicking | 11 | 0 | 0.000 |
+
+## Source coverage
+
+| state | count |
+|---|---|
+| ok | 165 |
+| UNVERIFIED (blocked, robots.txt) | 17 |
+| not a url | 9 |
+| UNVERIFIED (unreachable) | 8 |
+| UNVERIFIED (blocked) | 4 |
+| UNVERIFIED (reached, no text extracted) | 4 |
+| UNVERIFIED (provider unavailable) | 2 |
+
+76 of 200 source URLs are web.archive.org snapshots (38.0 %).
+
+## Without the browser
+
+0 of 165 read sources were read by the browser step. Treating them as unread gives a 3-way accuracy of **0.371**.
+
+Those pages never reached the Wayback step, which a run without the browser tries next, so this is a lower bound on the pages such a run reads. It is not a bound on its accuracy: dropping a page can lose a right verdict or a wrong one.
+
+165 read sources came from the cache, so the step that read them is unknown; they are kept as read.
+
+## Judge (hypothetical)
+
+The product never lets the judge change a verdict (spec section 11.1). This column is what the accuracy would be if it did: a medium or high model verdict still wins; otherwise the judge's opinions on the escalated sources decide, by majority of the asserting votes (an NEI opinion abstains), a tie being NEI. An escalation the judge did not answer votes with the models' label.
+
+3-way accuracy **0.371**, against 0.371 for the product. 1 sources were escalated; 0 got no opinion back.
+
+| label | n | correct | accuracy |
+|---|---|---|---|
+| Supported | 19 | 0 | 0.000 |
+| Refuted | 63 | 28 | 0.444 |
+| Not Enough Evidence | 7 | 5 | 0.714 |
+
+| judge model | escalation | opinions |
+|---|---|---|
+| groq openai/gpt-oss-120b | nei | 105 |
+| groq openai/gpt-oss-120b | escalated | 1 |
+
+| decided by | n | correct | accuracy |
+|---|---|---|---|
+| models (medium or high tier) | 41 | 28 | 0.683 |
+| nothing escalated | 48 | 5 | 0.104 |
+
+### Band plus NEI probes
+
+`--judge-nei` also sent every read source whose verdict was NEI to the judge, as NEI at low tier, with the passage the models came closest to deciding on. Here those opinions vote too, by the same rule; a probe the judge did not answer abstains.
+
+3-way accuracy **0.427**, against 0.371 for the product. 105 NEI sources were probed; 0 got no opinion back.
+
+Against the product, claim by claim: 5 fixed, 0 broken (exact McNemar, two-sided p = 0.062).
+
+Refuted called SUPPORTED plus Supported called REFUTED, the wrong way: 11 (5 + 6) with the probes, against 11 (5 + 6) for the product.
+
+| label | n | correct | accuracy |
+|---|---|---|---|
+| Supported | 19 | 4 | 0.211 |
+| Refuted | 63 | 29 | 0.460 |
+| Not Enough Evidence | 7 | 5 | 0.714 |
+
+### Dropped by the quote check
+
+A SUPPORTED or REFUTED opinion whose quote is not in the passage it was shown is dropped (OPEN-ITEMS 20.9), and the source then counts as unanswered above.
+
+| escalation | reason | dropped |
+|---|---|---|
+| none | — | 0 |
+
+## Notes
+
+- **Same read pages as the 2026-10-05 gold run.** The 165 read sources are the same pages
+  as in `2026-10-05-averitec-browser-judge-fresh.md`. Every one of them carries no step,
+  which marks a cache hit, so they were most likely served from the fetch cache that run
+  filled. The per-claim predictions are identical, so the product column (0.371) is that
+  run's; only the judge's part is new. The unread sources differ slightly: 8 that were
+  "provider unavailable" there are "unreachable" here. No verdict changes because of it.
+- **What the judge did with the NEIs.** 105 read sources came back NEI and were shown to
+  the judge with their closest passage. It answered NEI on 97, SUPPORTED on 4 and
+  REFUTED on 4. Counting those 8 as votes fixes 5 claims and breaks none: 0.427 against
+  0.371, exact McNemar p = 0.062. The wrong-way count stays at 11 (5 + 6). `Supported`
+  goes from 0/19 to 4/19.
+- **Below the bar set before the run.** OPEN-ITEMS §23 asked for p < 0.05 and no more
+  false SUPPORTED than the product (5 of 63). The second condition holds and the first
+  misses narrowly; with 5 discordant claims the smallest possible p is 0.0625. So this
+  is recorded, not shipped. A larger set (the 300 train claims planned for 20.14) could
+  settle it.
+- **Most NEIs look like something other than a reading failure.** For 97 of the 105,
+  a strong LLM shown the closest passage also finds nothing that settles the claim. That
+  points to the deciding passage not being the closest one, or not being on the page:
+  retrieval and coverage rather than entailment. This is an inference, not a measurement;
+  the judge may also be cautious.
+- **The quote check dropped nothing.** All 106 opinions (105 probes plus 1 escalation)
+  came from `groq openai/gpt-oss-120b` with no fallback, and every asserting opinion's
+  quote was found in its passage. The check costs nothing when the judge complies.
+- Measured with the quote check of OPEN-ITEMS 20.9 (`#q1` cache key), `--no-fallback`,
+  and the default NLI profile.
+

docs/superpowers/OPEN-ITEMS.md

@@ -698,3 +698,30 @@ notice outside the lock. If a concurrent re-switch announces its own notice insi
 microsecond window, the TUI line can show the back notice while `switched` (the
 footer) holds the cooling one. Fix: announce under a separate `_announce_lock`, and only
 when `switched` still equals the notice.
+
+## 23. Judge quote check (20.9) and NEI → judge (20.13) — 2026-10-05
+
+The user asked for both items to be finished end to end, with the best options picked
+without further questions. Design: `docs/superpowers/specs/2026-10-05-judge-quote-and-nei-design.md`.
+
+- **20.9** (product): the review answer gains a `quote` field. A SUPPORTED or REFUTED
+  opinion whose quote is not in the passage, after normalisation and an
+  in-order check of ellipsis parts, is dropped and noted. Cached judgements from
+  before the change are not served as checked.
+- **20.13** (measurement only): `--judge-nei` shows each NEI source's closest passage
+  to the judge, and `--no-fallback` keeps every opinion on Groq. It is a second
+  hypothetical column, compared claim by claim with the product. Shipping it needs
+  p < 0.05 and no more false SUPPORTED than the product (5/63); that stays a separate
+  decision.
+
+**Result (2026-10-05, `docs/eval/2026-10-05-averitec-browser-judge-nei-nofallback.md`).**
+
+- The judge answered 97 of 105 NEI probes NEI, 4 SUPPORTED and 4 REFUTED.
+- Band plus probes: 0.427 against 0.371. 5 claims fixed, 0 broken, exact McNemar
+  p = 0.062. Wrong-way verdicts unchanged at 11.
+- **Not shipped:** p misses the 0.05 bar. With 5 discordant claims, 0.0625 is the floor.
+  Re-measure on the 300 train claims of 20.14 before deciding.
+- Most NEIs look like something other than a reading failure: the closest passage does
+  not settle the claim even for the LLM. That points (an inference, not a measurement)
+  to retrieval and coverage of the deciding passage as the next lever.
+- **20.9 in the live run:** 0 of 106 opinions dropped by the quote check.