diff --git a/CHANGELOG.md b/CHANGELOG.md index 3bcbf19..ab8417c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,15 @@ All notable changes to this project are documented here. The format follows ## [Unreleased] ### Added +- **AVeriTeC eval: `--browser`, `--judge` and `--fresh`** (`scripts/eval_averitec.py`). + Every source now records the ladder step that read it and, under `--judge`, the + judge's opinion when its verdict was in the escalation band. The report adds two + numbers beside the product's: the accuracy with the browser's pages dropped, and a + hypothetical one in which the judge decides the low band. The product never does the + second, and the report says so. Each run setup gets its own results file and report. + Measured on 2026-10-05: gold URLs 0.371; search with sentence queries 0.371; search + with the judge writing the queries 0.483 (14 claims fixed, 4 broken, p = 0.031). The + majority baseline is 0.708. - **`proofpath update`** installs the latest release from PyPI, and **`proofpath update --check`** only says whether there is one. In the TUI they are `/update` and `/update --check`. Running the command is the consent, so nothing diff --git a/README.md b/README.md index 2b522e1..727897d 100644 --- a/README.md +++ b/README.md @@ -410,9 +410,21 @@ if it is not in English) before searching. If the judge is rate-limited, the loc fallback below takes over; only if that fails too is the plain sentence searched, and the report says so. -**This is experimental.** On AVeriTeC's search scenario, proofpath scores -[**0.404**](docs/eval/2026-09-29-averitec-search.md) against a 0.708 majority baseline, and every report from a -run that searched prints that line at the top. +**This is experimental.** On AVeriTeC's search scenario (2026-10-05, browser allowed), +proofpath scores [**0.371**](docs/eval/2026-10-05-averitec-search-browser-fresh.md) with +sentence queries and [**0.483**](docs/eval/2026-10-05-averitec-search-browser-judge-fresh.md) +when the judge writes the queries, against a 0.708 majority baseline. The judge's queries +are the one gain that cleared the noise: on the same 89 claims they fixed 14 and broke 4 +(paired exact McNemar p = 0.031). The share of pages from fact-check sites did not change +(22.0 % against 22.5 %). + +**Search also asserts the wrong way far more often than the cited sources do.** With the +judge's queries, 25 of the 89 claims came back the wrong way, and 20 of those were false +claims called SUPPORTED. With sentence queries the figure was 29, against 11 when the gold +URLs are read. ("Wrong way" counts Supported read as REFUTED and Refuted read as SUPPORTED.) +One likely cause, not yet measured: a page found by searching for a claim often repeats it, +and the models may read the repetition as support. Both scores are still below the +baseline, and every report from a run that searched prints that line at the top. ## Optional LLM judge (v0.3) @@ -480,16 +492,24 @@ switch to, the run says so and tells you how to add one, on the same channels. | Source access | 50 DOIs | 72 % full text, 18 % abstract only, 10 % nothing ([details](docs/eval/2026-09-11-coverage.md)) — a real biomedical paper in the live runs reached 33 % full text | | Citation pairing, numeric | 61 hand-built passages | 0.99 ([details](docs/eval/2026-09-11-pairing.md)) | | Citation pairing, author-year | 55 hand-built passages, 83 expectations | 0.940 ([details](docs/eval/2026-09-12-pairing-author-year.md)) | -| **End to end on real web claims** | AVeriTeC dev, 100 claims | **0.270 3-way accuracy against a 0.708 majority baseline — worse than always guessing "refuted"** ([details](docs/eval/2026-09-16-averitec.md)) | - -**The AVeriTeC row is the one to read before trusting this tool on a news claim.** A third -of those claims had no readable source at all: 32 of the source URLs needed the browser -step, 29 were unreachable, 14 were refused by `robots.txt`. On the claims that *did* have a -readable source the score is 0.361 — still below the baseline, and every one of the 19 -`Supported` claims was missed. The retrieval and entailment models were calibrated on -scientific abstracts, and a fact-check page is a different object: long, discursive, and -usually quoting the claim it debunks. Nothing was tuned after that measurement, and no -blocked URL was dropped from it. +| **End to end on real web claims** | AVeriTeC dev, 100 claims | **0.371 3-way accuracy from the gold source URLs (browser allowed) against a 0.708 majority baseline — still worse than always guessing "refuted"** ([details](docs/eval/2026-10-05-averitec-browser-judge-fresh.md)). With evidence search and a judge writing the queries: 0.483 ([details](docs/eval/2026-10-05-averitec-search-browser-judge-fresh.md)). The first measurement, without the browser, was 0.270 ([2026-09-16](docs/eval/2026-09-16-averitec.md)) | + +**The AVeriTeC row is the one to read before trusting this tool on a news claim.** Coverage +is no longer the main loss. On 2026-10-05, 165 of the 200 gold source URLs were read, +and only 10 of those needed the browser step, worth about two claims, which is within the +noise of 89. The loss is in the reading: + +- 36 of the 89 scorable claims had a readable source and still came back NEI where the + gold label was Supported or Refuted; +- 11 were called the wrong way; +- 7 had nothing readable. + +The likely reason, not separately measured: the retrieval and entailment models were +calibrated on scientific abstracts, and a fact-check page is a different object (long, +discursive, and usually quoting the claim it debunks). +As shipped, the judge cannot help with those NEIs. An NEI verdict carries no passage, so it +is never escalated (rule 1); only 1 of the 165 read sources reached the judge. Nothing was +tuned after these measurements, and no blocked URL was dropped from them. What proofpath is good at is the academic path the other rows measure: finding out whether a cited paper exists, whether it was retracted, and whether its text says what the sentence diff --git a/docs/eval/2026-10-05-averitec-browser-judge-fresh.md b/docs/eval/2026-10-05-averitec-browser-judge-fresh.md new file mode 100644 index 0000000..a0718ae --- /dev/null +++ b/docs/eval/2026-10-05-averitec-browser-judge-fresh.md @@ -0,0 +1,86 @@ +# AVeriTeC dev — 2026-10-05 + +- claims: 100 (limit=100) +- measured path: gold source URLs, no search +- browser step: allowed +- fetch: live, cache bypassed +- judge: groq openai/gpt-oss-120b (its opinions are the hypothetical column below; the verdicts are the models') +- dataset: `https://raw.githubusercontent.com/MichSchli/AVeriTeC/main/data/dev.json` (sha256 499793726b4a…) +- one run of the whole product: real claims, real source pages, real fetch ladder. + +## Headline + +3-way accuracy **0.371** vs majority baseline 0.708, over 89 of 100 claims. + +The 3-way number excludes the Conflicting Evidence/Cherrypicking rows, which proofpath has no verdict for; counting them as wrong gives a 4-way accuracy of 0.330. + +## Per label + +| label | n | correct | accuracy | +|---|---|---|---| +| Supported | 19 | 0 | 0.000 | +| Refuted | 63 | 28 | 0.444 | +| Not Enough Evidence | 7 | 5 | 0.714 | +| Conflicting Evidence/Cherrypicking | 11 | 0 | 0.000 | + +## Source coverage + +| state | count | +|---|---| +| ok | 165 | +| UNVERIFIED (blocked, robots.txt) | 17 | +| UNVERIFIED (provider unavailable) | 10 | +| not a url | 9 | +| UNVERIFIED (blocked) | 4 | +| UNVERIFIED (reached, no text extracted) | 4 | + +76 of 200 source URLs are web.archive.org snapshots (38.0 %). + +## Without the browser + +10 of 165 read sources were read by the browser step. Treating them as unread gives a 3-way accuracy of **0.348**. + +Those pages never reached the Wayback step, which a run without the browser tries next, so this is a lower bound on the pages such a run reads. It is not a bound on its accuracy: dropping a page can lose a right verdict or a wrong one. + +## Judge (hypothetical) + +The product never lets the judge change a verdict (spec section 11.1). This column is what the accuracy would be if it did: a medium or high model verdict still wins; otherwise the judge's opinions on the escalated sources decide, by majority of the asserting votes (an NEI opinion abstains), a tie being NEI. An escalation the judge did not answer votes with the models' label. + +3-way accuracy **0.371**, against 0.371 for the product. 1 sources were escalated; 0 got no opinion back. + +| label | n | correct | accuracy | +|---|---|---|---| +| Supported | 19 | 0 | 0.000 | +| Refuted | 63 | 28 | 0.444 | +| Not Enough Evidence | 7 | 5 | 0.714 | + +| judge model | opinions | +|---|---| +| groq openai/gpt-oss-120b | 1 | + +| decided by | n | correct | accuracy | +|---|---|---|---| +| models (medium or high tier) | 41 | 28 | 0.683 | +| nothing escalated | 48 | 5 | 0.104 | + +## Notes + +- Same 100 dev claims as `2026-09-16-averitec.md` (0.270), but not the same web. That + run used `--no-browser` and read 120 sources; this one read 165 of 200, the browser + step 10 of them. Pages that were unreachable then answer now, so the two numbers + measure different snapshots of the same claims. +- **The browser is worth about two claims here.** Dropping the 10 pages it read gives + 0.348, against 0.371. At 89 claims the standard error is about ±0.05, so this is + within the noise. +- **The judge saw one source.** `pipeline.decide` never gives an NEI verdict a + passage, and `verify.escalates` sends nothing without one (rule 1). 105 of the 165 + read sources are NEI. The hypothetical column therefore equals the product. The judge + never switched to its local fallback. +- **Where the 56 wrong claims are.** 36 had a readable source and came back NEI where + the gold label was Supported or Refuted. 11 were called the wrong way: Refuted read as + Supported 5 times, Supported as Refuted 6 times. 7 had nothing readable, and 2 gold + NEI claims got a verdict. Coverage is no longer the main loss; the reading is. +- Of the 33 claims scored right, one is a gold-NEI claim with nothing readable, which + the score counts as NEI. 32 are verdicts or NEIs on sources that were read. +- Measured with the default NLI profile (pinned), `--fresh`, `--sleep 1.0`. + diff --git a/docs/eval/2026-10-05-averitec-search-browser-fresh.md b/docs/eval/2026-10-05-averitec-search-browser-fresh.md new file mode 100644 index 0000000..9cd4872 --- /dev/null +++ b/docs/eval/2026-10-05-averitec-search-browser-fresh.md @@ -0,0 +1,56 @@ +# AVeriTeC dev — 2026-10-05 + +- claims: 100 (limit=100) +- measured path: sentence queries, no judge +- browser step: allowed +- fetch: live, cache bypassed +- dataset: `https://raw.githubusercontent.com/MichSchli/AVeriTeC/main/data/dev.json` (sha256 499793726b4a…) +- one run of the whole product: real claims, real source pages, real fetch ladder. + +## Headline + +3-way accuracy **0.371** vs majority baseline 0.708, over 89 of 100 claims. + +The 3-way number excludes the Conflicting Evidence/Cherrypicking rows, which proofpath has no verdict for; counting them as wrong gives a 4-way accuracy of 0.330. + +## Per label + +| label | n | correct | accuracy | +|---|---|---|---| +| Supported | 19 | 6 | 0.316 | +| Refuted | 63 | 26 | 0.413 | +| Not Enough Evidence | 7 | 1 | 0.143 | +| Conflicting Evidence/Cherrypicking | 11 | 0 | 0.000 | + +## Source coverage + +| state | count | +|---|---| +| ok | 313 | +| UNVERIFIED (blocked, robots.txt) | 6 | +| UNVERIFIED (blocked) | 5 | +| UNVERIFIED (provider unavailable) | 1 | + +0 of 325 source URLs are web.archive.org snapshots (0.0 %). + +## Without the browser + +18 of 313 read sources were read by the browser step. Treating them as unread gives a 3-way accuracy of **0.348**. + +Those pages never reached the Wayback step, which a run without the browser tries next, so this is a lower bound on the pages such a run reads. It is not a bound on its accuracy: dropping a page can lose a right verdict or a wrong one. + +## Notes + +- Evidence search with sentence queries (no judge), browser allowed, every page fetched + live. The 2026-09-29 run of the same path, without the browser, scored 0.404. The + three-claim gap sits inside the ±0.05 standard error at 89 claims, and the search + results themselves differ between the days. +- The browser read 18 of the 313 pages read. Dropping them gives 0.348. +- Coverage is close to complete: 313 of 325 pages read, 11 blocked (6 by `robots.txt`). +- **Wrong-way verdicts:** 29 of 89. 22 false claims were called SUPPORTED and 7 true + ones REFUTED, against 11 when the gold URLs are read. The count leaves out verdicts + given to gold-NEI claims. One likely cause, not yet measured: a page found by + searching for a claim often repeats it. +- This run is the baseline for `2026-10-05-averitec-search-browser-judge-fresh.md`, + which differs only in who writes the queries. + diff --git a/docs/eval/2026-10-05-averitec-search-browser-judge-fresh.md b/docs/eval/2026-10-05-averitec-search-browser-judge-fresh.md new file mode 100644 index 0000000..a67d6be --- /dev/null +++ b/docs/eval/2026-10-05-averitec-search-browser-judge-fresh.md @@ -0,0 +1,87 @@ +# AVeriTeC dev — 2026-10-05 + +- claims: 100 (limit=100) +- measured path: judge queries (groq openai/gpt-oss-120b) +- browser step: allowed +- fetch: live, cache bypassed +- judge: groq openai/gpt-oss-120b (writes the search queries; its opinions are the hypothetical column below) +- dataset: `https://raw.githubusercontent.com/MichSchli/AVeriTeC/main/data/dev.json` (sha256 499793726b4a…) +- one run of the whole product: real claims, real source pages, real fetch ladder. + +## Headline + +3-way accuracy **0.483** vs majority baseline 0.708, over 89 of 100 claims. + +The 3-way number excludes the Conflicting Evidence/Cherrypicking rows, which proofpath has no verdict for; counting them as wrong gives a 4-way accuracy of 0.430. + +## Per label + +| label | n | correct | accuracy | +|---|---|---|---| +| Supported | 19 | 10 | 0.526 | +| Refuted | 63 | 31 | 0.492 | +| Not Enough Evidence | 7 | 2 | 0.286 | +| Conflicting Evidence/Cherrypicking | 11 | 0 | 0.000 | + +## Source coverage + +| state | count | +|---|---| +| ok | 319 | +| UNVERIFIED (blocked, robots.txt) | 9 | +| UNVERIFIED (blocked) | 4 | + +0 of 332 source URLs are web.archive.org snapshots (0.0 %). + +## Without the browser + +17 of 319 read sources were read by the browser step. Treating them as unread gives a 3-way accuracy of **0.483**. + +Those pages never reached the Wayback step, which a run without the browser tries next, so this is a lower bound on the pages such a run reads. It is not a bound on its accuracy: dropping a page can lose a right verdict or a wrong one. + +## Judge (hypothetical) + +The product never lets the judge change a verdict (spec section 11.1). This column is what the accuracy would be if it did: a medium or high model verdict still wins; otherwise the judge's opinions on the escalated sources decide, by majority of the asserting votes (an NEI opinion abstains), a tie being NEI. An escalation the judge did not answer votes with the models' label. + +3-way accuracy **0.483**, against 0.483 for the product. 1 sources were escalated; 0 got no opinion back. + +| label | n | correct | accuracy | +|---|---|---|---| +| Supported | 19 | 10 | 0.526 | +| Refuted | 63 | 31 | 0.492 | +| Not Enough Evidence | 7 | 2 | 0.286 | + +| judge model | opinions | +|---|---| +| groq openai/gpt-oss-120b | 1 | + +| decided by | n | correct | accuracy | +|---|---|---|---| +| models (medium or high tier) | 71 | 41 | 0.577 | +| nothing escalated | 18 | 2 | 0.111 | + +## Notes + +- **The one gain that cleared the noise.** The only difference from + `2026-10-05-averitec-search-browser-fresh.md` (0.371) is who writes the search + queries: here the judge, `groq openai/gpt-oss-120b`. Paired over the same 89 claims, + the judge's queries were right where the sentence queries were wrong 14 times, and + the reverse 4 times (exact McNemar p = 0.031). `Supported` went from 6/19 to 10/19. +- **The share of fact-check pages did not change.** By a host/URL heuristic, they were + 22.0 % of the pages here and 22.5 % with sentence queries. That says the mix did not + shift, not that no better fact-check page was found. The checker's own site is + excluded in both, as in every search run. +- **Many claims still come back the wrong way.** 25 of the 89 claims came back the wrong + way: 20 false claims were called SUPPORTED, and 5 true ones REFUTED. Reading the gold + URLs gives 11. The count leaves out verdicts given to gold-NEI claims. One likely + cause, not yet measured: a page found by searching for a claim often repeats it. +- **One risk this run cannot rule out.** A model writing the queries may carry what it + remembers about a claim into them. The verdicts still come from the pages read, each + with its passage, but the choice of pages may not be blind. +- All queries were written by Groq. The judge never switched to its local fallback; + the run printed no switch notice. +- The browser read 17 of the 319 pages read; dropping them leaves the score at 0.483. + The hypothetical judge column equals the product: one source was escalated (see the + gold-URL run's note on NEI verdicts). +- Still below the 0.708 majority baseline, so the search stays experimental. + diff --git a/docs/superpowers/OPEN-ITEMS.md b/docs/superpowers/OPEN-ITEMS.md index 299764d..b8263f8 100644 --- a/docs/superpowers/OPEN-ITEMS.md +++ b/docs/superpowers/OPEN-ITEMS.md @@ -607,10 +607,64 @@ five packages. The user's decisions: | 20.7 | Skip model loads on a fully cached re-run | same as 10.2 | | 20.8 | Bibliography parsing that was measured but never shipped | two-column rejoin (9.10/12.1), multi-block headless fallback (11.11), more headings (16.5), `[1]-[3]` ranges (9.4) | | 20.9 | The judge's quote is never checked | `judge._collect` does not verify that the quoted span occurs in the passage. Checking it is a cheap way to enforce product rule 1 | -| 20.10 | Eval honesty | add a `--judge` flag to `scripts/eval_averitec.py`; do one run with the browser allowed; show the 0.404 search score in the README "Measured" table | +| 20.10 | Eval honesty | **done 2026-10-05**: `--browser`, `--judge`, `--fresh`; three runs and their results are under "20.10 results" below | | 20.11 | NLI bake-off result | **Winner: `large-fever` (MoritzLaurer DeBERTa-v3-large mnli-fever-anli-ling-wanli) × k=2 × `max`.** SciFact dev macro-F1 0.697, against 0.580 for the `base` × k=1 × `max` baseline and 0.608 for the best `base` row. Accuracy 0.697. AVeriTeC 3-way 0.438, or 0.475 on the readable subset, against a 0.708 majority baseline. Cost: 643 MB and 54 ms/pair, against 244 MB and 19 ms/pair. Full report: `docs/eval/2026-10-01-nli-bakeoff.md`. Next: (a) and (b) are settled by the accurate-NLI design (`docs/superpowers/specs/2026-10-01-accurate-nli-design.md`), which adds the winner as an opt-in "accurate" profile behind a consent download (spec §7.1); (c) stays open. (a) **settled** — rule-decided rows are left out of the tier walk (the numeric layer fired 4–6 times on train with 0 correct and pinned `high` at 1.000000); the accurate cuts are `high` 0.999142. (b) **settled** — the `medium` cut still sits on `decide`, and the report now says so itself (`NO_LOW_TIER`, default runs included). (c) **still open** — on AVeriTeC the winner is below `base` × k=2 (0.475 against 0.537), and `Supported` is still 2/19; the README recommends the profile for scientific sources, not for news | | 20.12 | Accurate-NLI follow-ups | (1) A shared `browser.read_terminal_answer` helper: `model_gate.ask_model_terminal` repeats `browser.ask_terminal`'s prompt loop and imports the private `_CHOICE_ANSWERS` to do it. (2) The `-q` `nli` line carries the fallback reason but not the advice. (3) Ctrl-C and `/stop` do not interrupt a running 643 MB download. (4) An "installed" but corrupt accurate model fails the run instead of falling back. (5) A multilingual NLI model is still deferred (out of scope in the accurate-NLI design); non-English claims still need the judge to translate | +### 20.10 decisions — 2026-10-05 + +The user picked the recommended option on all four questions. + +- **The judge is scored as a hypothetical column.** The product never lets the judge change + a verdict (spec §11.1), so with gold URLs `--judge` cannot move the product's number. The + report keeps the product's accuracy and adds a second one, labelled as something the + product does not do. In it, a confident model verdict (medium or high tier) still wins. + Otherwise the judge's opinions on the escalated sources decide (`verify.escalates`). + A majority of the asserting votes picks the label: an NEI opinion abstains, and a + tie is NEI. An escalation the judge did not answer votes with the models' label. +- **Three runs and an ablation.** (1) gold URLs, browser, judge. (2) search, browser, + sentence queries. (3) search, browser, judge queries. Every source records the ladder + step that read it. The "without the browser" number comes from the same run, with the + step-3 sources dropped. Wayback was never tried for those pages, so the ablation is a + lower bound on the pages such a run reads. It is not a bound on its accuracy, which can + move either way. Runs fetch live (`--fresh`), because a cache hit has no step. Each + setup gets its own results file and report. +- **Rate limits.** When Groq rate-limits, the judge falls back to the local + `qwen3.5:9b`, as the product does. The report counts opinions per model and splits the + hypothetical accuracy by the model whose vote won. +- **Git.** One feature branch and one PR: the script, its tests, the `docs/eval` + reports and the README "Measured" row. + +### 20.10 results — 2026-10-05 + +| Run | 3-way | Without the browser | Judge (hypothetical) | +|---|---|---|---| +| Gold URLs, browser, judge | 0.371 | 0.348 | 0.371 | +| Search, browser, sentence queries | 0.371 | 0.348 | — | +| Search, browser, judge queries (Groq `gpt-oss-120b`) | **0.483** | 0.483 | 0.483 | + +All three are against a 0.708 majority baseline over 89 claims. Reports: +`docs/eval/2026-10-05-averitec-*.md`. + +- **Coverage is no longer the main loss.** 165 of 200 gold sources were read. The + browser was worth about two claims, within the ±0.05 noise. +- **The reading is the loss.** 36 of 89 claims had a readable source and came back NEI, + and 11 went the wrong way. +- **The judge's queries are the one gain that cleared the noise.** On the same claims + they fixed 14 and broke 4 (McNemar p = 0.031). The share of fact-check pages did not + change. +- **Search asserts the wrong way much more.** 25 of 89 claims (judge queries) and 29 of + 89 (sentence queries) came back the wrong way, mostly false claims called SUPPORTED. + The gold URLs give 11 (new item 20.15). +- **The judge cannot see NEIs** (new item 20.13). This is why the hypothetical column + equals the product. + +| # | Item | Note | +|---|---|---| +| 20.13 | The judge never sees an NEI | `pipeline.decide` returns NEI with `passage=None`, so `verify.escalates` never sends it (rule 1). Yet NEI-on-a-read-source is 36 of the 56 wrong claims on AVeriTeC. Measure first: give the judge the best passage of each NEI source, verify its quote (20.9), and score it as a hypothetical column. Run it without the fallback so it measures Groq alone. Only then decide whether the product should escalate NEI with its best passage | +| 20.14 | Web cuts | Fit `decide`/`medium` on AVeriTeC **train** and report them on dev, as the accurate model's cuts were fitted on SciFact train. This targets the same NEI slice without touching rule 1 | +| 20.15 | Search calls more false claims SUPPORTED | With search, 20–22 of the 63 false claims come back SUPPORTED, against 5 from the gold URLs. Hypothesis, not yet measured: a page found by searching for a claim often quotes it, and the models read the quote as support. Check that on the failing rows first. Candidate fix: drop passages that restate the claim (near-duplicate of the hypothesis) before entailment, and measure it on AVeriTeC train first. This bears on rule 3's spirit: a wrong SUPPORTED is the costliest error | + ## 21. `proofpath update` — 2026-10-05 The user asked for an update command like `claude update`. They picked the recommended diff --git a/scripts/eval_averitec.py b/scripts/eval_averitec.py index 48f0d5f..a1ecd4e 100644 --- a/scripts/eval_averitec.py +++ b/scripts/eval_averitec.py @@ -1,8 +1,9 @@ """Phase 10 harness: measure proofpath end to end on AVeriTeC dev (spec section 14). Usage: - uv run python scripts/eval_averitec.py [--limit 100] [--no-browser] [--sleep 1.0] - [--resume] [--search] [--out docs/eval/-averitec.md] + uv run python scripts/eval_averitec.py [--limit 100] [--browser | --no-browser] + [--judge] [--fresh] [--sleep 1.0] [--resume] [--search] + [--out docs/eval/-averitec.md] SciFact measures retrieval and entailment over abstracts we are handed. AVeriTeC measures the whole product: a real-world claim, its real source pages, fetched over @@ -17,6 +18,13 @@ answer. Source coverage is reported per state, never collapsed (product rule 6): a run that reached nothing must not read like a run that reached everything. +Every source keeps the ladder step that read it and, under ``--judge``, the judge's +opinion when its verdict was in the escalation band. Two more numbers come from +those rows: the accuracy with the browser's pages dropped (what a run without it +would have read, less whatever Wayback would have rescued), and a hypothetical one +in which the judge's opinions decide the low band. The product never does the +second (spec section 11.1); the report says so. + The live half below ``main`` is run by hand. Importing this module touches neither the network nor a model. """ @@ -28,8 +36,8 @@ import sys import time from collections import Counter -from collections.abc import Sequence -from dataclasses import dataclass, replace +from collections.abc import Callable, Sequence +from dataclasses import asdict, dataclass, replace from datetime import date from pathlib import Path from typing import TYPE_CHECKING @@ -37,16 +45,18 @@ from proofpath import claims as claims_mod from proofpath import ingest, pipeline, retrieval +from proofpath import judge as judge_mod from proofpath.config import NliProfileName, SearchConfig, load_config from proofpath.eval import averitec from proofpath.fetch import Fetched -from proofpath.models import Label, Passage +from proofpath.models import Label, Passage, Verdict from proofpath.paths import cache_dir from proofpath.polite import ProviderError +from proofpath.providers import NO_TEXT from proofpath.report import LANGUAGE_UNSUPPORTED, SEARCH_UNAVAILABLE from proofpath.search import Searcher, SearchKeyError, canonical, search_claim from proofpath.search.queries import plan_queries -from proofpath.verify import NO_TEXT, Engine +from proofpath.verify import Engine, escalates if TYPE_CHECKING: from proofpath.judge import Judge @@ -69,6 +79,36 @@ SENTENCE_PATH = "sentence queries, no judge" RESULTS_NAME = "averitec_results.json" SEARCH_RESULTS_NAME = "averitec_search_results.json" +# The ladder step the ablation drops (``fetch.STEP_NAMES``). +BROWSER_STEP = 3 +# Who decided a claim's hypothetical judged label, when it was not a judge model. +DECIDED_BY_MODELS = "models (medium or high tier)" +NO_OPINION = "models (the judge gave no opinion)" +NOTHING_ESCALATED = "nothing escalated" +# A row from a results file written before sources were kept: only its label is known. +UNRECORDED = "not recorded (row has no per-source record)" + + +@dataclass(frozen=True) +class Source: + """One fetchable source of one claim, as the run read it. + + ``step`` is the ladder step that read the page, or the last one tried when it + failed; ``None`` for a cache hit, which carries no step. ``label`` is the models' + verdict, ``None`` when nothing was read. ``escalated`` says the verdict sat in the + band the product hands the judge (``verify.escalates``); ``judge_label`` and + ``judge_model`` are the judge's opinion of it, when one came back. + """ + + url: str + state: str + step: int | None + label: str | None + score: float = 0.0 + tier: str | None = None + escalated: bool = False + judge_label: str | None = None + judge_model: str | None = None @dataclass(frozen=True) @@ -86,6 +126,78 @@ class Row: predicted: str | None states: tuple[str, ...] # one honesty/fetch state per source value urls: tuple[str, ...] # the fetchable URLs the states came from, in fetch order + # One per fetchable URL. Empty in results files written before 2026-10-05. + sources: tuple[Source, ...] = () + + +def _best(sources: Sequence[Source]) -> Source: + """The strongest source, the first one on a tie, as ``_decide_claim`` always kept.""" + best = sources[0] + for source in sources[1:]: + if source.score > best.score: + best = source + return best + + +def _asserts(label: str | None) -> bool: + return label is not None and label != Label.NEI.value + + +def product_label(sources: Sequence[Source]) -> str | None: + """What the product says of the claim: its strongest assertion over every source, + NEI when something was read and nothing asserted, ``None`` when nothing was read.""" + asserted = [source for source in sources if _asserts(source.label)] + if asserted: + return _best(asserted).label + return Label.NEI.value if any(source.label is not None for source in sources) else None + + +def judged_label(sources: Sequence[Source]) -> str | None: + """The claim's label if the judge's opinions decided the low band. + + The product never does this (spec section 11.1): this is the column that says + whether it should. A confident model verdict (medium or high, not escalated) + still wins, as the escalation rule implies. Otherwise every escalated source + votes with the judge's label, or with the models' label when the judge gave no + opinion. An NEI vote abstains; the majority of the asserting votes wins, and a + tie is NEI. + """ + confident = [s for s in sources if _asserts(s.label) and not s.escalated] + if confident: + return _best(confident).label + votes = [s.judge_label or s.label for s in sources if s.escalated] + tally = Counter(vote for vote in votes if _asserts(vote)).most_common() + if tally: + if len(tally) > 1 and tally[0][1] == tally[1][1]: + return Label.NEI.value + return tally[0][0] + return Label.NEI.value if any(source.label is not None for source in sources) else None + + +def judge_decider(sources: Sequence[Source]) -> str: + """Who decided ``judged_label``: the models, or the judge model(s) whose vote won. + + When the vote ends in NEI (every vote abstained, or a tie) no vote won, and every + escalated source's answerer is named: together they are what left it at NEI.""" + if any(_asserts(s.label) and not s.escalated for s in sources): + return DECIDED_BY_MODELS + escalated = [s for s in sources if s.escalated] + if not escalated: + return NOTHING_ESCALATED + winner = judged_label(sources) + voters = [s for s in escalated if (s.judge_label or s.label) == winner and _asserts(winner)] + return " + ".join(sorted({s.judge_model or NO_OPINION for s in voters or escalated})) + + +def without_browser(sources: Sequence[Source]) -> tuple[Source, ...]: + """The sources as a run without the browser would have them: the pages the browser + read are unread. A cache hit has no step, so it is kept (the report counts those).""" + return tuple( + replace(source, label=None, escalated=False, judge_label=None, judge_model=None) + if source.step == BROWSER_STEP and source.label is not None + else source + for source in sources + ) def _is_archive(url: str) -> bool: @@ -190,6 +302,33 @@ def _site(url: str) -> str: return (urlsplit(url).hostname or "").removeprefix("www.") +@dataclass(frozen=True) +class Ablation: + """The run with the browser's pages dropped. A page the browser read was never + offered to the Wayback step, which a run without the browser tries next, so this + is a lower bound on the pages such a run reads -- not on its accuracy, which can + move either way (a wrong verdict dropped can turn a claim into a right NEI).""" + + read_sources: int + browser_sources: int # of ``read_sources``, read at ``BROWSER_STEP`` + unknown_step: int # of ``read_sources``, cache hits: kept, since no step is known + unrecorded: int # rows with URLs and no per-source record: scored as the product did + accuracy_3way: float + + +@dataclass(frozen=True) +class JudgedScore: + """The hypothetical column: what ``judged_label`` would have scored.""" + + accuracy_3way: float + per_label: dict[str, tuple[int, int]] # gold label -> (n, correct) + escalated: int # sources in the judge's band + unanswered: int # of those, how many got no opinion back + opinions: Counter[str] # judge model -> opinions it gave + deciders: dict[str, tuple[int, int]] # ``judge_decider`` -> (3-way n, correct) + unrecorded: int # rows with URLs and no per-source record: scored as the product did + + @dataclass(frozen=True) class AveritecResult: n: int @@ -203,9 +342,82 @@ class AveritecResult: coverage: Counter[str] # source state -> how many values ended there archive_urls: int # of ``total_urls``, how many were web.archive.org snapshots total_urls: int + ablation: Ablation | None = None # ``None`` when no row carries its sources + judged: JudgedScore | None = None # ``None`` unless the run had a judge -def score(rows: Sequence[Row]) -> AveritecResult: +def _three_way( + rows: Sequence[Row], label_of: Callable[[Row], str | None] +) -> tuple[float, dict[str, tuple[int, int]]]: + """3-way accuracy and per-gold-label (n, correct) for another way of labelling the + same rows, scored by the same rule as ``score``: no label is NEI.""" + per_label: dict[str, tuple[int, int]] = {} + counted = correct = 0 + for row in rows: + gold = averitec.to_label(row.gold) + if gold is None: + continue + predicted = label_of(row) + right = (Label(predicted) if predicted is not None else Label.NEI) is gold + seen, hits = per_label.get(row.gold, (0, 0)) + per_label[row.gold] = (seen + 1, hits + int(right)) + counted += 1 + correct += int(right) + return (correct / counted if counted else 0.0), per_label + + +def _unrecorded(row: Row) -> bool: + """Written before sources were kept (a ``--resume`` of an older file): the row has + URLs but no per-source record, so the product's own label is all that is known.""" + return not row.sources and bool(row.urls) + + +def _ablation(rows: Sequence[Row]) -> Ablation | None: + if not any(row.sources for row in rows): + return None + read = [s for row in rows for s in row.sources if s.label is not None] + accuracy, _ = _three_way( + rows, + lambda row: product_label(without_browser(row.sources)) if row.sources else row.predicted, + ) + return Ablation( + read_sources=len(read), + browser_sources=sum(1 for s in read if s.step == BROWSER_STEP), + unknown_step=sum(1 for s in read if s.step is None), + unrecorded=sum(1 for row in rows if _unrecorded(row)), + accuracy_3way=accuracy, + ) + + +def _judged_label_of(row: Row) -> str | None: + return judged_label(row.sources) if row.sources else row.predicted + + +def _judged(rows: Sequence[Row]) -> JudgedScore: + accuracy, per_label = _three_way(rows, _judged_label_of) + escalated = [s for row in rows for s in row.sources if s.escalated] + deciders: dict[str, tuple[int, int]] = {} + for row in rows: + gold = averitec.to_label(row.gold) + if gold is None: + continue + predicted = _judged_label_of(row) + right = (Label(predicted) if predicted is not None else Label.NEI) is gold + decider = UNRECORDED if _unrecorded(row) else judge_decider(row.sources) + seen, hits = deciders.get(decider, (0, 0)) + deciders[decider] = (seen + 1, hits + int(right)) + return JudgedScore( + accuracy_3way=accuracy, + per_label=per_label, + escalated=len(escalated), + unanswered=sum(1 for s in escalated if s.judge_label is None), + opinions=Counter(s.judge_model for s in escalated if s.judge_model is not None), + deciders=deciders, + unrecorded=sum(1 for row in rows if _unrecorded(row)), + ) + + +def score(rows: Sequence[Row], *, judge: bool = False) -> AveritecResult: """Turn decided rows into the numbers the report prints. A row with no prediction counts as NEI: having fetched nothing, NEI is the only @@ -245,6 +457,8 @@ def score(rows: Sequence[Row]) -> AveritecResult: coverage=coverage, archive_urls=archive_urls, total_urls=total_urls, + ablation=_ablation(rows), + judged=_judged(rows) if judge else None, ) @@ -252,8 +466,21 @@ def _md_row(*cells: object) -> str: return "| " + " | ".join(str(c) for c in cells) + " |" -def render_report(result: AveritecResult, *, date: str, limit: int, path: str = GOLD_PATH) -> str: - """The markdown skeleton. ``## Notes`` is left for the controller to fill in.""" +def render_report( + result: AveritecResult, + *, + date: str, + limit: int, + path: str = GOLD_PATH, + browser: bool | None = None, + fresh: bool = False, + judge: str = "", +) -> str: + """The markdown skeleton. ``## Notes`` is left for the controller to fill in. + + ``browser`` is ``None`` when the run left the browser to the config, so the report + claims nothing about it; ``judge`` is the judge's name, empty when there was none. + """ # ``result.counted`` is the denominator the 3-way accuracy and its baseline are # read over. Printed beside `n`, because "0.80 over 100 claims" would otherwise # claim a coverage the number does not have when some claims were never scorable. @@ -262,6 +489,18 @@ def render_report(result: AveritecResult, *, date: str, limit: int, path: str = "", f"- claims: {result.n} (limit={limit or 'none'})", f"- measured path: {path}", + ] + if browser is not None: + lines.append(f"- browser step: {'allowed' if browser else 'not used'}") + lines.append(f"- fetch: {'live, cache bypassed' if fresh else 'cache allowed'}") + if judge: + role = ( + "its opinions are the hypothetical column below; the verdicts are the models'" + if path == GOLD_PATH + else "writes the search queries; its opinions are the hypothetical column below" + ) + lines.append(f"- judge: {judge} ({role})") + lines += [ f"- dataset: `{averitec.URL}` (sha256 {averitec.SHA256[:12]}…)", "- one run of the whole product: real claims, real source pages, real fetch ladder.", "", @@ -298,22 +537,115 @@ def render_report(result: AveritecResult, *, date: str, limit: int, path: str = # report says how much of its own coverage came from one. lines.append("") if result.total_urls: - share = 100.0 * result.archive_urls / result.total_urls + percent = 100.0 * result.archive_urls / result.total_urls lines.append( f"{result.archive_urls} of {result.total_urls} source URLs are " - f"{ARCHIVE_HOST} snapshots ({share:.1f} %)." + f"{ARCHIVE_HOST} snapshots ({percent:.1f} %)." ) else: lines.append(f"No source URLs were measured, so no {ARCHIVE_HOST} share applies.") + if result.ablation is not None: + lines.extend(_ablation_lines(result.ablation)) + if result.judged is not None: + lines.extend(_judged_lines(result.judged, product=result.accuracy_3way)) lines.extend(["", "## Notes", "", "", ""]) return "\n".join(lines) + "\n" +def _ablation_lines(ablation: Ablation) -> list[str]: + lines = [ + "", + "## Without the browser", + "", + f"{ablation.browser_sources} of {ablation.read_sources} read sources were read by " + f"the browser step. Treating them as unread gives a 3-way accuracy of " + f"**{ablation.accuracy_3way:.3f}**.", + "", + "Those pages never reached the Wayback step, which a run without the browser " + "tries next, so this is a lower bound on the pages such a run reads. It is not a " + "bound on its accuracy: dropping a page can lose a right verdict or a wrong one.", + ] + if ablation.unknown_step: + lines.extend( + [ + "", + f"{ablation.unknown_step} read sources came from the cache, so the step " + "that read them is unknown; they are kept as read.", + ] + ) + if ablation.unrecorded: + lines.extend(["", _unrecorded_line(ablation.unrecorded)]) + return lines + + +def _unrecorded_line(count: int) -> str: + return ( + f"{count} rows have no per-source record (written before sources were kept); " + "they are scored with the product's own label." + ) + + +def _judged_lines(judged: JudgedScore, *, product: float) -> list[str]: + lines = [ + "", + "## Judge (hypothetical)", + "", + "The product never lets the judge change a verdict (spec section 11.1). This " + "column is what the accuracy would be if it did: a medium or high model verdict " + "still wins; otherwise the judge's opinions on the escalated sources decide, by " + "majority of the asserting votes (an NEI opinion abstains), a tie being NEI. An " + "escalation the judge did not answer votes with the models' label.", + "", + f"3-way accuracy **{judged.accuracy_3way:.3f}**, against {product:.3f} for the " + f"product. {judged.escalated} sources were escalated; {judged.unanswered} got " + "no opinion back." + + (f" {_unrecorded_line(judged.unrecorded)}" if judged.unrecorded else ""), + "", + "| label | n | correct | accuracy |", + "|---|---|---|---|", + ] + for label in [label for label in averitec.LABELS if label in judged.per_label]: + seen, right = judged.per_label[label] + lines.append(_md_row(label, seen, right, f"{right / seen:.3f}" if seen else "—")) + lines.extend(["", "| judge model | opinions |", "|---|---|"]) + for model, count in judged.opinions.most_common(): + lines.append(_md_row(model, count)) + if not judged.opinions: + lines.append(_md_row("none", 0)) + lines.extend(["", "| decided by | n | correct | accuracy |", "|---|---|---|---|"]) + for decider, (seen, right) in sorted(judged.deciders.items()): + lines.append(_md_row(decider, seen, right, f"{right / seen:.3f}" if seen else "—")) + return lines + + # --- live half: run by hand, not covered by tests ------------------------ -def _results_path(*, search: bool) -> Path: - return cache_dir() / "datasets" / (SEARCH_RESULTS_NAME if search else RESULTS_NAME) +def _setup_suffix(*, browser: bool, judge: bool, fresh: bool) -> str: + """What sets this run apart from the legacy setup, for its file names.""" + return ( + ("-browser" if browser else "") + ("-judge" if judge else "") + ("-fresh" if fresh else "") + ) + + +def _results_path( + *, search: bool, browser: bool = False, judge: bool = False, fresh: bool = False +) -> Path: + """One file per run setup, so a resume does not mix rows from two setups. The + legacy names stay the runs made without ``--browser``, ``--judge`` or ``--fresh``; + they are shared by ``--no-browser`` and by a run that leaves the browser to the + config, as they always were, so that older files can still be resumed.""" + name = Path(SEARCH_RESULTS_NAME if search else RESULTS_NAME) + suffix = _setup_suffix(browser=browser, judge=judge, fresh=fresh) + return cache_dir() / "datasets" / f"{name.stem}{suffix}{name.suffix}" + + +def _report_path(*, day: str, search: bool, browser: bool, judge: bool, fresh: bool) -> Path: + """The default ``--out``: one report per run setup, so two runs on one day keep both.""" + suffix = ("-search" if search else "") + _setup_suffix( + browser=browser, judge=judge, fresh=fresh + ) + return Path("docs/eval") / f"{day}-averitec{suffix}.md" def _load_rows(path: Path) -> list[Row]: @@ -327,6 +659,7 @@ def _load_rows(path: Path) -> list[Row]: predicted=None if item["predicted"] is None else str(item["predicted"]), states=tuple(str(state) for state in item["states"]), urls=tuple(str(url) for url in item["urls"]), + sources=tuple(Source(**source) for source in item.get("sources", ())), ) for item in raw ] @@ -343,6 +676,7 @@ def _save_rows(path: Path, rows: Sequence[Row]) -> None: "predicted": row.predicted, "states": list(row.states), "urls": list(row.urls), + "sources": [asdict(source) for source in row.sources], } for row in rows ], @@ -354,26 +688,38 @@ def _save_rows(path: Path, rows: Sequence[Row]) -> None: def _decide_claim( - engine: Engine, claim: averitec.Claim, *, sleep: float, anonymous: bool = False + engine: Engine, + claim: averitec.Claim, + *, + sleep: float, + anonymous: bool = False, + fresh: bool = False, + judge: Judge | None = None, ) -> Row: """Fetch every source of one claim and keep the strongest non-NEI verdict. ``anonymous`` is for pages the search found: fetched without the contact address, - as the product fetches them (round 2).""" + as the product fetches them (round 2). ``fresh`` climbs past the cache, so every + source keeps the step that read it. ``judge`` is asked about the sources whose + verdict sits in the product's escalation band; its opinions are recorded beside + the verdicts and never change ``predicted`` (spec section 11.1).""" states: list[str] = [] - best: pipeline.Verdict | None = None - fetched_any = False + sources: list[Source] = [] + # What the judge is shown for each escalated source, by its index in ``sources``. + escalated: dict[int, Verdict] = {} for url in claim.source_urls: # Before every fetch, not between them: the ladder has no crawl delay of its # own, so this is the whole of the run's politeness and it has to hold across # claims too, not just within one. if sleep: time.sleep(sleep) - fetched = engine.fetcher.fetch(url, anonymous=anonymous) - states.append(state_for(fetched)) + fetched = engine.fetcher.fetch(url, anonymous=anonymous, use_cache=not fresh) + state = state_for(fetched) + states.append(state) + step = None if fetched.from_cache else fetched.step if not fetched.ok or not fetched.text.strip(): + sources.append(Source(url=url, state=state, step=step, label=None)) continue - fetched_any = True passages = [ Passage(sentence, url, ordinal) for ordinal, sentence in enumerate(retrieval.split_sentences(fetched.text)) @@ -386,41 +732,91 @@ def _decide_claim( k=engine.k, thresholds=engine.thresholds, ) - # No passage, no verdict (product rule 1): an assertion without the sentence - # it rests on is dropped rather than reported. - if verdict.label is Label.NEI or verdict.passage is None: - continue - if best is None or verdict.score > best.score: - best = verdict + if escalates(verdict): + escalated[len(sources)] = verdict + # A source whose verdict is NEI was still read: it stays in the row, and + # ``product_label`` keeps only assertions, each with the passage it rests on + # (product rule 1, enforced by ``Verdict`` itself). + sources.append( + Source( + url=url, + state=state, + step=step, + label=verdict.label.value, + score=verdict.score, + tier=verdict.tier, + escalated=escalates(verdict), + ) + ) + if judge is not None and escalated: + sources = _ask_judge(judge, claim.text, sources, escalated) # One state per value the dataset gave, fetchable or not, so the coverage table # accounts for every annotation rather than only the ones we could try. states.extend([NOT_A_URL] * claim.non_urls) if not claim.source_urls and not claim.non_urls: states.append(NO_SOURCE) - if best is not None: - predicted: str | None = best.label.value - elif fetched_any: - predicted = Label.NEI.value - else: - predicted = None return Row( claim_id=claim.id, gold=claim.label, - predicted=predicted, + predicted=product_label(sources), states=tuple(states), urls=claim.source_urls, + sources=tuple(sources), ) +def _ask_judge( + judge: Judge, claim: str, sources: list[Source], escalated: dict[int, Verdict] +) -> list[Source]: + """The judge's opinion on each escalated source, shown exactly what ``verify`` + shows it: the claim, the passage, the models' verdict and its tier.""" + items = [ + judge_mod.JudgeItem( + id=f"s{index}", + claim=claim, + passage=verdict.passage.text, + verdict=verdict.label, + tier=verdict.tier, + ) + for index, verdict in escalated.items() + if verdict.passage is not None # ``escalates`` already refused the others + ] + opinions = judge.review(items) + answered = list(sources) + for index in escalated: + opinion = opinions.get(f"s{index}") + if opinion is not None: + answered[index] = replace( + sources[index], judge_label=opinion.label.value, judge_model=opinion.model + ) + return answered + + NLI_PROFILE: NliProfileName = "default" def main(argv: list[str] | None = None) -> int: parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("--limit", type=int, default=100, help="first N dev claims") - parser.add_argument( + browser_flags = parser.add_mutually_exclusive_group() + browser_flags.add_argument( "--no-browser", action="store_true", help="never use the browser step of the ladder" ) + browser_flags.add_argument( + "--browser", + action="store_true", + help="allow the browser step whatever the config says (installs it if missing)", + ) + parser.add_argument( + "--judge", + action="store_true", + help="ask the configured judge about the escalated verdicts (a hypothetical column)", + ) + parser.add_argument( + "--fresh", + action="store_true", + help="fetch every page live, past the cache, so each source keeps its ladder step", + ) parser.add_argument("--sleep", type=float, default=1.0, help="seconds between fetches") parser.add_argument("--resume", action="store_true", help="skip claim ids already scored") parser.add_argument( @@ -431,7 +827,9 @@ def main(argv: list[str] | None = None) -> int: parser.add_argument("--out", default="") args = parser.parse_args(argv) - results = _results_path(search=args.search) + results = _results_path( + search=args.search, browser=args.browser, judge=args.judge, fresh=args.fresh + ) if results.exists() and not args.resume: # The file is the only record of a run that costs hours of network; a fresh # run must not quietly replace it. Checked before anything is downloaded, so @@ -443,6 +841,21 @@ def main(argv: list[str] | None = None) -> int: ) return 2 + config = load_config() + key: judge_mod.ApiKey | None = None + if args.judge: + # Checked before anything is downloaded, as the CLI's ``--judge`` checks it: a + # run told to measure the judge must not quietly measure the models alone. + key = judge_mod.resolve_api_key(config.judge.api_key_env) + if config.judge.api_key_env and key is None: + print( + f"error {config.judge.api_key_env} is not set (put it in .env)", + file=sys.stderr, + ) + return 2 + if config.judge.provider == "gemini": + print(f"note {judge_mod.GEMINI_DATA_USE}", file=sys.stderr) + dataset = averitec.ensure_downloaded(cache_dir()) claims = averitec.load(dataset, limit=args.limit or None) print(f"dataset averitec/dev {len(claims)} claims (sha256 {averitec.SHA256[:12]}…)") @@ -455,15 +868,22 @@ def main(argv: list[str] | None = None) -> int: if done: print(f"resume {len(done)} claims already scored in {results}") - config = load_config() # ``interactive=False`` so an `ask` permission is denied and reported rather than # prompted for (product rule 4); ``--no-browser`` forces the browser step off - # outright, otherwise the config's own permission decides. + # outright and ``--browser`` on (asking for it is the consent, spec section 7.1); + # otherwise the config's own permission decides. # ``nli="default"`` is pinned: the published numbers describe the default install, # so the user's ``models.nli`` must not change what this script measures. + browser = True if args.browser else False if args.no_browser else None + # Built here, not where the key is checked: ``Engine.default`` owns it from this + # call on and closes it even when it fails, so nothing in between can leak it. + judge = judge_mod.Judge(judge_mod.build_client(config.judge, key)) if args.judge else None engine = Engine.default( - config, interactive=False, browser=False if args.no_browser else None, nli=NLI_PROFILE + config, interactive=False, browser=browser, nli=NLI_PROFILE, judge=judge ) + # The configured judge's name, taken before it can switch to its local fallback: + # the report names the judge the run was told to use, the table who answered. + judge_name = "" if judge is None else judge.name print(f"nli {NLI_PROFILE} profile (pinned; ignores models.nli)") searcher: Searcher | None = None if args.search: @@ -481,6 +901,7 @@ def main(argv: list[str] | None = None) -> int: searcher = engine.searcher # The judge the product would write queries with: the same rule ``verify`` uses. + # Under ``--judge`` it also gives its opinion of the escalated verdicts. judge = engine.judge if engine.escalate else None path = measured_path(judge) if searcher is not None else GOLD_PATH scored = 0 @@ -489,7 +910,9 @@ def main(argv: list[str] | None = None) -> int: if claim.id in done: continue if searcher is None: - stored.append(_decide_claim(engine, claim, sleep=args.sleep)) + stored.append( + _decide_claim(engine, claim, sleep=args.sleep, fresh=args.fresh, judge=judge) + ) else: # The gold URLs are the answer key; search mode reads what evidence # search would actually find, which is the §17.1 gate this run exists @@ -507,7 +930,16 @@ def main(argv: list[str] | None = None) -> int: found = replace( claim, text=searched.text, source_urls=searched.urls, non_urls=0 ) - stored.append(_decide_claim(engine, found, sleep=args.sleep, anonymous=True)) + stored.append( + _decide_claim( + engine, + found, + sleep=args.sleep, + anonymous=True, + fresh=args.fresh, + judge=judge, + ) + ) scored += 1 # After every claim, not at the end: a run this long is interrupted more # often than it finishes, and the file has to survive that. @@ -529,17 +961,33 @@ def main(argv: list[str] | None = None) -> int: return 2 finally: engine.close() + if judge is not None and judge.switched: + # The run's one switch notice, as the product prints it (spec section 11). + print(f"note {judge.switched}", file=sys.stderr) wanted = {claim.id for claim in claims} rows = [row for row in stored if row.claim_id in wanted] - report = render_report(score(rows), date=date.today().isoformat(), limit=args.limit, path=path) + report = render_report( + score(rows, judge=args.judge), + date=date.today().isoformat(), + limit=args.limit, + path=path, + browser=browser, + fresh=args.fresh, + judge=judge_name, + ) print() print(report) - suffix = "-search" if args.search else "" out = ( Path(args.out) if args.out - else Path("docs/eval") / f"{date.today().isoformat()}-averitec{suffix}.md" + else _report_path( + day=date.today().isoformat(), + search=args.search, + browser=args.browser, + judge=args.judge, + fresh=args.fresh, + ) ) out.parent.mkdir(parents=True, exist_ok=True) out.write_text(report, encoding="utf-8") diff --git a/site/src/data/measured.ts b/site/src/data/measured.ts index 11d22b1..ad19916 100644 --- a/site/src/data/measured.ts +++ b/site/src/data/measured.ts @@ -81,12 +81,12 @@ export const measures: readonly Measure[] = [ }, { id: 'averitec', - figure: '0.270', + figure: '0.371', reading: '3-way accuracy', - against: 'against a 0.708 majority baseline — worse than always guessing "refuted"', + against: 'against a 0.708 majority baseline — still worse than always guessing "refuted"', what: 'End to end on real web claims', - set: 'AVeriTeC dev, 100 claims', - doc: '2026-09-16-averitec.md', + set: 'AVeriTeC dev, 100 claims, browser allowed', + doc: '2026-10-05-averitec-browser-judge-fresh.md', weak: true, }, ]; @@ -98,4 +98,4 @@ export const accessDoc = '2026-09-11-coverage.md'; export const evalUrl = (doc: string): string => `${repo}/blob/main/docs/eval/${doc}`; export const averitecNote = - 'A third of those claims had no readable source at all: 32 URLs needed the browser step, 29 were unreachable, 14 were refused by robots.txt. On the claims that did have one the score is 0.361 — still below the baseline. The models were calibrated on scientific abstracts; a fact-check page is a different object.'; + 'Coverage is no longer the loss: 165 of 200 sources were read. 36 of 89 claims had a readable source and still came back NEI, and 11 went the wrong way. The models were calibrated on scientific abstracts; a fact-check page is a different object. With evidence search and the judge writing the queries it scores 0.483 (14 claims fixed, 4 broken, p = 0.031), the one gain beyond the noise — but 25 of 89 claims come back the wrong way there, 20 of them false claims called supported (likely cause, not yet measured: search finds pages that repeat the claim).'; diff --git a/src/proofpath/verify.py b/src/proofpath/verify.py index 4e49963..74558b6 100644 --- a/src/proofpath/verify.py +++ b/src/proofpath/verify.py @@ -1944,7 +1944,7 @@ class _Judged: cancelled: bool -def _escalates(result: ClaimResult) -> bool: +def escalates(verdict: Verdict) -> bool: """Whether this verdict is the judge's business (spec section 9 step 8). Three exclusions, each with a reason the product cannot do without: @@ -1957,8 +1957,10 @@ def _escalates(result: ClaimResult) -> bool: * Anything the models were confident about. The judge is an escape hatch for the low band, not a second pass over the whole document (spec section 11: 2-4 calls per paper, not one per claim). + + Public because ``scripts/eval_averitec.py`` measures the same band: an eval + that drew its own line would measure a judge the product never runs. """ - verdict = result.verdict if verdict.passage is None or verdict.reason.startswith(NUMERIC_REASON): return False return verdict.tier == "low" or verdict.label is Label.NEI @@ -1985,7 +1987,9 @@ def _judging( began = time.monotonic() emit(StageStart(name=JUDGING, by=judge.name)) - escalated = [(index, result) for index, result in enumerate(results) if _escalates(result)] + escalated = [ + (index, result) for index, result in enumerate(results) if escalates(result.verdict) + ] opinions: dict[str, JudgeOpinion] = {} fresh: dict[str, JudgeOpinion] = {} keys: dict[str, tuple[str, str]] = {} @@ -2006,7 +2010,7 @@ def _judging( opinions[ident] = stored continue passage = result.verdict.passage - if passage is None: # pragma: no cover - _escalates already refused these + if passage is None: # pragma: no cover - escalates already refused these continue ask.append( JudgeItem( diff --git a/tests/test_eval_averitec.py b/tests/test_eval_averitec.py index 2239931..f041e46 100644 --- a/tests/test_eval_averitec.py +++ b/tests/test_eval_averitec.py @@ -12,6 +12,7 @@ import sys from pathlib import Path from types import ModuleType, SimpleNamespace +from typing import Any import pytest @@ -411,3 +412,572 @@ def test_a_rejected_search_key_stops_the_run_at_the_first_claim( assert not out.exists() message = capsys.readouterr().err assert "TAVILY_API_KEY" in message and "rejected" in message + + +# --- per-source rows: the browser ablation and the judge's hypothetical column ------ + +Source = eval_averitec.Source +product_label = eval_averitec.product_label +judged_label = eval_averitec.judged_label +judge_decider = eval_averitec.judge_decider +without_browser = eval_averitec.without_browser + +GROQ = "groq openai/gpt-oss-120b" +LOCAL = "ollama qwen3.5:9b" + + +def _read( + label: str, + score: float = 0.5, + *, + step: int | None = 1, + tier: str = "medium", + escalated: bool = False, + judge_label: str | None = None, + judge_model: str | None = None, + url: str = LIVE, +) -> Any: + return Source( + url=url, + state=OK, + step=step, + label=label, + score=score, + tier=tier, + escalated=escalated, + judge_label=judge_label, + judge_model=judge_model, + ) + + +def _unread(state: str = UNREACHABLE, *, step: int = 1) -> Any: + return Source(url=LIVE, state=state, step=step, label=None) + + +def test_the_product_label_is_the_strongest_assertion_among_the_sources() -> None: + sources = (_read("SUPPORTED", 0.6), _read("REFUTED", 0.9), _read("NEI", 0.99)) + assert product_label(sources) == "REFUTED" + + +def test_on_a_tied_score_the_first_source_wins_as_it_always_did() -> None: + first = _read("SUPPORTED", 0.7, url="https://a.example/1") + second = _read("REFUTED", 0.7, url="https://b.example/2") + assert product_label((first, second)) == "SUPPORTED" + assert product_label((second, first)) == "REFUTED" + + +def test_the_product_label_is_nei_when_something_was_read_but_nothing_asserted() -> None: + assert product_label((_read("NEI"), _unread())) == "NEI" + + +def test_the_product_label_is_none_when_nothing_was_read() -> None: + assert product_label((_unread(), _unread(BLOCKED))) is None + assert product_label(()) is None + + +def test_a_confident_model_verdict_wins_over_the_judge() -> None: + sources = ( + _read("SUPPORTED", 0.8, tier="high"), + _read("NEI", tier="low", escalated=True, judge_label="REFUTED", judge_model=GROQ), + ) + assert judged_label(sources) == "SUPPORTED" + assert judge_decider(sources) == eval_averitec.DECIDED_BY_MODELS + + +def test_without_a_confident_verdict_the_judge_opinions_decide() -> None: + sources = ( + _read("NEI", tier="low", escalated=True, judge_label="SUPPORTED", judge_model=GROQ), + _read( + "REFUTED", 0.3, tier="low", escalated=True, judge_label="SUPPORTED", judge_model=GROQ + ), + ) + assert product_label(sources) == "REFUTED" # the product keeps the model's verdict + assert judged_label(sources) == "SUPPORTED" + assert judge_decider(sources) == GROQ + + +def test_a_tied_judge_vote_is_nei() -> None: + sources = ( + _read("NEI", tier="low", escalated=True, judge_label="SUPPORTED", judge_model=GROQ), + _read("NEI", tier="low", escalated=True, judge_label="REFUTED", judge_model=LOCAL), + ) + assert judged_label(sources) == "NEI" + # No vote won, so every answerer is named: together they left it at NEI. + assert judge_decider(sources) == f"{GROQ} + {LOCAL}" + + +def test_the_decider_is_the_model_whose_vote_won_not_every_model_that_answered() -> None: + sources = ( + _read("NEI", tier="low", escalated=True, judge_label="SUPPORTED", judge_model=GROQ), + _read("NEI", tier="low", escalated=True, judge_label="SUPPORTED", judge_model=GROQ), + _read("NEI", tier="low", escalated=True, judge_label="REFUTED", judge_model=LOCAL), + ) + assert judged_label(sources) == "SUPPORTED" + assert judge_decider(sources) == GROQ + + +def test_an_escalation_the_judge_never_answered_keeps_the_model_label_as_its_vote() -> None: + sources = (_read("REFUTED", 0.3, tier="low", escalated=True),) + assert judged_label(sources) == "REFUTED" + assert judge_decider(sources) == eval_averitec.NO_OPINION + + +def test_a_judge_that_says_nei_everywhere_leaves_a_read_claim_at_nei() -> None: + sources = ( + _read("REFUTED", 0.3, tier="low", escalated=True, judge_label="NEI", judge_model=GROQ), + ) + assert judged_label(sources) == "NEI" + + +def test_a_claim_with_nothing_escalated_or_read_has_no_judged_label() -> None: + assert judged_label((_unread(),)) is None + assert judge_decider((_unread(),)) == eval_averitec.NOTHING_ESCALATED + + +def test_without_browser_drops_only_what_the_browser_read() -> None: + browser = _read("SUPPORTED", 0.9, step=3) + cached = _read("REFUTED", 0.4, step=None) + plain = _read("NEI", step=2) + dropped = without_browser((browser, cached, plain)) + assert dropped[0].label is None + assert dropped[1:] == (cached, plain) + assert product_label(dropped) == "REFUTED" + + +# --- results files: one per run setup, the legacy name untouched --------------------- + + +@pytest.mark.parametrize( + ("search", "browser", "judge", "fresh", "name"), + [ + (False, False, False, False, "averitec_results.json"), + (True, False, False, False, "averitec_search_results.json"), + (False, True, True, True, "averitec_results-browser-judge-fresh.json"), + (True, True, False, True, "averitec_search_results-browser-fresh.json"), + (True, True, True, True, "averitec_search_results-browser-judge-fresh.json"), + (False, False, False, True, "averitec_results-fresh.json"), + ], +) +def test_every_run_setup_gets_its_own_results_file( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + search: bool, + browser: bool, + judge: bool, + fresh: bool, + name: str, +) -> None: + monkeypatch.setattr(eval_averitec, "cache_dir", lambda: tmp_path) + path = eval_averitec._results_path(search=search, browser=browser, judge=judge, fresh=fresh) + assert path == tmp_path / "datasets" / name + + +def test_two_search_setups_on_one_day_write_two_reports() -> None: + day = "2026-10-05" + sentence = eval_averitec._report_path( + day=day, search=True, browser=True, judge=False, fresh=True + ) + judged = eval_averitec._report_path(day=day, search=True, browser=True, judge=True, fresh=True) + assert sentence == Path("docs/eval/2026-10-05-averitec-search-browser-fresh.md") + assert judged == Path("docs/eval/2026-10-05-averitec-search-browser-judge-fresh.md") + legacy = eval_averitec._report_path( + day=day, search=False, browser=False, judge=False, fresh=False + ) + assert legacy == Path("docs/eval/2026-10-05-averitec.md") + + +def test_rows_round_trip_with_their_sources(tmp_path: Path) -> None: + row = Row( + claim_id=7, + gold=SUPPORTED, + predicted="REFUTED", + states=(OK, UNREACHABLE), + urls=(LIVE, ARCHIVED), + sources=( + _read( + "REFUTED", + 0.3, + step=3, + tier="low", + escalated=True, + judge_label="SUPPORTED", + judge_model=GROQ, + ), + _unread(), + ), + ) + path = tmp_path / "results.json" + eval_averitec._save_rows(path, [row]) + assert eval_averitec._load_rows(path) == [row] + + +def test_a_legacy_results_file_without_sources_still_loads(tmp_path: Path) -> None: + path = tmp_path / "results.json" + path.write_text( + '[{"claim_id": 1, "gold": "Refuted", "predicted": null, "states": ["ok"], "urls": []}]', + encoding="utf-8", + ) + (row,) = eval_averitec._load_rows(path) + assert row.sources == () + + +# --- score and report: the ablation and the hypothetical judge ---------------------- + +JUDGED_ROWS = [ + # The browser read the only assertion; the judge agrees with the model. + Row( + claim_id=0, + gold=SUPPORTED, + predicted="SUPPORTED", + states=(OK,), + urls=(LIVE,), + sources=(_read("SUPPORTED", 0.9, step=3, tier="high"),), + ), + # The model is unsure and wrong; Groq's opinion would have been right. + Row( + claim_id=1, + gold=SUPPORTED, + predicted="REFUTED", + states=(OK,), + urls=(LIVE,), + sources=( + _read( + "REFUTED", + 0.3, + tier="low", + escalated=True, + judge_label="SUPPORTED", + judge_model=GROQ, + ), + ), + ), + # Read from the cache: the step is unknown, and the local judge got it wrong. + Row( + claim_id=2, + gold=REFUTED, + predicted="NEI", + states=(OK,), + urls=(LIVE,), + sources=( + _read( + "NEI", + step=None, + tier="low", + escalated=True, + judge_label="SUPPORTED", + judge_model=LOCAL, + ), + ), + ), + # Nothing read; nothing escalated. + Row( + claim_id=3, + gold=NEI, + predicted=None, + states=(UNREACHABLE,), + urls=(LIVE,), + sources=(_unread(),), + ), +] + + +def test_score_leaves_the_product_accuracy_alone() -> None: + result = score(JUDGED_ROWS, judge=True) + assert result.accuracy_3way == pytest.approx(2 / 4) + + +def test_score_drops_browser_reads_for_the_ablation() -> None: + ablation = score(JUDGED_ROWS).ablation + assert ablation is not None + assert ablation.read_sources == 3 + assert ablation.browser_sources == 1 + assert ablation.unknown_step == 1 + # Claim 0 loses its only source and falls to "nothing read" (NEI): 1 of 4 right. + assert ablation.accuracy_3way == pytest.approx(1 / 4) + + +def test_rows_without_sources_have_no_ablation_and_no_judged_column() -> None: + result = score(ROWS) + assert result.ablation is None + assert result.judged is None + + +def test_score_reports_the_hypothetical_judge_accuracy_and_who_decided() -> None: + judged = score(JUDGED_ROWS, judge=True).judged + assert judged is not None + # Claims 0 (models) and 1 (Groq) right, 2 (local) wrong, 3 NEI right. + assert judged.accuracy_3way == pytest.approx(3 / 4) + assert judged.escalated == 2 + assert judged.unanswered == 0 + assert judged.opinions == {GROQ: 1, LOCAL: 1} + assert judged.deciders[eval_averitec.DECIDED_BY_MODELS] == (1, 1) + assert judged.deciders[GROQ] == (1, 1) + assert judged.deciders[LOCAL] == (1, 0) + + +def test_a_judge_run_with_nothing_escalated_still_has_a_judged_column() -> None: + judged = score(ROWS[:1], judge=True).judged + assert judged is not None + assert judged.escalated == 0 + + +def test_the_report_states_the_run_setup() -> None: + text = render_report( + score(JUDGED_ROWS, judge=True), + date="2026-10-05", + limit=4, + browser=True, + fresh=True, + judge=GROQ, + ) + assert "browser step: allowed" in text + assert "fetch: live, cache bypassed" in text + assert f"judge: {GROQ}" in text + + +def test_the_ablation_bounds_the_pages_read_and_not_the_accuracy() -> None: + text = render_report(score(JUDGED_ROWS), date="2026-10-05", limit=4, browser=True) + assert "## Without the browser" in text + assert "lower bound on the pages such a run reads" in text + assert "not a bound on its accuracy" in text + assert "1 of 3 read sources" in text + assert "step that read them is unknown" in text # a cache read is said, not hidden + + +def test_the_judge_column_says_the_product_does_not_do_this() -> None: + text = render_report(score(JUDGED_ROWS, judge=True), date="2026-10-05", limit=4, judge=GROQ) + assert "## Judge (hypothetical)" in text + assert "never lets the judge change a verdict" in text + assert "**0.750**" in text + assert _md_line(GROQ, 1, 1, "1.000") in text # per deciding model + assert _md_line(LOCAL, 1, 0, "0.000") in text + + +def test_a_report_without_sources_or_judge_has_neither_section() -> None: + text = render_report(score(ROWS), date="2026-10-05", limit=6) + assert "## Without the browser" not in text + assert "## Judge (hypothetical)" not in text + + +def _md_line(*cells: object) -> str: + return "| " + " | ".join(str(c) for c in cells) + " |" + + +# --- _decide_claim: live fetch, step kept, the escalated verdicts asked -------------- + + +def _fetched( + url: str, *, step: int, text: str = "Some text. More text.", outcome: str = "ok" +) -> object: + from proofpath.fetch import Fetched, Outcome + + return Fetched( + url=url, + final_url=url, + step=step, + outcome=Outcome(outcome), + status=200, + content_type="text/html", + kind="html", + body=b"", + text=text, + notes=[], + from_cache=step == 0, + ) + + +class _Fetcher: + def __init__(self, pages: dict[str, object]) -> None: + self.pages = pages + self.calls: list[tuple[str, bool, bool]] = [] + + def fetch(self, url: str, *, anonymous: bool = False, use_cache: bool = True) -> object: + self.calls.append((url, anonymous, use_cache)) + return self.pages[url] + + +class _Judge: + def __init__(self, answers: dict[str, str], model: str = GROQ) -> None: + self.answers = answers # passage text -> label value + self.model = model + self.asked: list[object] = [] + + def review(self, items: list[object], **_: object) -> dict[str, object]: + from proofpath.judge import JudgeOpinion + from proofpath.models import Label + + self.asked.extend(items) + return { + item.id: JudgeOpinion(Label(self.answers[item.passage]), "why", self.model) # type: ignore[attr-defined] + for item in items + if item.passage in self.answers # type: ignore[attr-defined] + } + + +def _engine(fetcher: _Fetcher) -> object: + return SimpleNamespace( + fetcher=fetcher, + get_embedder=lambda: None, + get_scorer=lambda: None, + k=1, + thresholds=None, + ) + + +def _verdicts(monkeypatch: pytest.MonkeyPatch, by_url: dict[str, object]) -> None: + def decide(_claim: str, passages: list[object], *_a: object, **_k: object) -> object: + return by_url[passages[0].source_id] # type: ignore[attr-defined] + + monkeypatch.setattr(eval_averitec.pipeline, "decide", decide) + + +def _verdict(label: str, score: float, tier: str, url: str, reason: str = "") -> object: + from proofpath.models import Label, Passage, Verdict + + return Verdict(Label(label), score, tier, Passage(f"passage of {url}", url, 0), reason) # type: ignore[arg-type] + + +def test_a_fresh_run_climbs_past_the_cache_and_keeps_the_step( + monkeypatch: pytest.MonkeyPatch, +) -> None: + one, two = "https://a.example/1", "https://b.example/2" + fetcher = _Fetcher( + { + one: _fetched(one, step=3), + two: _fetched(two, step=1, outcome="UNVERIFIED (unreachable)", text=""), + } + ) + _verdicts(monkeypatch, {one: _verdict("SUPPORTED", 0.9, "high", one)}) + claim = averitec.Claim(id=1, text="A claim.", label="Supported", source_urls=(one, two)) + + row = eval_averitec._decide_claim(_engine(fetcher), claim, sleep=0, fresh=True) + + assert [call[2] for call in fetcher.calls] == [False, False] + assert row.predicted == "SUPPORTED" + assert [(s.step, s.label) for s in row.sources] == [(3, "SUPPORTED"), (1, None)] + + +def test_only_the_low_band_is_sent_to_the_judge(monkeypatch: pytest.MonkeyPatch) -> None: + low, high, numeric = "https://a.example/low", "https://b.example/high", "https://c.example/num" + fetcher = _Fetcher({url: _fetched(url, step=1) for url in (low, high, numeric)}) + _verdicts( + monkeypatch, + { + low: _verdict("REFUTED", 0.3, "low", low), + high: _verdict("SUPPORTED", 0.9, "high", high), + numeric: _verdict("REFUTED", 1.0, "low", numeric, reason="numeric mismatch: 3 vs 4"), + }, + ) + judge = _Judge({f"passage of {low}": "SUPPORTED"}) + claim = averitec.Claim( + id=2, text="A claim.", label="Supported", source_urls=(low, high, numeric) + ) + + row = eval_averitec._decide_claim(_engine(fetcher), claim, sleep=0, judge=judge) # type: ignore[arg-type] + + assert [item.passage for item in judge.asked] == [f"passage of {low}"] # type: ignore[attr-defined] + by_url = {s.url: s for s in row.sources} + assert by_url[low].escalated and by_url[low].judge_label == "SUPPORTED" + assert by_url[low].judge_model == GROQ + assert not by_url[high].escalated and not by_url[numeric].escalated + # The product's rule is unchanged: the strongest assertion, here the numeric 1.0. + assert row.predicted == "REFUTED" + assert row.predicted == product_label(row.sources) + + +def test_without_a_judge_nothing_is_asked_and_escalation_is_still_recorded( + monkeypatch: pytest.MonkeyPatch, +) -> None: + low = "https://a.example/low" + fetcher = _Fetcher({low: _fetched(low, step=1)}) + _verdicts(monkeypatch, {low: _verdict("REFUTED", 0.3, "low", low)}) + claim = averitec.Claim(id=3, text="A claim.", label="Refuted", source_urls=(low,)) + + row = eval_averitec._decide_claim(_engine(fetcher), claim, sleep=0) + + assert fetcher.calls[0][2] is True # not fresh: the cache may answer + (source,) = row.sources + assert source.escalated and source.judge_label is None + + +# --- main: the new flags --------------------------------------------------------------- + + +def test_browser_and_no_browser_cannot_both_be_asked_for( + capsys: pytest.CaptureFixture[str], +) -> None: + with pytest.raises(SystemExit) as exc: + eval_averitec.main(["--browser", "--no-browser"]) + assert exc.value.code == 2 + + +def test_a_judge_run_without_its_key_stops_before_the_dataset( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] +) -> None: + from proofpath.config import Config + + monkeypatch.setattr(eval_averitec, "cache_dir", lambda: tmp_path) + monkeypatch.setattr(eval_averitec, "load_config", Config) + monkeypatch.setattr(eval_averitec.judge_mod, "resolve_api_key", lambda _name: None) + monkeypatch.setattr( + eval_averitec.averitec, + "ensure_downloaded", + lambda _cache: pytest.fail("the dataset was fetched before the key was checked"), + ) + + assert eval_averitec.main(["--judge"]) == 2 + assert "GROQ_API_KEY" in capsys.readouterr().err + + +# --- review round 1: a resume over an older file, the judge's role in search mode ----- + +# A row from a results file written before sources were kept, right as the product +# scored it, beside one new row whose only source the browser read. +MIXED_ROWS = [ + Row(claim_id=0, gold=SUPPORTED, predicted="SUPPORTED", states=(OK,), urls=(LIVE,)), + Row( + claim_id=1, + gold=REFUTED, + predicted="REFUTED", + states=(OK,), + urls=(LIVE,), + sources=(_read("REFUTED", 0.9, step=1, tier="high"),), + ), +] + + +def test_an_older_row_keeps_the_products_label_in_the_ablation() -> None: + ablation = score(MIXED_ROWS).ablation + assert ablation is not None + assert ablation.browser_sources == 0 + assert ablation.unrecorded == 1 + # No browser page was dropped, so nothing may move: 2 of 2, as the product scored. + assert ablation.accuracy_3way == pytest.approx(1.0) + + +def test_an_older_row_keeps_the_products_label_in_the_judged_column() -> None: + judged = score(MIXED_ROWS, judge=True).judged + assert judged is not None + assert judged.accuracy_3way == pytest.approx(1.0) + assert judged.unrecorded == 1 + assert judged.deciders[eval_averitec.UNRECORDED] == (1, 1) + + +def test_the_report_counts_the_rows_with_no_per_source_record() -> None: + text = render_report(score(MIXED_ROWS, judge=True), date="2026-10-05", limit=2, judge=GROQ) + assert "1 rows have no per-source record" in text + + +def test_in_search_mode_the_report_says_the_judge_wrote_the_queries() -> None: + result = score(JUDGED_ROWS, judge=True) + gold = render_report(result, date="2026-10-05", limit=4, judge=GROQ) + searched = render_report( + result, + date="2026-10-05", + limit=4, + judge=GROQ, + path=eval_averitec.measured_path(SimpleNamespace(name=GROQ)), + ) + assert "the verdicts are the models'" in gold + assert "writes the search queries" in searched + assert "writes the search queries" not in gold diff --git a/tests/test_verify.py b/tests/test_verify.py index e7b83b8..dcda3e7 100644 --- a/tests/test_verify.py +++ b/tests/test_verify.py @@ -27,7 +27,7 @@ from proofpath.browser import Answer, ConsentGate from proofpath.cache import Cache from proofpath.config import Config, ModelsConfig, Permission, Permissions -from proofpath.document import Claim, Document, Locator, PageError +from proofpath.document import Document, PageError from proofpath.entailment import Scorer, pick_onnx_file from proofpath.events import ( Cancelled, @@ -49,7 +49,6 @@ from proofpath.report import ( ABSTRACT_BASIS, JUDGE_DETAIL_PREFIX, - ClaimResult, Kind, Report, judge_detail, @@ -85,9 +84,9 @@ VERIFYING, Engine, Prepared, - _escalates, _parser_for, decide_all, + escalates, prepare, target_document, verify, @@ -2734,17 +2733,6 @@ def test_asking_for_a_summary_without_a_judge_is_a_programming_error() -> None: verify(draft(JUDGE_BODY, [REAL]), built, summarize=True) -def _result(verdict: Verdict) -> ClaimResult: - claim = Claim( - text="the method is faster", - locator=Locator(line=1), - cited_refs=(1,), - paragraph=0, - sentence=0, - ) - return ClaimResult(claim, 1, "doi:10.1/x", verdict, from_cache=False) - - PASSAGE = Passage("the method is faster", "doi:10.1/x", 0) @@ -2765,4 +2753,4 @@ def _result(verdict: Verdict) -> ClaimResult: def test_the_escalation_set_is_the_low_band_with_something_to_judge_against( verdict: Verdict, escalated: bool ) -> None: - assert _escalates(_result(verdict)) is escalated + assert escalates(verdict) is escalated