Repository navigation
97 evidence snippets remain unverifiable, effectively all on closed-access sources #596
Description
Follow-up from #593. Filed so the backlog that PR exposes is tracked rather than left implicit in a diff.
What the number is
scripts/evidence_snippet_audit.py classifies every one of the 5271 evidence snippets. After #593:
| bucket | count | meaning |
|---|---|---|
| MATCH | 4765 | literal substring of real source text |
| RENDERING | 66 | faithful quote, cache carries a PDF/XML extraction artefact |
| WEAK | 3 | |
| MISMATCH | 296 | real source text exists and the snippet is not in it |
| NOCONTENT | 141 | no source on disk to check against — that is #591 |
MISMATCH is the bucket that matters. Every one of these is a claim where the paper is cached and the quote is not in it.
Why it grew, and why that is good
On main MISMATCH was 190 and NOCONTENT was 379. #593 gave 238 previously-unverifiable snippets a real source; 132 turned out faithful and 106 did not. The 106 were always wrong — they were simply unexaminable.
The worklist
296 snippets across 78 records. Concentrated:
16 BioAsteroid_ISS_Chondrite_Biomining_Consortium
15 Lunar_Simulant_Phosphate_Solubilizing_Bacteria_Nicotiana
15 Microcoleus_Massilia_Cyanosphere_Urea_Mutualism
13 Bacteroides_Eubacterium_Gnotobiotic_Gut_Model
13 Ostreococcus_Dinoroseobacter_BVitamin_Mutualism
11 Thalassiosira_Ruegeria_Phycosphere_Coculture
9 Anammox_Granule_Metabolic_Interaction_Community
8 Drosophila_FiveSpecies_Gnotobiotic_Gut_Microbiota
The top three are worth looking at first not because they are largest but because two of them (BioAsteroid, Lunar_Simulant) come from the space/regolith curation cohort, which suggests a systematic problem with one curation pass rather than 31 independent errors.
How to work it — and one rule that is easy to get wrong
uv run python scripts/evidence_snippet_audit.py --list-mismatch prints them with a diff against the closest cached text.
A MISMATCH is not automatically a fabrication. Before rewriting anything, check in this order:
- Is the cache abstract-only while the snippet is from Methods/Results? Then it is a retrieval gap, not a curation error — run
cache_fulltext.pyfirst. This is the mistake corrected in Two records I curated from abstracts were wrong, not just thin (#183) #577/The partial record was not partial, I searched badly (#183) #578 and it accounted for several apparent "misattributions" there. - Is it a rendering artefact the normaliser did not catch? e.g.
B S _PETaseforBsPETase. Those belong in RENDERING; if one is landing in MISMATCH the normaliser needs the case, and the snippet must not be edited to match the cache (the justfile is explicit). - Is the reference simply the wrong paper for the claim? Seen twice recently — a genome announcement cited for cultivation conditions.
- Only then treat it as a paraphrase to be replaced with a real quote, or a claim to be withdrawn.
Doing (4) first would "fix" the corpus by editing quotes to match whatever text happens to be cached, which is worse than the current state.
Activity
realmarcin commentedon Aug 15, 2026
Correcting this issue's emphasis — I filed it wrong
I opened this describing 296 MISMATCH snippets as "the real evidence backlog" and "fabrication suspects", and pointed at the space/regolith cohort as evidence of "one bad curation pass". I then ran the check I had told everyone else to run first, and the framing does not survive it.
290 of the 296 (98%) sit on an abstract-only cache. Only 6 are mismatches against a paper whose full text is actually on disk.
The two records I singled out are the clearest cases. All 16 of
BioAsteroid's mismatches cite one reference (PMID:41617698, 3873-byte abstract-only cache); all 15 ofLunar_Simulant's cite one (PMID:37945659, 2967 bytes). Their snippets are Methods-level detail —— quoted from a Methods section against a cache that only ever held the abstract. That is step 1 of the triage order in the issue body, and it is the #577/#578 error again: a property of my retrieval mistaken for a property of the curation. There is no evidence of a bad curation pass; there is a retrieval gap behind 78 references.
So #591 is the fix for 98% of this issue, not curation triage. This should be worked after the re-sweep, not before, or the sweep will be re-deriving the same 290.
The 6 that are real, characterised
These are exactly the snippets that were validating circularly before #593 — they passed only because they were listed in the cache file. Now that the listing is gone they are checked against real full text, and they are three different things, none of them fabrication:
1 — a rendering artefact the normaliser does not cover.
PET ... P. putida KT2440 (ATCC(R) 47054). The cache says(ATCC® 47054). The curator wrote(R)for®. This belongs in RENDERING, not MISMATCH, and per the justfile must not be fixed by editing the snippet.2 — table flattening.
OMM12 ... "Lactobacillus reuteri I49, Enterococcus faecalis KB1, Blautia coccoides YL58". All three names are in the cache, but as separate rows of a strain table:... Lactobacillus reuteri I49 DSM 32035 1 Bifidobacterium .... The curator joined three cells with commas. Faithful in content, not a contiguous quote.3 — a quote stitched from two places.
PET ... "R. jostii was added to reduce the inhibition caused by terephthalic acid". The paper hasR. jostii was added to the existing consortium to break down TPAat one point andadding R. jostii to reduce the inhibition caused by terephthalic acid (TPA)at another. The snippet welds the opening of one to the tail of the other. This is the only one of the six that is a genuine curation defect, and it is a paraphrase presented as a quote — the #347 class.Revised recommendation
®/(R)and the table-flattening case, so they land in RENDERING where they belong rather than inflating the suspect count.Leaving the original text above intact rather than editing it, since the wrong framing is the more useful record.
Stop calling two quote artefacts fabrication suspects (#596) (#597)
data: repair the 13 evidence snippets that a full text actually contr…
realmarcin commentedon Aug 31, 2026
Measured the split this issue's own correction called for, then cleared the actionable half. PR #719.
The number that matters is neither 296 nor 107
What decides whether a MISMATCH means anything is whether the cited source is cached as full text. On an abstract-only cache, a Methods-level quote being absent proves nothing — the #605 rule. Split across 117:
That shape agrees with your earlier correction (98% abstract-only); this pins the residual at 13.
Two of my measurements were wrong before they were right, both in the same direction:
cache_text()'s second return value reads like "is full text" and means "has real content" — so my first pass reported 100% full-text, which is what made me check it..mdonly, so three.txtPubMed dumps looked absent. Those dumps are abstracts, and that is the very distinction the split turns on.None of the 13 was a fabrication
Every one had a contiguous fragment of 23–113 characters already present in its source. They were real quotes with a wrong span:
..;Carbon Dioxide/metabolism, whose explanation admitted it: "The cached record indexes…");Ewaste_Bioleaching_Consortiumcited "bioleaching of 10% (w/v) PCBs in a stirred tank bioreactor" where the source says abiotic chemical leaching, the step that prepared a metal-ion solution rather than the microbial one.All repaired to verbatim contiguous spans. Full-text mismatches: 13 → 0.
The
..elisions were repaired rather than taught to the audit. CLAUDE.md forbids combining non-contiguous text, so making the tool accept the marker would have been weakening a check to make existing prose pass — and only 4 snippets KB-wide use.., so it is ad hoc rather than a convention.What is left here
The residual 104 are all abstract-only. That is a retrieval problem, not a curation one, and the same class as #591. The next step is a
cache-fulltextsweep over their references, after which each either resolves or becomes a real finding — a batch that wants a canary first and its own PR. The concentration is unchanged from your original worklist:Bacteroides_Eubacterium_Gnotobiotic_Gut_Model(13),Ostreococcus_Dinoroseobacter_BVitamin_Mutualism(13),Drosophila_FiveSpecies_Gnotobiotic_Gut_Microbiota(8),Yogurt_TwoSpecies_Starter_Culture(8).Worth retitling: "296 MISMATCH snippets across 78 records" now overstates by an order of magnitude, and the headline is the number people quote.
data: fetch what full text is actually obtainable for #596's residual (…
realmarcin commentedon Sep 1, 2026
The residual is not a retrieval gap. Measured in #728.
Following the abstract-only split, the next step was a
cache-fulltextsweep over the 27 distinct references behind the remaining mismatches. It is a much smaller job than the snippet count suggests.Three canaries, each one useful:
PMID:19321416(13 of the 104)not open-access in Europe PMC (oa=False)— clean skip, file byte-identicalPMID:30228381(13 more)PMID:25692519The first two proved only the no-op path, so a third was needed to exercise a successful write — and it also skipped.
Then the free probe. A read-only
isOpenAccessquery over all 26 (PMID:38150661excluded — an untracked local WIP cache):A PMCID is not evidence of fetchability — that is the trap here, and 18 references sit in it.
Fetched, verified on disk:
PMID_42098792.md5,129 → 81,464 bytes andPMID_41343683.md5,010 → 100,238 bytes, both with the full-text marker. The corpus moved MISMATCH 99 → 97, MATCH 5316 → 5318.The new full text also exposed one more
..non-contiguous snippet, repaired the same way as the thirteen in #719 — a defect that was unverifiable in either direction against the previous abstract-only cache.What this means for the issue
The remaining 97 are effectively all on closed-access sources. That is neither a curation backlog nor a retrieval one — it is the institutional-access constraint #183 already tracks. Worth saying so in the title, because "296 MISMATCH snippets across 78 records" now overstates by 3× and reads like unfinished work that somebody could finish.
Suggested disposition: retitle to the measured residual and cross-link #183, or close here and let #183 carry it.
[-]296 MISMATCH snippets across 78 records: the real evidence backlog, now that it is visible[/-][+]97 evidence snippets remain unverifiable, effectively all on closed-access sources[/+]realmarcin commentedon Sep 2, 2026
Retitled from "296 MISMATCH snippets across 78 records: the real evidence backlog, now that it is visible".
The old title was the number people quoted, and it had stopped being true in two different ways:
More importantly, "backlog" implied work someone could finish. Measured:
So what is left is not a curation backlog and not a retrieval one: it is the institutional-access constraint #183 already tracks. A snippet on an abstract-only cache is unverifiable, which is a different claim from wrong — and the #605 rule says absence there proves nothing.
The new title says the measured residual and what it is. Closing versus keeping this open alongside #183 is worth a decision; the number will not move again without institutional full-text access.
realmarcin commentedon Oct 6, 2026
Additional source-coverage work exposed by the provenance repair in #1363 (head
d35e526e30385056f3b99c9a93c85a0d791b309b): DCPP/PMID42613016 has 19 MISMATCH items against independently retrieved abstract text after curator-selected appendices are excluded; two snippets still match. This is a retrieval/provenance gap, not a finding that all 19 claims are false. RAMC/PMID41381587 retains 20 exact matches plus one ASSEMBLED item atgrowth_media[0].evidence[1]and one RENDERING difference, with its retrieved full body preserved. No biological record or cache was edited. Detailed paths and before/after counts are committed inreports/causal_graph_review/cache-provenance-20261006-batch26.json. This comment does not claim a new complete primary-source review or resolve this backlog.realmarcin commentedon Oct 6, 2026
Full-record DCPP assessment recorded in #1367: unchanged/needs_research, not a completed graph review. The entire independent abstract was read; current Europe PMC metadata reports isOpenAccess=N and lists free HTML, but ordinary HTML retrieval returned HTTP 403. No access control was bypassed and no primary body was committed. The repaired snippet audit retains two MATCH and 19 MISMATCH items; absence from the retrieved abstract is not proof of false biological claims. Exact three-strain comparisons, partner-specific detoxification and LC-MS details still require the primary body. This issue remains open.