Skip to content

docs: rewrite the README for first-time readers and AI assistants - #39

Merged
moonweave merged 2 commits into
mainfrom
docs/readme-rewrite
Oct 9, 2026
Merged

moonweave merged 2 commits into
mainfrom
docs/readme-rewrite

Conversation

@moonweave

Copy link
Copy Markdown
Collaborator

Why

Within the first screen, the README did not answer what ref-verify is, who it is for, how to start, or how to read a result. The reference-list check, which is the feature most people want, sat below the skill install and usage. Reference material (modes, error codes, cache, scope) sat in the body and buried the story. This PR changes docs only; the engine is untouched.

What

README.md and README.ko.md

Both files have the same structure in the same order, so a reader can switch languages:

  1. Value statement, then who it is for, what you get, and how it runs.
  2. Try it in 30 seconds. pipx install ref-verify (or uvx), then check-bib on the new examples/references.txt. The README shows the real ref-verify 1.3.1 output. The file holds seven labelled references: three real, one real with the wrong year, one made up, one retracted, and one Korean thesis not in CrossRef. Together they produce all four verdicts.
  3. How to read a result. One table of verdicts, and an UNVERIFIED breakdown from the three test sets run on 1.3.1. Of 82 UNVERIFIED results, 42 were made up and 36 were legitimate but not in CrossRef. Each set links to its results file.
  4. Three ways to use it: agent skill, command line, saved HTML report.
  5. Scorecard from held-out v2 (99 items, benchmarks/results/2026-10-09-8170e66-holdout-v2.json):
    • Each rate is shown with n and a 95% Wilson interval.
    • The two in-sample sets are labelled as in-sample.
    • The section names the known misses, including the Feynman book that passed on a journal review in 1.3.1.
    • It ends with the method and what is not measured.
  6. What it checks and what it does not (one table).
  7. Privacy and network use. Every service contacted, what is sent, the user agent, the email and key environment variables, and what the cache stores, with how to turn it off. All of this was checked against crossref.py, openalex.py, semantic_scholar.py, pubmed.py, http.py, and cache.py.
  8. For AI agents. Commands, inputs, exit codes, JSON keys, which verdicts must never be reported as verified, and wording rules. Links to SKILL.md and AGENT_USAGE.md.
  9. More documentation; contributing, license, related projects.

Other changes

  • docs/REFERENCE.md and docs/REFERENCE.ko.md: everything the README used to carry (skill install, request examples, check-bib details, CLI, modes, error codes, saved reports, cache, scope, what it catches, examples, development checks), moved verbatim except for heading renames.
  • Scorecard SVGs: regenerated with scripts/render_scorecard.py from the held-out v2 results, with --dev pointing at the v1 results from the same run. The renderer's retracted row now reads "n = 12 (4 unmarked)" and "8/8 marked ones caught", so the rate's denominator is visible.
  • tests/test_skill_docs.py:
    • Facts the README used to carry are asserted in README + REFERENCE. No phrase was removed from any assertion.
    • Links, logo, install command, AGENT_USAGE link, check-bib, and --report stay asserted in the README itself, and the Korean README is now held to the same.
    • The heading-order test follows the new README and REFERENCE structure, and adds a check that the English and Korean files have the same heading structure.
    • The Korean examples test reads docs/REFERENCE.ko.md.
  • benchmarks/README.md now says the README chart comes from held-out v2. CHANGELOG: an Unreleased "Changed" entry.

Content removed on purpose

An automated diff of the old README sentences against the new README plus the docs found these drops. Everything else is present verbatim or under a renamed heading.

  • Taglines "Stop citing papers that do not say what you think they say." and "논문에 없는 내용을 인용하지 않도록 막습니다." were replaced by a value statement about reference lists. Claim checking is still described in the README table and in REFERENCE.
  • Old scorecard: held-out v1 measured on 1.2.2 at 01c7a37 (86 items, its numbers, its results-file links). It is superseded by held-out v2 on 1.3.1. The held-out v1 numbers now appear as in-sample results from 1.3.1, and the old results files stay in benchmarks/results/.
  • Korean "Related" list: anneal-skill and decide-skill. Both repositories are archived on GitHub; the Korean README now lists decision-kernel, as the English one already did.

Checks

  • Install lines work from PyPI 1.3.1 with uvx --from ref-verify==1.3.1, a clean venv pip install ref-verify==1.3.1, and pipx install ref-verify==1.3.1 (isolated PIPX_HOME).
  • The sample output was produced by uvx 1.3.1 with a fresh cache and no mailto. Exit code: 2.
  • Every relative anchor and every blob/main link was checked: the target file exists in this branch and the anchor exists in GitHub's rendered HTML.
    • External and image URLs return 200.
    • docs/REFERENCE*.md, examples/references.txt, and its raw URL in the 30-second block return 404 until this merges.
    • Images keep absolute raw.githubusercontent.com URLs because the README is also the PyPI description; the test forbids relative .github/ image paths. The scorecard SVGs at those URLs show the old chart until this merges.
  • README and README.ko were rendered with gh api markdown using mode=markdown and context=Moonweave-Research/ref-verify. mode=gfm turns every source line break into a hard break, the way comments render, which a repository README does not do. The renders were wrapped in github-markdown-css and screenshotted in light and dark at 830 px and at a 390 px layout. They were reviewed by eye: code blocks scroll horizontally on phones, as on GitHub.
  • The Korean text was reviewed with the korean-tech-copy audit. Connective-ending comma density went from 10.7% to 4.3%. "보류 세트" became "홀드아웃 세트". Verdict names stay in English.
  • PYTHONPATH=src python3 -m unittest discover -s tests: OK. python3 -m py_compile src/ref_verify/*.py tests/*.py scripts/*.py: OK.

🤖 Generated with Claude Code

moonweave and others added 2 commits October 9, 2026 15:42
Holdout-v1 was used to tune #30 and #33, so the chart's headline no
longer measured unseen references. The chart now comes from the 99-item
held-out v2 run on 1.3.1, and the retracted row says how many of its
retractions CrossRef cannot mark, so the 8/8 rate and the 12-item bar
no longer read as a contradiction.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first screen did not say what ref-verify is, who it is for, how to
try it, or how to read a verdict; the reference-list check sat below
the skill install, and reference material buried the story. The README
now leads with the problem, a 30-second run on a labelled sample list
with real 1.3.1 output, what each verdict asks of the reader (with how
often UNVERIFIED was legitimate in the test sets), three ways to use
it, the held-out scorecard, limits, privacy and network use, and the
facts an agent needs. Commands, modes, error codes, the cache, scope,
and examples move unchanged to docs/REFERENCE(.ko).md, and the doc
tests now read them there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@moonweave
moonweave merged commit d43da07 into main Oct 9, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant