Skip to content

adopt who guideline scraper with optional pdf extraction - #31

Open
NisargOza wants to merge 2 commits into
MedARC-AI:mainfrom
NisargOza:who-scraper-update
Open

NisargOza wants to merge 2 commits into
MedARC-AI:mainfrom
NisargOza:who-scraper-update

Conversation

@NisargOza

@NisargOza NisargOza commented Oct 9, 2026 •

Copy link
Copy Markdown

WHO landing pages contain only an Overview. This adopts Ashu's WHO implementation from #6 (commit ca7a330fc2e87022e24e6b1b72130b3ec5002408) and ports it onto current main's shared scraper registry, combining the Overview with linked PDF text when conversion succeeds.

  • Add the optional Docling pdf extra and root PDF group; record full versus overview content scope and recover from PDF download/conversion failures.
  • Reject invalid URLs/limits early, stop safely on later-page listing API failures, and log/skip failed publication pages while counting successful documents.
  • Preserve headings, table captions and detected footnotes omitted by Docling's default markdown export. Treat partial conversion as failure.
  • Record publisher reuse policy separately from unverified individual-document licensing. Retain attribution for the source code, real PDF excerpts and inherited historical backend measurements.
  • Include focused public-entry-point tests and opt-in real-PDF/reference-output checks. Shared scraper modules, NICE and MedlinePlus are unchanged.
  • Ignore local corpora, environment variants and handoff notes; document branch names without the codex/ prefix.

Credit: Ashu supplied the original WHO implementation, fixtures and historical five-document backend comparison. The adoption commit includes a co-author trailer. Original #6 remains open. New OCR diagnostics are isolated in #32; CCO is isolated in #30.

Validation: uv run ruff check ., uv run ruff format --check ., AMFV_CHECK_GOLDEN=1 uv run --group pdf pytest -q (88 passed, four third-party deprecation warnings), and uv build --package amfv-datasets. A normal development install without PDF dependencies previously passed 81 tests with seven optional tests skipped. Live checks covered the first listing API page and an Overview-only publication; full-text checks used two six-page real excerpts and mocked HTTP. No full-corpus scrape was performed.

NisargOza and others added 2 commits October 9, 2026 12:03
Build on the WHO implementation, fixtures and historical backend comparison from PR MedARC-AI#6.

Co-authored-by: Ashu <ashutosh@root.credit>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant