Skip to content

update cco scraper for the shared scraping registry - #30

Open
NisargOza wants to merge 2 commits into
MedARC-AI:mainfrom
NisargOza:cco-scraper-update
Open

NisargOza wants to merge 2 commits into
MedARC-AI:mainfrom
NisargOza:cco-scraper-update

Conversation

@NisargOza

Copy link
Copy Markdown

CCO listing runs previously emitted empty placeholder records for failed pages, counted them toward the document limit, and used source-specific CLI dispatch that could stop --source all early. This ports the scraper from #17 onto the shared registry and listing contract on current main, so failed pages are logged/skipped and only successful documents count.

  • Register CCO alongside NICE and MedlinePlus; reject HTTP error pages and unreadable HTML, canonicalize numeric detail URLs, and retain the master listing's Load more pagination.
  • Declare Playwright in the optional cco extra, gate imports, and document Chromium installation. Normal runtime imports work without Playwright.
  • Preserve version/status/authors/PDF links and mark the extracted HTML as summary; linked PDF bodies are not converted.
  • Add local-output/environment ignore rules and document branch names without the codex/ prefix in AGENTS.md. Keep the existing uv lockfile tracked.

Continues #17 and addresses its review comments: dependency declaration, all-source dispatch, and failed documents counted as success. The original PR remains open for its review history.

Validation: uv run ruff check ., uv run ruff format --check ., uv run pytest -q (70 passed), and uv build --package amfv-datasets. A fresh install without Playwright successfully imports the CLI and reports actionable installation instructions when CCO is invoked. Live checks scraped one explicit URL and two listing documents; the complete corpus was not run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant