[Abduhu1]

What this adds

This PR adds an AAFP scraper for Clinical Insights pages.

It discovers eligible pages from the AAFP sitemap or accepts a specific page through --url. Each clinical recommendation drawer is saved as a separate ScrapedDocument. This avoids mixing recommendations with different populations, sources, dates, and AAFP positions into one document.

The scraper preserves the recommendation content as Markdown and records useful metadata, including:

• Topic and category
• Recommendation title and subtitle
• Source links
• AAFP status and status date
• Topic URL and section anchor

It also reports errors when discovery or page parsing fails instead of silently returning an empty result.

Files changed

• datasets/amfv_datasets/scraping/aafp.py

  • Adds AAFP discovery, parsing, metadata extraction, and error handling.

• datasets/amfv_datasets/scraping/cli.py

  • Registers aafp as a scraper source.
  • Rejects --source all when used with a single --url.

• datasets/test/test_scraping_aafp.py

  • Adds offline tests using synthetic HTML, XML, and mocked HTTP requests.

• datasets/test/test_scraping_cli.py

  • Updates CLI source tests and adds coverage for the new validation rule.

No new dependencies were added.

How to test

uv sync --frozen

uv run --frozen pytest datasets/test/test_scraping_aafp.py -v

uv run --frozen pytest -q

uv run --frozen ruff check .

uv run --frozen ruff format --check .

uv run --frozen amfv-scrape --help

Optional live smoke test:

uv run --frozen amfv-scrape \
  --source aafp \
  --url "https://www.aafp.org/clinical-insights/cardiometabolic-health/hypertension" \
  --documents 2 \
  --no-progress \
  --output /tmp/aafp-smoke.jsonl

Results

• AAFP scraper tests: 74 passed
• Full test suite: 105 passed
• Ruff checks and formatting passed
• Direct URL and automatic discovery smoke tests produced valid documents

This PR contains scraper code and synthetic test fixtures only. It does not add scraped AAFP content.

[CLAassistant]

All committers have signed the CLA.

[zndr27]

The _category_drawers helper in aafp.py, the source collector, opens categories only at accordion headings. Its drawer branch ignores drawers when no category is open.

An accepted fix recognizes drawers directly under the guidelines heading while retaining the boundary that excludes unrelated resources. A regression fixture should cover both arrangements and the unrelated section. Next we examine a source link inside an accepted drawer.

TLDR A page with the same recommendations arranged differently can contribute no documents.

[Abduhu1]

Fixed this by using the Guidelines and recommendations heading as a fallback category for direct drawers, while accepted accordion headings still provide the more specific category. Any unrelated h2 closes the active category. I also added a regression test covering direct drawers, accordion drawers, and an unrelated section.

[zndr27]

Moving down to _full_guideline_url in aafp.py, the metadata collector, the selection rule requires the literal full prefix. The drawer builder therefore leaves its dedicated guideline address empty for a link labelled Read the recommendation.

An accepted fix recognizes the source’s observed recommendation-link wording and tests it against unrelated references so a broader pattern does not choose the wrong link. Next we assemble the final decision.

TLDR The document keeps its link, but the dedicated field for finding the full guideline can stay empty.

[Abduhu1]

Fixed this by recognizing “Read the recommendation” as a full recommendation link. I added a regression test with an unrelated reference before it to make sure the wrong link is not selected.

[Abduhu1]

Thanks for the review. I’ve rebased this onto the latest main and addressed both parsing comments. All 123 tests pass locally, and Ruff and formatting checks are clean. The new GitHub Actions run currently says approval is required. When you get a chance, could you approve the run and take another look?