[MrKoga]

Summary

Adds a MAGICapp scraper to datasets, following the shared scraping base from #21: every network call takes client: httpx.Client first, output is a ScrapedDocument, and the CLI gains a magic source. MAGIC is one of the Meditron guideline sources in the project doc.

MAGICapp is the authoring and publishing platform for GRADE-method guidelines. 48 organizations publish in English through it, among them WHO, the Stroke Foundation of Australia, Cancer Council Australia, the European Stroke Organisation, and BMJ Rapid Recommendations. On 2026-09-14 the catalog held 455 guidelines, 233 in English, of which 226 render into a corpus of 31.7M characters (median 67k per guideline, largest 1.8M: WHO's health-worker task-shifting guideline).

How discovery works

MAGICapp has a public JSON API. One catalog request (api.magicapp.org/api/v1/guidelines) lists every published guideline with its metadata, and each entry links the full guideline as structured JSON: a section tree holding recommendation and PICO objects. So there is no listing page to parse, and this does not port Meditron's scraper, which drives the single-page app with Selenium and is marked UNTESTED in its source.

Catalog selection: English by default (languages= keeps others; 222 non-English guidelines are skipped). Dropped: guidelines from four tutorial and workshop accounts, guidelines the publisher has archived, titles the publisher marks a draft, demonstration guidelines, and anything that renders under 400 characters.

Licensing / redistributability

The project doc lists MAGIC as non-redistributable, and MAGICapp's terms say the same: guideline authors retain copyright (§12), and published content carries no open license. So the scraped text is for the internal search index, not for release as a dataset. 57 of the 233 English guidelines carry a publisher disclaimer, which is kept in metadata["disclaimer"].

Content

Each guideline renders to one markdown document. Sections become headings; section bodies are HTML fragments and go through the shared html_to_markdown.

Recommendation objects render as a heading carrying the publisher's own label ("Evidence-based recommendation", "Practice point", "Good practice statement") plus GRADE strength and certainty, then the population it applies to, the text, remarks, rationale, practical advice, and the evidence-to-decision judgments.

PICO objects render as a short heading plus one line per outcome. They sit beside a section's body in the JSON, not inside it, and their data is stored as data fields rather than as text, so pulling the main text section HTML alone would drop them.

A section is kept when its subtree holds a recommendations or picos object, or when its heading or body carries guidance written as prose ("we recommend", "is not recommended", "practice point"): five guidelines in today's catalog write all their advice that way, with no recommendation or PICO objects at all. Four narrower guards keep a GRADE evidence table, a recommendation-strength key, a scope statement, or a finding of absence ("no RCTs were identified"). Everything else is dropped: authorship, funding, methods narrative, bibliographies, changelogs. Bibliographies are dropped even inside a section that is otherwise kept.

Five repairs run on the HTML before conversion, each because without it the corpus holds text that is wrong:

• Tracked-changes deletions are removed. MAGICapp publishes editor suggestion markup as-is, so an inserted "6" next to a deleted "8" reads "968%" for what is actually 96%.
• Split bold and italic runs are merged and their edges trimmed, so emphasis markers do not land mid-word and glue words together.
• Inline citation markers are removed and the text on either side rejoined.
• colspan and rowspan are expanded, so table cells stay under the right header.
• <img> tags are dropped. Embedded base64 images were 54% of raw output by bytes.

Smaller passes: superscripts and subscripts become Unicode characters (10⁹ stays 10⁹), GRADE certainty circles (⊕⊕⊝) go, question-number prefixes ("PICO 3:") leave headings, and a block that repeats one already rendered in the same document is dropped unless it carries an effect estimate or a dose.

CLI

uv run amfv-scrape --source magic --documents 3 -f markdown -o ./magic-out/
uv run amfv-scrape --source magic --url https://app.magicapp.org/#/guideline/nyxpZL
uv run amfv-scrape --source magic --documents all

--documents caps the catalog entries attempted. A skipped guideline still uses one of them, so --documents 3 returns 2 when the first entry is a consultation draft, as it is today. A full catalog run takes about 33 minutes with the 5-second delay. --url accepts any app.magicapp.org link containing /guideline/<code>.

Components affected

• datasets/amfv_datasets/scraping/magic.py (new, 1,510 lines)
• datasets/test/test_scraping_magic.py (new, 10 tests)
• datasets/amfv_datasets/scraping/cli.py: one import, one SCRAPERS entry
• datasets/test/test_scraping_cli.py: the unknown-source test now checks that each registered source name appears in the error instead of matching the exact joined string. With four sources Typer wraps the list across two lines of its error box, and the exact match fails.

Testing

• uv run pytest datasets/test: 56 passed. The MAGIC tests run offline through MockTransport with no fixture files: catalog parsing, the training-account and draft skips, one refused guideline not aborting the run, tracked deletions, no image data in output, table spans, the structural section filter, prose guidance surviving it, bibliographies going.
• uv run ruff check datasets/ && uv run ruff format --check datasets/: clean.
• Live on 2026-09-14, rebased on main after add medlineplus scraper #23: a --url run, a 3-document run, and the full catalog.

Known limitations

• Flowcharts and dosing tables that MAGICapp stores only as images or linked PDFs are not carried.
• Output has only been read in English. languages= exists but is untested on other languages.
• Publishers use the platform differently: some put process narrative in recommendation remarks, some put guidance in plain sections. The filter errs toward keeping a section when in doubt. Happy to refine further.

[CLAassistant]

All committers have signed the CLA.

[zndr27]

Too big to provide full detailed review. 14,016 lines is more than 8 other scrapers combined. Needs to be ~1,000.

I emptied the six tables in magic_rules.py and re-ran three guidelines. Output grew 5 to 14%, recommendation count unchanged. 11,000 lines stripping boilerplate.

• Drop magic_rules.py, 5,157 lines.
• Drop the 56 cleanup functions and their regexes, ~1,550 lines.
• Most of the 206 tests go with them.
• If you want filtering, clinical sections have picos and recommendations and appendices don't.

[MrKoga]

Hi Zander,

Thanks for the notes! The picos-and-recommendations reframing is a great idea and reworking the filter from that angle I got the code down to ~1,500 lines. magic_rules.py is gone and tests down to 10.

One thing I noticed by running the pico and recommendation rule is that six guidelines write all their advice as text with no recommendation objects, so I kept that rule as the main test and added a few text checks ("we recommend...") to keep those sections. Can also get rid of that if we're ok losing those six guidelines.

Happy to prune more, but I think the remaining cleanup functions are helpful:

• dropping embedded base64 image text
• removing text the MAGIC platform editor marked as deleted
• merging split bold/italic markup so words don't run together
• dropping inline citation markers
• expanding table spans so columns don't slide under the wrong headers

There may be a few more I can cut, and happy to show you examples of the tradeoffs if you'd like. Let me know what you think!

[MrKoga]

Hi @zndr27, I rebased onto main after the MedlinePlus merge and rewrote the description to match my new pared down version.

Now magic_rules.py is gone, the filter keeps a section when it or anything under it has picos or recommendations (plus a text check for the five guidelines that write everything as prose), and tests are down to 10. Five HTML repairs stayed because each one fixes text that comes out wrong otherwise: tracked deletions turning 96% into 968%, split emphasis gluing words together, citation markers, table spans, base64 images. It's 1,510 lines, plus the 10 tests. I'm happy to try to cut further if you'd like.

One thing from the rebase: the CLI test that matches "all, medlineplus, nice" breaks as soon as a fourth source lands, because Typer wraps the error box at 80 columns. It now checks each registered name instead. The other open scraper PRs will hit the same line but I fixed it here.