mcp-tool-catalog

Pipeline that builds the automatelab/mcp-servers-tool-catalog HuggingFace dataset.

The dataset is a machine-introspected catalog of public Model Context Protocol (MCP) servers and the tools each one exposes. We don't scrape READMEs — we spawn each server over stdio, send a JSON-RPC tools/list, and capture the real response.

Dataset	HuggingFace
Companion site	automatelab.tech/products/datasets/mcp-tool-catalog/
Servers in latest snapshot	922 attempted, 359 successfully introspected
Tools captured	9,922
Refresh	Monthly via GitHub Actions (1st of month, 03:00 UTC)
License	Code: MIT. Data: CC-BY-4.0

Pipeline

servers-final.validated.json   → candidate npm packages (curated seed list)
        │
        ▼
validate_npm.py                → drop hallucinated packages
        │
        ▼
introspect.py                  → spawn each server, call tools/list
        │
        ├── data/servers.jsonl  → one row per server (incl. failed)
        └── data/tools.jsonl    → one row per (server, tool) on success
        │
        ▼
diff_snapshot.py               → diff vs previous month's snapshot
        │
        ▼
package_dataset.py             → write hf-dataset/{data/*.parquet, README.md}
        │
        ▼
huggingface-cli upload         → push to automatelab/mcp-servers-tool-catalog

How a refresh works (monthly)

The .github/workflows/monthly-refresh.yml workflow runs on the 1st of every month at 03:00 UTC, or on-demand via workflow_dispatch:

1. Checks out the repo (previous month's data/*.jsonl is the snapshot).
2. Installs Node 20 + Python 3.11.
3. Runs validate_npm.py to catch packages that disappeared from npm.
4. Runs introspect.py with CONCURRENCY=8 TIMEOUT=120 on the full validated list.
5. Diffs vs the snapshot → changelogs/changelog-YYYY-MM-DD.md.
6. Packages parquet + dataset card via package_dataset.py.
7. Uploads hf-dataset/ to HuggingFace using HF_TOKEN secret.
8. Commits new data/*.jsonl + changelog back to main.

Budget: ~3 hours of runner time for the introspection step.

Run locally

pip install -r requirements.txt

# Full refresh (mirrors the cron):
./monthly_refresh.sh

# Just rebuild the parquet card from existing data/:
python package_dataset.py

Introspection spawns up to CONCURRENCY npm processes in parallel; each gets TIMEOUT seconds. The pipeline is Windows-friendly (UTF-8 forced, threaded non-blocking stderr reader).

Adding new servers to the catalog

Edit servers-final.validated.json:

[
  { "name": "stripe", "package": "@stripe/mcp", "category": "payments" }
]

validate_npm.py will reject any package that 404s on npm view. The next refresh picks up the addition.

Failure classification

Each non-ok server gets one of these status values, derived by stderr pattern matching in introspect.py:

• npm_404_not_found — package gone from npm
• npm_network_error — transient install failure
• needs_<vendor>_token / needs_<vendor>_key — credentials required at init
• needs_database_url / needs_aws_creds / needs_azure_creds / needs_google_creds — backing service auth
• needs_config_file / needs_cli_args — server expects positional input
• needs_python — Python runtime missing
• broken_install — install succeeded but server crashes on load
• init_timeout — no response to initialize
• tools_list_timeout — initialize ok but tools/list hung

Repository layout

.
├── .github/workflows/monthly-refresh.yml   ← the cron
├── servers-final.validated.json             ← candidate seed list (one entry per server)
├── validate_npm.py                          ← npm existence check
├── introspect.py                            ← stdio JSON-RPC client + classifier
├── introspect_append.py                     ← retry-only helper
├── retry_failed.py                          ← re-introspect just the failures
├── merge_candidates.py                      ← merge multiple candidate sources
├── diff_snapshot.py                         ← month-to-month changelog
├── package_dataset.py                       ← parquet + dataset card
├── monthly_refresh.sh                       ← orchestrator (local + CI use the same script)
├── requirements.txt                         ← Python deps
├── data/
│   ├── servers.jsonl                        ← rolling state (committed each refresh)
│   └── tools.jsonl
├── hf-dataset/                              ← HF upload bundle (regenerated each refresh)
│   └── README.md                            ← dataset card
└── changelogs/                              ← one file per refresh

License

MIT for the pipeline code (this repo). CC-BY-4.0 for the dataset rows on HuggingFace.

Maintenance and verification

See the maintenance guide for interpretation limits, current tool names where applicable, and reproducible verification.

October 2026 snapshot

926 attempted servers, 363 successful probes and 9,971 recorded tools. This refresh re-probed six owned servers only; other records retain their previous observation dates. Failed probes are explicit coverage gaps.

Snapshot prepared 12 October 2026. Publication time is recorded separately in the release receipt.