[shekharprateek]

Description

The vended swe-router skill shipped the real model names from the 23-model run. This replaces them with class placeholders, so the published data no longer names any model, and adds the disclosure to both READMEs.

The grammar is the one this repository already uses for the cost/quality chart and the evaluation docs:

	Scheme	Examples
Closed-source	tier + generation	closed-frontier-gen5, closed-mid-gen6, closed-efficient-gen1
Open-weight	parameter count + dense/MoE	open-moe-743b-a39b, open-dense-27b, open-moe-30b-a3b-code

Every score, cost, completion count and frontier flag is untouched — only the names are substituted.

Generator left alone, on purpose

benchmarks/scripts/build_vended_models.py still emits real names. Benchmarking your own repository therefore produces a models.json naming models your developers can actually select, and that is now the documented way to get real results — stated in the root README, in vend/swe-router/README.md, in the new model_names key of models.json, and in the header of allowed-models.txt. The committed copy is the one hand-masked exception, and the README says so where it previously claimed the file is never edited by hand.

Generation numbering

Numbers run over all 23 measured models, so they are relative to models.json rather than to the chart's 12-model subset — the two differ for the closed-source tiers because the full set contains releases the chart omits. docs/swe-router-evaluation.md and docs/swe-router-evaluation-judged.md cross-reference models.json directly, so they move to the router's numbering. docs/diagrams/cost-quality-abstracted.html keeps its own, as its caption already states, and is not touched here (re-exporting the PNG would be needed to change it).

Changed files

• vend/swe-router/models.json — 23 model values, both judge names, the rubric filename; new top-level model_names key.
• vend/swe-router/model-aliases.json — real vendor spellings removed along with the names, since keeping them would have made the mask reversible. What remains demonstrates the matching mechanism and needs rebuilding against a real model list. Matching rules genericised to us.<vendor>. / <vendor>/.
• vend/swe-router/allowed-models.txt — frontier entries, commented self-hosted suggestions, and a header noting nobody can select a model called open-dense-27b.
• vend/swe-router/README.md — the 23-row column table, all prose references, the disclosure blockquote, and a caveat under Read these caveats.
• vend/swe-router/route.py — two vendor names in normalize() comments.
• README.md — the disclosure line; the /benchmark example now takes <your-model-id>.
• docs/swe-router-evaluation{,-judged}.md — realigned to the router's numbering; the judge named by class.
• benchmarks/tests/test_route.py — asserted the old slugs against the vended files, so it moves with them.

Not in scope

Real model names remain in the harness and serving plumbing (self-hosted/vllm/models/, benchmarks/scripts/, bedrock_pricing.py, harness fixtures), where the ids are functional. Three published results files are in the same category as the data masked here and were left alone to keep this diff focused: docs/cost-per-task-methodology.md, benchmarks/README.md, benchmarks/swe-benchmark-data/README.md.

Type of Change

• Bug fix
• New feature / enhancement
• Documentation update
• Refactor (no functional change)

Checklist

• I have read CONTRIBUTING.md
• Shell scripts pass bash -n syntax check
• Python files pass python3 -m py_compile
• I have updated documentation if needed
• I have not included credentials, keys, or sensitive data

Verification

python3 vend/swe-router/route.py --confidence high \
  --rubric '{"hidden_contract":1,"silent_invariant":0,"coordinated_surfaces":2,"ambiguity":1,"verification":1,"blast_radius":1}'
cd benchmarks && uv run python -m unittest discover -s tests

The router runs end to end on the masked data and recommends closed-efficient-gen6. 614 tests: one failure, test_judge_common.test_lowercase_preferred_on_collision, which also fails on a stashed tree — a case-insensitive-filesystem issue unrelated to this change. ruff lint, ruff format and pre-commit pass.