[shekharprateek]

Why

The vended skill was still 0.1.x. It asked the assistant's model to judge a quality floor and a complexity tier in prose, then took the cheapest model clearing that floor at that tier.

Measuring that version against its own evidence showed the judgment, not the arithmetic, was the weak part:

• Three identical passes agreed on the floor for only 14 of 21 tasks, and on the tier for 18 of 21.
• The tier classifier matched the dataset's own complexity label 16 of 21 times.
• Some floors were unreachable — the skill never checked that any model could clear the floor it had just set.

docs/swe-router-evaluation.md and -judged.md document that run, so they stay. This PR adds a note to each saying they score the previous router — the instability they document is what motivated the change.

What 0.2.0 does

Six scored questions replace the prose judgment. Four set a difficulty bucket (easy, moderate, hard, very_hard); two set the required quality. Each is 0–2 against written criteria in the new rubric.md, and routing-policy.json holds the thresholds, so an organisation can retune what a bucket demands without touching route.py.

The router then takes the cheapest permitted model whose benchmark score in that bucket clears the bar — and skips a model that failed a benchmark task in that bucket rather than trusting its mean.

Area	Change
vend/swe-router/	route.py (accepts both schema majors; --rubric requires 2), SKILL.md, README.md, allowed-models.txt, model-aliases.json, models.json → schema 2.0 with score_by_difficulty / completion_by_difficulty
New files	rubric.md (the questions) and routing-policy.json (the thresholds)
install.sh	Seven files instead of five; preserves both policy files on upgrade; refuses to install without python3 3.8+
benchmarks/scripts/	swe_router_rubric.py (new — shared bucket/quality arithmetic), build_vended_models.py, run-swe-router-headless.py (--mode rubric, keeping --mode floor for one release), eval_swe_router.py
Tests	test_swe_router_rubric.py (new) plus the route/generator/headless suites

swe_router_rubric.py exists so the skill, the generator and the evaluator share one implementation of the bucket arithmetic and cannot drift apart.

Three deviations from upstream's version of these files

Each avoids shipping something broken here:

1. models.json is committed as a vended artifact, not regenerated. The generator reads docs/metrics/pareto-frontier-omp-swe3.json and the per-task run artifacts, neither committed in this repo, so it cannot run here. Same situation as the 0.1.x file.
2. The skip guards this repo added in 56560da are re-applied to test_build_vended_models.py. Upstream commits the frontier data and so has no guards; without them CommittedArtifactTest and the real-data tier test fail here for want of input. They still run wherever the data is present.
3. Five links to docs that don't exist here are dropped rather than left dangling — swe-router-how-it-works.md, swe-router-rubric-design.md, -evaluation.md, results-cli-cli-v1.md. Those pages carry un-abstracted model names and haven't been ported. Prose that stood on its own is kept, and a repo issue link became a bare issue #221 ref, matching the convention used in 28 places already.

Verification

Routing behaviour, exercised directly:

Rubric	Bucket	Floor	Recommends
all zeros	easy	60	claude-haiku-5-5 @ $0.14/task
all twos	very_hard	80	claude-opus-5-5 @ $4.44/task
all twos, --confidence low	very_hard	80	nothing — keeps the developer on their model

A 32x cost spread across the range, which is the point of routing. Low confidence declining to answer is deliberate: the policy keeps the developer put rather than guessing.

Installer, against this branch:

• Installs all 7 files.
• A re-run reports Kept your existing routing-policy.json allowed-models.txt and both edits survive.
• Run against main (where rubric.md doesn't exist yet) it 404s, writes nothing, and says so — verified.
• The installed copy routes standalone and honours the local allowed-models.txt.

Checks: 549 tests pass (19 more than before), ruff, ruff format, mypy, bandit (0 issues at every severity), no dangling relative links in any changed markdown.

test_judge_common.test_lowercase_preferred_on_collision fails — it fails on main too, assuming a case-sensitive filesystem. Untouched here.

Follow-ups

• The four design/results docs above are worth porting once their model names are abstracted, the same pass as PRs docs: add gpu-selection-h200-vs-l40s with model names abstracted #24–docs: add swe-router-evaluation-judged with ranked comparison abstracted #27. Until then vend/swe-router/README.md carries the explanation inline.
• benchmarks/README.md's routing-evaluation section still documents the --mode floor workflow end to end; this PR documents --mode rubric ahead of it, but the older walk-through is now of historical interest only.