From f7b4fa168fab9013b2aee51351713192087fdc3b Mon Sep 17 00:00:00 2001 From: shekharprateek Date: Mon, 19 Oct 2026 16:48:52 -0400 Subject: [PATCH 1/2] feat(swe-router): route on a six-question rubric (0.2.0) The vended skill was still 0.1.x: it asked the assistant's model to judge a quality floor and a complexity tier in prose, then took the cheapest model clearing that floor at that tier. Measuring that version against its own evidence showed the judgment, not the arithmetic, was the weak part -- three identical passes agreed on the floor for only 14 of 21 tasks. The two pages that document that run are the motivation for this change, so they stay, with a note saying they score the previous router. 0.2.0 replaces the prose judgment with six scored questions. Four set a difficulty bucket (easy, moderate, hard, very_hard) and two set the required quality, each 0-2 against written criteria in rubric.md. routing-policy.json holds the thresholds, so an organisation can retune what a bucket demands without touching route.py. The router then takes the cheapest permitted model whose benchmark score IN THAT BUCKET clears the bar, and skips a model that failed a benchmark task there rather than trusting its mean. - vend/swe-router/: route.py (both schema majors; --rubric needs 2), SKILL.md, README.md, allowed-models.txt, model-aliases.json, models.json (schema 2.0, with score_by_difficulty/completion_by_difficulty columns), and two new files -- rubric.md (the questions) and routing-policy.json (the thresholds). - install.sh: installs seven files instead of five, preserves BOTH policy files on upgrade (overwriting routing-policy.json would silently replace an organisation's quality bar), and refuses to install without python3 3.8+ rather than leaving a skill that fails the first time it fires. - benchmarks/scripts/: swe_router_rubric.py (new; the shared bucket/quality arithmetic, so the skill, the generator and the evaluator cannot drift), build_vended_models.py (builds the difficulty columns from a rubric pass), run-swe-router-headless.py (--mode rubric alongside the kept --mode floor), eval_swe_router.py. - Tests: test_swe_router_rubric.py (new) plus the route/generator/headless suites. 549 tests pass, 19 more than before. Three deviations from the upstream version of these files, all to avoid shipping something broken here: 1. models.json is committed as a vended artifact rather than regenerated. The generator reads docs/metrics/pareto-frontier-omp-swe3.json and the per-task run artifacts, neither of which is committed in this repository, so it cannot run here. Same as the 0.1.x file. 2. The skip guards this repo added in 56560da are re-applied to test_build_vended_models.py. Upstream commits the frontier data and so has no guards; without them, CommittedArtifactTest and the real-data tier test fail here for want of input. They still run wherever the data is present. 3. Five links to docs that do not exist here are dropped rather than left dangling (swe-router-how-it-works, swe-router-rubric-design and -evaluation, results-cli-cli-v1). Those pages carry un-abstracted model names and have not been ported. The prose that stood on its own is kept, and the one-off repo issue link became a bare ref, matching convention. Verified end to end: a rubric scoring all zeros buckets easy, floor 60, and recommends claude-haiku-5-5 at $0.14/task; all twos buckets very_hard, floor 80, and recommends claude-opus-5-5 at $4.44. Low confidence keeps the developer on their model instead of guessing. install.sh installs all seven files from a ref, a re-run keeps an edited allowed-models.txt and routing-policy.json, and the installed copy routes on its own and honours the local allow-list. ruff, ruff format, mypy and bandit (0 issues) clean. The one failing test, test_judge_common.test_lowercase_preferred_on_collision, fails on main too: it assumes a case-sensitive filesystem. --- README.md | 6 +- benchmarks/README.md | 12 + benchmarks/scripts/build_vended_models.py | 373 +++++++++++----- benchmarks/scripts/eval_swe_router.py | 374 +++++++++++++++- benchmarks/scripts/run-swe-router-headless.py | 254 ++++++++++- benchmarks/scripts/swe_router_rubric.py | 328 ++++++++++++++ benchmarks/tests/test_build_vended_models.py | 214 ++++++--- benchmarks/tests/test_route.py | 270 +++++++++++- .../tests/test_run_swe_router_headless.py | 54 +++ benchmarks/tests/test_swe_router_rubric.py | 383 ++++++++++++++++ docs/swe-router-evaluation-judged.md | 3 + docs/swe-router-evaluation.md | 3 + vend/swe-router/README.md | 323 +++++--------- vend/swe-router/SKILL.md | 293 ++++-------- vend/swe-router/allowed-models.txt | 46 +- vend/swe-router/install.sh | 68 ++- vend/swe-router/model-aliases.json | 32 ++ vend/swe-router/models.json | 356 ++++++++++++++- vend/swe-router/route.py | 416 +++++++++++++++--- vend/swe-router/routing-policy.json | 49 +++ vend/swe-router/rubric.md | 91 ++++ 21 files changed, 3218 insertions(+), 730 deletions(-) create mode 100644 benchmarks/scripts/swe_router_rubric.py create mode 100644 benchmarks/tests/test_swe_router_rubric.py create mode 100644 vend/swe-router/routing-policy.json create mode 100644 vend/swe-router/rubric.md diff --git a/README.md b/README.md index 1d14ab56..fcc71225 100644 --- a/README.md +++ b/README.md @@ -52,7 +52,7 @@ One command per model. The `/benchmark` skill runs the pre-flight checks, the ha ## Step 2 — Developers install the skill -Five files copied into a skills directory. The skill imports nothing and needs no build step. Run this from the root of the repository you want it in, and it lands in `.claude/skills/swe-router`: +Seven files copied into a skills directory. The skill needs `python3` 3.8 or later on PATH, with no packages and no build step; the installer refuses to run without it. Run this from the root of the repository you want it in, and it lands in `.claude/skills/swe-router`: ```bash curl -sL https://raw.githubusercontent.com/aws-samples/sample-agentic-coding-harness-benchmarks/main/vend/swe-router/install.sh | bash @@ -64,9 +64,9 @@ To install it once for every repository instead, send it to your home directory: curl -sL https://raw.githubusercontent.com/aws-samples/sample-agentic-coding-harness-benchmarks/main/vend/swe-router/install.sh | bash -s -- --dir ~/.claude/skills ``` -Re-run either command to upgrade; your edited `allowed-models.txt` is kept. +Re-run either command to upgrade; your edited `allowed-models.txt` and `routing-policy.json` are kept. -`swe-router` engages on its own before a substantial task. It sets a quality floor from what happens if the change is wrong, classifies how hard the task is, and takes the cheapest model that clears that floor at that tier. **Edit `allowed-models.txt`.** The skill treats every name in it as a model the developer can select, so listing one your team cannot reach costs them the cheaper option. +`swe-router` engages on its own before a substantial task. Your assistant's model answers six rubric questions about the task: four set its difficulty bucket (easy, moderate, hard or very hard), and two set the quality a model must reach, from what happens if the change is wrong. The skill then takes the cheapest permitted model whose benchmark score in that bucket clears the bar, skipping any model that failed a benchmark task there. **Edit `allowed-models.txt`.** The skill treats every name in it as a model the developer can select, so listing one your team cannot reach costs them the cheaper option. Install notes, the file-by-file breakdown and the measured results: **[vend/swe-router/README.md](vend/swe-router/README.md)**. diff --git a/benchmarks/README.md b/benchmarks/README.md index bc215bf9..4b9d5593 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -76,6 +76,18 @@ uv run python summarize_run.py --folder ../swe-benchmark-data/{model-slug}/{harn The [`/swe-router`](../.claude/skills/swe-router/SKILL.md) skill recommends a model per task. Two scripts measure whether taking its advice would have been worth it, using runs already on disk. Neither script re-runs a model. +The router now routes on a six-question rubric rather than a floor and tier judged in prose, so `run-swe-router-headless.py` has two modes. `--mode rubric` is the current one; `--mode floor` (the default, kept for one release) reproduces the earlier evaluation described below. + +**Scoring a dataset's tasks against the rubric.** Each task's difficulty bucket comes from a rubric pass: the judge answers the six questions in [rubric.md](../vend/swe-router/rubric.md) for every task, and `routing-policy.json` turns the answers into a bucket. `build_vended_models.py` reads that pass to build `models.json`'s `score_by_difficulty` columns, so a new dataset -- or any edit to `rubric.md` or the policy -- needs a new pass before the router can route on it: + +```bash +cd benchmarks +uv run scripts/run-swe-router-headless.py --mode rubric --agent omp --provider bedrock \ + --model us.anthropic.claude-opus-5-5 --dataset dataset/.yaml --repeats 3 +``` + +Use a strong judge and more than one repeat: a bucket that moves between repeats moves the model the router picks. Writes `docs/metrics/swe-router-rubric--.json`. + **1. Collect the skill's judgments.** For every task in a dataset, clone the repo at the task's pinned ref and run the skill's step 1 in it (decide a quality floor from the consequence of the change being wrong, and a complexity tier). The agent returns the judgment only. It never selects a model. ```bash diff --git a/benchmarks/scripts/build_vended_models.py b/benchmarks/scripts/build_vended_models.py index f00f1d6a..660604c4 100644 --- a/benchmarks/scripts/build_vended_models.py +++ b/benchmarks/scripts/build_vended_models.py @@ -6,7 +6,7 @@ are copied into ``vend/swe-router/models.json``, which is committed and served raw, and this script is the only thing that writes it. -Two differences from the internal ``docs/metrics/pareto-frontier-*.json`` it +Three differences from the internal ``docs/metrics/pareto-frontier-*.json`` it reads: * **Every measured model is included, not just the frontier.** The skill ranks @@ -20,6 +20,13 @@ knows this repo. A vended file has no such reader, so it carries the schema version, when it was measured, the harness, the skill, the dataset and the judge. A consumer holding a stale copy can see that it is stale. +* **Scores are grouped by difficulty bucket.** The skill compares a task's + required quality against each model's mean score over the benchmark tasks + in the same bucket (easy, moderate, hard, very hard). The bucket of each + benchmark task comes from the dataset's rubric scores file + (``docs/metrics/swe-router-rubric--.json``), recomputed here + from its six stored scores with the shipped ``routing-policy.json``, so a + policy edit never leaves a stale bucket behind. Usage: uv run scripts/build_vended_models.py @@ -31,11 +38,13 @@ import argparse import json import logging -import subprocess # nosec B404 - used with list args, no shell, hardcoded 'git' -from datetime import date +import hashlib +from collections.abc import Callable from pathlib import Path from typing import Any +import swe_router_rubric as rubric + logging.basicConfig( level=logging.INFO, format="%(asctime)s,p%(process)s,{%(filename)s:%(lineno)d},%(levelname)s,%(message)s", @@ -51,10 +60,23 @@ DEFAULT_RUNS_DIR = _REPO_ROOT / "benchmarks" / "swe-benchmark-data" TIERS = ("trivial", "low", "medium", "high") DEFAULT_OUT = _REPO_ROOT / "vend" / "swe-router" / "models.json" +# The rubric pass whose buckets build the columns. One judge per dataset, and a +# strong one: on a cross-repository check the frontier judge agreed with itself +# across three repeats on 20 of 21 tasks, where a cheap judge managed 13. A +# bucket that moves between repeats moves the model the router picks. +RUBRIC_JUDGE = "claude-opus-5-5" +RUBRIC_SCORES_TEMPLATE = "swe-router-rubric-{dataset}-{judge}.json" +DEFAULT_RUBRIC_DIR = _REPO_ROOT / "docs" / "metrics" # Bumped when the shape of models.json changes in a way a consumer would notice. -# Consumers pin this; the skill refuses a major it does not know. -SCHEMA_VERSION = "1.0" +# Consumers pin this; the skill refuses a major it does not know. 1.1 replaced +# provenance.source_commit (a git sha) with source_sha256 (a content hash) and +# took measured_on from the run summaries instead of a commit date. 2.0 added +# score_by_difficulty and completion_by_difficulty, which the skill now selects +# on, and the rubric provenance behind them. +SCHEMA_VERSION = "2.0" +# provenance.source_sha256 length: enough to tell versions apart, short to read. +SOURCE_HASH_CHARS = 12 # The judge is not recorded in the frontier JSON, so it is stated here. Keep in # step with codex_judge.py's default (JUDGE_MODEL overrides it per run). @@ -77,78 +99,59 @@ } -def _git_commit(source: Path) -> str | None: - """Return the short sha of the commit that last changed ``source``. +def _source_sha256(source: Path) -> str: + """Return the first 12 hex digits of the SHA-256 of ``source``'s bytes. - Deliberately not HEAD. HEAD moves with every commit to the repository, so - stamping it would make the generated file differ after any unrelated change - and turn the --check guard into noise. The commit that last touched the - source is what actually identifies this version of the data. + Identifies the exact version of the data from the file itself. An earlier + version stamped the commit that last touched the file, which a squash merge + replaces: the stamp made on the branch then no longer matched ``main``, and + the committed-artifact check failed with every score still correct (after + #184, #210 and #219). A content hash is the same on the branch and after the + merge, and anyone can check it with ``sha256sum | cut -c1-12``. Args: source: The frontier JSON. Returns: - The abbreviated commit, or None outside a git checkout or for a file - that has never been committed. + A 12-character lowercase hex string. """ - try: - out = subprocess.run( # nosec B603 B607 - hardcoded 'git', list args, no shell - [ - "git", - "-C", - str(source.parent), - "log", - "-1", - "--format=%h", - "--", - source.name, - ], - capture_output=True, - text=True, - timeout=10, - check=False, - ) - except (OSError, subprocess.SubprocessError): - return None - return out.stdout.strip() or None + return hashlib.sha256(source.read_bytes()).hexdigest()[:SOURCE_HASH_CHARS] -def _measured_on(source: Path) -> str: - """Return the date the source frontier was last modified, ISO format. +def _latest_run_date( + runs_dir: Path, harness: str, skill: str, dataset: str, models: set[str] +) -> str | None: + """Return the latest ``run_date`` among the run summaries behind the frontier. - Uses the file's last git commit date when available, so regenerating without - changing the data does not advance the measurement date. Falls back to the - filesystem mtime. + That is when the newest measurement in the file was taken, which is what a + consumer means by "measured on". It comes from committed data, so it does + not move when a commit is rewritten, and it does not depend on anyone's + timezone. Args: - source: The frontier JSON. + runs_dir: The swe-benchmark-data root. + harness: Harness folder, e.g. "omp". + skill: Skill folder, e.g. "swe3". + dataset: Dataset scope folder. + models: The model slugs the frontier JSON lists; other runs are ignored. Returns: - An ISO date string. + An ISO date, or None when no matching summary records a ``run_date``. """ - try: - out = subprocess.run( # nosec B603 B607 - hardcoded 'git', list args, no shell - [ - "git", - "-C", - str(source.parent), - "log", - "-1", - "--format=%cs", - "--", - source.name, - ], - capture_output=True, - text=True, - timeout=10, - check=False, + dates = [] + pattern = f"*/{harness}/{skill}/{dataset}/run-summary.json" + for path in runs_dir.glob(pattern): + summary = json.loads(path.read_text(encoding="utf-8")) + if summary.get("model_slug") in models and summary.get("run_date"): + dates.append(summary["run_date"]) + if not dates: + logger.warning( + "no run_date in any %s summary for the frontier's models; " + "measured_on will be null", + pattern, ) - if out.stdout.strip(): - return out.stdout.strip() - except (OSError, subprocess.SubprocessError): - pass - return date.fromtimestamp(source.stat().st_mtime).isoformat() + return None + return max(dates) def _relative_or_name(source: Path, repo_root: Path) -> str: @@ -171,36 +174,26 @@ def _relative_or_name(source: Path, repo_root: Path) -> str: return source.name -def per_tier_stats( - runs_dir: Path, harness: str, skill: str, dataset: str +def _per_level_stats( + runs_dir: Path, + harness: str, + skill: str, + dataset: str, + level_of: Callable[[dict[str, Any]], str | None], + levels: tuple[str, ...], ) -> dict[str, dict[str, dict[str, Any]]]: - """Return model -> {"score": tier->mean, "completion": tier->"n/m"}. - - Two numbers, because difficulty affects two different things. - - **Score.** The whole-dataset mean is the wrong number to compare a quality - floor against when the task is hard. Models do not degrade equally: on this - data ``claude-opus-5`` loses 3.1 points on high-complexity tasks while - ``qwen3.8-27b`` loses 17.6. A model clearing a floor of 70 on its overall - mean can sit at 57 on the tier that actually matters. - - **Completion.** A model that fails a task outright is worse than one that - scores badly, and failures concentrate somewhere: ``devstral-2-123b`` - finished 3 of 6 medium tasks. A mean over the ones it survived says nothing - about that. - - Failed tasks are excluded from the mean rather than averaged in as zero, - matching how the frontier JSON computes ``mean_score``. Including them - would make the tier means disagree with the overall score they sit beside. + """Group every model's task scores by a level and summarise each level. Args: runs_dir: The swe-benchmark-data root. harness: Harness folder, e.g. "omp". skill: Skill folder, e.g. "swe3". dataset: Dataset scope folder. + level_of: Maps a run-summary task row to its level, or None to skip it. + levels: Every level, in output order. Returns: - Model slug -> {"score": {tier: mean}, "completion": {tier: "n/m"}}. + Model slug -> {"score": {level: mean}, "completion": {level: "n/m"}}. """ out: dict[str, dict[str, dict[str, Any]]] = {} pattern = f"*/{harness}/{skill}/{dataset}/run-summary.json" @@ -210,40 +203,163 @@ def per_tier_stats( scored: dict[str, list[float]] = {} attempted: dict[str, int] = {} for row in summary.get("tasks", []): - tier = row.get("complexity") - if not tier: + level = level_of(row) + if not level: continue - attempted[tier] = attempted.get(tier, 0) + 1 + attempted[level] = attempted.get(level, 0) + 1 score = row.get("task_score") # A failed task scores 0 and is excluded, not averaged in -- the # completion counter is where that failure shows up instead. if score and not row.get("failed"): - scored.setdefault(tier, []).append(score) + scored.setdefault(level, []).append(score) if not model or not attempted: continue out[model] = { "score": { - tier: round(sum(v) / len(v), 2) - for tier in TIERS - if (v := scored.get(tier)) + level: round(sum(v) / len(v), 2) + for level in levels + if (v := scored.get(level)) }, "completion": { - tier: f"{len(scored.get(tier, []))}/{attempted[tier]}" - for tier in TIERS - if tier in attempted + level: f"{len(scored.get(level, []))}/{attempted[level]}" + for level in levels + if level in attempted }, } return out +def per_tier_stats( + runs_dir: Path, harness: str, skill: str, dataset: str +) -> dict[str, dict[str, dict[str, Any]]]: + """Return model -> {"score": tier->mean, "completion": tier->"n/m"}. + + Two numbers, because difficulty affects two different things. + + **Score.** The whole-dataset mean is the wrong number to compare a quality + floor against when the task is hard. Models do not degrade equally: on this + data ``claude-opus-5`` loses 3.1 points on high-complexity tasks while + ``qwen3.8-27b`` loses 17.6. A model clearing a floor of 70 on its overall + mean can sit at 57 on the tier that actually matters. + + **Completion.** A model that fails a task outright is worse than one that + scores badly, and failures concentrate somewhere: ``devstral-2-123b`` + finished 3 of 6 medium tasks. A mean over the ones it survived says nothing + about that. + + Failed tasks are excluded from the mean rather than averaged in as zero, + matching how the frontier JSON computes ``mean_score``. Including them + would make the tier means disagree with the overall score they sit beside. + + Args: + runs_dir: The swe-benchmark-data root. + harness: Harness folder, e.g. "omp". + skill: Skill folder, e.g. "swe3". + dataset: Dataset scope folder. + + Returns: + Model slug -> {"score": {tier: mean}, "completion": {tier: "n/m"}}. + """ + return _per_level_stats( + runs_dir, harness, skill, dataset, lambda row: row.get("complexity"), TIERS + ) + + +def task_buckets(rubric_scores: Path, policy_path: Path) -> dict[str, str]: + """Each benchmark task's difficulty bucket, from its stored rubric scores. + + The bucket is recomputed from the six scores with the current policy rather + than read from the file's ``bucket`` field, so moving a bucket edge in + ``routing-policy.json`` takes effect at the next build without a new pass. + + Args: + rubric_scores: A rubric pass (``run-swe-router-headless.py --mode rubric``). + policy_path: The routing policy that sets the bucket ranges. + + Returns: + Task id -> bucket. + + Raises: + SystemExit: If the file is not a rubric pass. + """ + data = json.loads(rubric_scores.read_text(encoding="utf-8")) + if data.get("mode") != "rubric": + raise SystemExit(f"{rubric_scores} is not a rubric pass") + policy = rubric.load_policy(policy_path) + return { + task_id: rubric.bucket(entry["scores"], policy) + for task_id, entry in (data.get("tasks") or {}).items() + } + + +def per_bucket_stats( + runs_dir: Path, harness: str, skill: str, dataset: str, buckets: dict[str, str] +) -> dict[str, dict[str, dict[str, Any]]]: + """Return model -> {"score": bucket->mean, "completion": bucket->"n/m"}. + + The same arithmetic as ``per_tier_stats``, grouped by each task's rubric + bucket instead of the tier written in the dataset. + + Raises: + SystemExit: If a benchmark task has no bucket, which would drop its + scores from every column without saying so. + """ + + def level_of(row: dict[str, Any]) -> str | None: + task = row.get("task") + if task not in buckets: + raise SystemExit( + f"task {task!r} has no rubric scores; re-run the rubric pass for " + f"{dataset} (run-swe-router-headless.py --mode rubric)" + ) + return buckets[task] + + return _per_level_stats( + runs_dir, harness, skill, dataset, level_of, rubric.BUCKET_ORDER + ) + + +def _rubric_provenance( + rubric_scores: Path | None, repo_root: Path +) -> dict[str, Any] | None: + """Where the buckets came from, so a consumer can trace a column.""" + if rubric_scores is None: + return None + data = json.loads(rubric_scores.read_text(encoding="utf-8")) + judged = data.get("judged_by") or {} + return { + "file": _relative_or_name(rubric_scores, repo_root), + "sha256": _source_sha256(rubric_scores), + "judge": judged.get("model"), + "repeats": judged.get("repeats"), + "buckets_recomputed_with": "vend/swe-router/routing-policy.json", + } + + +def default_rubric_scores(dataset: str, rubric_dir: Path = DEFAULT_RUBRIC_DIR) -> Path: + """The rubric pass a dataset's columns are built from.""" + return rubric_dir / RUBRIC_SCORES_TEMPLATE.format( + dataset=dataset, judge=RUBRIC_JUDGE + ) + + def build( - source: Path, repo_root: Path, runs_dir: Path | None = None + source: Path, + repo_root: Path, + runs_dir: Path | None = None, + rubric_scores: Path | None = None, + policy_path: Path = rubric.DEFAULT_POLICY, ) -> dict[str, Any]: """Build the vended payload from a frontier JSON. Args: source: Path to a ``pareto-frontier-*.json``. - repo_root: Repository root, used for the provenance commit. + repo_root: Repository root, used to record the source path. + runs_dir: The swe-benchmark-data root; defaults to the committed one. + rubric_scores: The dataset's rubric pass. None takes the default path + for the frontier's dataset, and builds no bucket columns when that + file does not exist. + policy_path: The routing policy that sets the bucket ranges. Returns: The payload to write as models.json. @@ -264,12 +380,34 @@ def build( bedrock = {m["model"] for m in data.get("bedrock_frontier", [])} self_hosted = {m["model"] for m in data.get("self_hosted_frontier", [])} + runs_dir = runs_dir if runs_dir is not None else DEFAULT_RUNS_DIR by_tier = per_tier_stats( - runs_dir if runs_dir is not None else DEFAULT_RUNS_DIR, + runs_dir, data.get("harness", ""), data.get("skill", ""), data.get("repo", ""), ) + if rubric_scores is None: + candidate = default_rubric_scores(data.get("repo", "")) + rubric_scores = candidate if candidate.is_file() else None + if rubric_scores is None: + logger.warning("no rubric scores for %s; no bucket columns", data.get("repo")) + by_bucket: dict[str, dict[str, dict[str, Any]]] = {} + else: + by_bucket = per_bucket_stats( + runs_dir, + data.get("harness", ""), + data.get("skill", ""), + data.get("repo", ""), + task_buckets(rubric_scores, policy_path), + ) + measured_on = _latest_run_date( + runs_dir, + data.get("harness", ""), + data.get("skill", ""), + data.get("repo", ""), + {m["model"] for m in models}, + ) out_models = [] for m in sorted(models, key=lambda x: -x["mean_score"]): name = m["model"] @@ -296,6 +434,13 @@ def build( # it as context; select on score_by_complexity. "on_combined_frontier": name in combined, "on_hosting_frontier": name in bedrock or name in self_hosted, + # What the skill selects on: the mean score over the benchmark + # tasks in each difficulty bucket, and how many it finished. + "score_by_difficulty": by_bucket.get(name, {}).get("score", {}), + "completion_by_difficulty": by_bucket.get(name, {}).get( + "completion", {} + ), + # The dataset's hand-set tiers, kept for the legacy --tier path. "score_by_complexity": by_tier.get(name, {}).get("score", {}), "completion_by_complexity": by_tier.get(name, {}).get("completion", {}), } @@ -305,13 +450,14 @@ def build( "schema_version": SCHEMA_VERSION, "generated_by": "benchmarks/scripts/build_vended_models.py", "provenance": { - "measured_on": _measured_on(source), + "measured_on": measured_on, "source_file": _relative_or_name(source, repo_root), - "source_commit": _git_commit(source), + "source_sha256": _source_sha256(source), "harness": data.get("harness"), "skill": data.get("skill"), "dataset": data.get("repo"), "judge": JUDGE, + "rubric_scores": _rubric_provenance(rubric_scores, repo_root), "repository": "https://github.com/aws-samples/sample-agentic-coding-harness-benchmarks", }, "measurement_basis": { @@ -332,15 +478,25 @@ def build( "nothing else beats on both score and cost across the WHOLE " "dataset. They are context for a reader, not a selection key: " "they come from overall means, so they can disagree with the " - "per-tier ranking. Select on score_by_complexity at the task's " - "tier. on_hosting_frontier is computed within one hosting basis " + "per-bucket ranking. Select on score_by_difficulty at the task's " + "bucket. on_hosting_frontier is computed within one hosting basis " "(Bedrock or self-hosted), which is the apples-to-apples " "comparison; on_combined_frontier mixes the two and is " "directional." ), + "difficulty_buckets": ( + "easy / moderate / hard / very_hard: each benchmark task's " + "difficulty level, from four rubric scores (hidden contract, " + "silent invariant, coordinated surfaces, ambiguity) summed and " + "bucketed by routing-policy.json. score_by_difficulty is a " + "model's mean over the tasks in a bucket, failures excluded; " + "completion_by_difficulty is where failures show. A bucket with " + "no benchmark task has no entry." + ), "complexity_tiers": ( - "trivial / low / medium / high, assigned by the scope of the " - "change. The hardest tasks measured are bounded single-repo " + "trivial / low / medium / high, set by hand in the dataset from " + "the scope of the change; score_by_complexity serves the legacy " + "--tier path only. The hardest tasks measured are bounded single-repo " "changes -- a rate-limiting subsystem, server-side OAuth token " "storage. Nothing here approaches a rewrite or a language port, " "so a task far beyond that range is outside what these numbers " @@ -357,6 +513,13 @@ def _parse_args() -> argparse.Namespace: p = argparse.ArgumentParser(description="Generate the vended models.json.") p.add_argument("--source", type=Path, default=DEFAULT_SOURCE) p.add_argument("--out", type=Path, default=DEFAULT_OUT) + p.add_argument( + "--rubric-scores", + type=Path, + default=None, + help="The dataset's rubric pass. Default: " + f"docs/metrics/{RUBRIC_SCORES_TEMPLATE.format(dataset='', judge=RUBRIC_JUDGE)}.", + ) p.add_argument( "--check", action="store_true", @@ -368,7 +531,13 @@ def _parse_args() -> argparse.Namespace: def main() -> None: """Generate models.json, or verify the committed copy matches.""" args = _parse_args() - payload = build(args.source, _REPO_ROOT) + payload = build(args.source, _REPO_ROOT, rubric_scores=args.rubric_scores) + if payload["provenance"]["rubric_scores"] is None: + raise SystemExit( + "no rubric scores for this dataset, so models.json would have no " + "bucket columns. Run run-swe-router-headless.py --mode rubric, or " + "pass --rubric-scores." + ) text = json.dumps(payload, indent=2) + "\n" if args.check: diff --git a/benchmarks/scripts/eval_swe_router.py b/benchmarks/scripts/eval_swe_router.py index 07f88b6d..4a6edfae 100644 --- a/benchmarks/scripts/eval_swe_router.py +++ b/benchmarks/scripts/eval_swe_router.py @@ -52,6 +52,8 @@ import tempfile from collections import Counter from pathlib import Path +from collections.abc import Callable +from functools import partial from typing import Any import yaml @@ -78,6 +80,12 @@ sys.path.insert(0, str(_SKILL_DIR)) from route import RouteError, route # noqa: E402 +from swe_router_rubric import ( # noqa: E402 + BUCKET_TO_ROUTE_TIER, + load_policy, + relative_bucket, + relative_edges, +) # Hardware-derived $/token for self-hosted models, same source and same # "cheapest sustainable concurrency level" rule plot_cost_quality.py uses, so a @@ -102,6 +110,11 @@ # tasks run once each. Mirrors the skill's tie band; kept as a constant so the # report can state the band it applied. DEFAULT_TIE_BAND = 3.0 +# --column-model regression routes on one predicted score per model and task. +# route.py still selects by tier key, so the predictions travel under this one. +PREDICTED_TIER = "medium" +# Below this many scored tasks a model's line is not fitted; its mean stands in. +MIN_POINTS_FOR_FIT = 3 def _read_json(path: Path) -> dict[str, Any] | None: @@ -319,6 +332,7 @@ def _route_task( allowed_file: Path | None, no_allow_list: bool, tie_band: float, + min_completion: float = 0.0, ) -> dict[str, Any]: """Ask the router for one task's model. @@ -330,6 +344,8 @@ def _route_task( allowed_file: Explicit allow-list path, or None to let route.py find one. no_allow_list: Ignore organisational policy entirely. tie_band: Points below which two models count as tied. + min_completion: Skip a model that finished less than this share of the + tasks in the column it is read from. Returns: The route.py result dict. @@ -346,9 +362,92 @@ def _route_task( allowed_file=allowed_file, no_allow_list=no_allow_list, tie_band=tie_band, + min_completion=min_completion, ) +def _regrouped( + results: dict[str, dict[str, dict[str, Any]]], + columns: dict[str, str], +) -> dict[str, dict[str, dict[str, Any]]]: + """A copy of the results with each task filed under the given column.""" + return { + model: { + task_id: { + **record, + "complexity": columns.get(task_id, record["complexity"]), + } + for task_id, record in per_task.items() + } + for model, per_task in results.items() + } + + +def _fit_line(points: list[tuple[float, float]]) -> tuple[float, float]: + """Least-squares line through (difficulty sum, score) points. + + Falls back to a flat line at the mean when there are too few points or no + spread in difficulty, where a slope would be noise. + + Args: + points: ``(x, y)`` pairs. + + Returns: + ``(intercept, slope)``. + """ + ys = [y for _, y in points] + if len(points) < MIN_POINTS_FOR_FIT: + return (statistics.fmean(ys) if ys else 0.0), 0.0 + xs = [x for x, _ in points] + x_mean, y_mean = statistics.fmean(xs), statistics.fmean(ys) + sxx = sum((x - x_mean) ** 2 for x in xs) + if sxx == 0: + return y_mean, 0.0 + slope = sum((x - x_mean) * (y - y_mean) for x, y in points) / sxx + return y_mean - slope * x_mean, slope + + +def _predicted_stats( + results: dict[str, dict[str, dict[str, Any]]], + sums: dict[str, int], + task_id: str, + holdout: bool, +) -> dict[str, dict[str, Any]]: + """Each model's stats for routing one task, its score predicted from difficulty. + + For every model, fits score against the rubric's difficulty sum over its + scored runs on the other tasks (all tasks without ``holdout``) and predicts + its score at this task's sum. Every task informs every prediction, so no + model's estimate rests on a thin column. Failed runs are left out of the + fit, as they are left out of the column means; the completion count still + reports them. + + Args: + results: Output of ``_load_results``. + sums: Each task's consolidated difficulty sum from the rubric pass. + task_id: The task being routed. + holdout: Fit without the routed task. + + Returns: + ``_tier_stats``-shaped stats with the prediction under ``PREDICTED_TIER``. + """ + exclude = task_id if holdout else None + stats = _tier_stats(results, exclude_task=exclude) + for model, s in stats.items(): + points = [ + (float(sums[tid]), rec["score"]) + for tid, rec in results[model].items() + if tid != exclude and tid in sums and not rec["failed"] + ] + intercept, slope = _fit_line(points) + predicted = min(100.0, max(0.0, intercept + slope * sums[task_id])) + s["score_by_complexity"] = {PREDICTED_TIER: round(predicted, 2)} + s["completion_by_complexity"] = { + PREDICTED_TIER: f"{s['tasks_completed']}/{s['tasks_total']}" + } + return stats + + def _evaluate( tasks: list[dict[str, Any]], results: dict[str, dict[str, dict[str, Any]]], @@ -361,6 +460,10 @@ def _evaluate( holdout: bool, provenance: dict[str, Any], judged_tiers: dict[str, str] | None = None, + column_for: Callable[[str], dict[str, str]] | None = None, + stats_for: Callable[[str], dict[str, dict[str, Any]]] | None = None, + fixed_stats: dict[str, dict[str, Any]] | None = None, + min_completion: float = 0.0, ) -> dict[str, Any]: """Replay the router over every task and compare it to the baseline. @@ -377,6 +480,17 @@ def _evaluate( provenance: Provenance block for the synthesized models.json. judged_tiers: Per-task tier from a judged run, overriding the dataset's own ``complexity`` label. None keeps the dataset label. + column_for: Given the task being routed, the column of EVERY task, used + both to route it and to rebuild the columns it is routed against. + Lets the grouping itself depend on the held-out task, as relative + rubric buckets do. Overrides ``judged_tiers`` and the dataset label. + stats_for: Given the task being routed, every model's stats with one + predicted score under ``PREDICTED_TIER``. Overrides ``column_for``. + fixed_stats: Every model's stats from ANOTHER dataset, routed against + for every task. ``results`` then only supplies each routed task's + outcome. ``column_for`` still names the routed task's column. + min_completion: Passed to route.py: skip a model whose completion in + the column falls below this share. Returns: A result mapping with ``rows`` (one per task) and ``totals``. @@ -394,7 +508,8 @@ def _evaluate( tmp_dir = Path(tempfile.mkdtemp(prefix="router-eval-")) in_sample_path = tmp_dir / "models-in-sample.json" in_sample_path.write_text( - json.dumps(_models_json(_tier_stats(results), provenance)), encoding="utf-8" + json.dumps(_models_json(fixed_stats or _tier_stats(results), provenance)), + encoding="utf-8", ) for task in tasks: task_id = task["id"] @@ -403,14 +518,26 @@ def _evaluate( # the router a perfect classifier it would not have in real use. tier = judged_tiers.get(task_id, task["complexity"]) floor = floors.get(task_id, DEFAULT_FLOOR) - if holdout: + grouped = results + if column_for is not None: + columns = column_for(task_id) + tier = columns[task_id] + grouped = _regrouped(results, columns) + if fixed_stats is not None: + models_path = in_sample_path + elif stats_for is not None: + tier = PREDICTED_TIER models_path = tmp_dir / f"models-{task_id}.json" models_path.write_text( - json.dumps( - _models_json(_tier_stats(results, exclude_task=task_id), provenance) - ), + json.dumps(_models_json(stats_for(task_id), provenance)), encoding="utf-8", ) + elif holdout or column_for is not None: + models_path = tmp_dir / f"models-{task_id}.json" + stats = _tier_stats(grouped, exclude_task=task_id if holdout else None) + models_path.write_text( + json.dumps(_models_json(stats, provenance)), encoding="utf-8" + ) else: models_path = in_sample_path routed = _route_task( @@ -421,6 +548,7 @@ def _evaluate( allowed_file=allowed_file, no_allow_list=no_allow_list, tie_band=tie_band, + min_completion=min_completion, ) row = _row(task_id, tier, floor, routed, results, baseline) row["dataset_complexity"] = task["complexity"] @@ -553,9 +681,30 @@ def _totals(rows: list[dict[str, Any]], baseline: str) -> dict[str, Any]: "baseline_mean_score": round(base_mean, 2), "routed_mean_score": round(routed_mean, 2), "mean_score_delta": round(routed_mean - base_mean, 2), + # The same comparison over EVERY task, a failed run counted as 0. The + # paired means above leave out a task the routed model failed, which + # flatters a router that picks a model that then fails outright. + "baseline_mean_score_failures_as_zero": round( + statistics.fmean(r["baseline_score"] or 0.0 for r in rows), 2 + ) + if rows + else 0.0, + "routed_mean_score_failures_as_zero": round( + statistics.fmean(r["actual_score"] or 0.0 for r in rows), 2 + ) + if rows + else 0.0, # The router's own failure mode: it picked a model to clear a floor and # the model then landed under it. Counted for both arms so the baseline # is held to the same test. + "mean_score_delta_failures_as_zero": round( + statistics.fmean( + (r["actual_score"] or 0.0) - (r["baseline_score"] or 0.0) for r in rows + ), + 2, + ) + if rows + else 0.0, "tasks_below_floor_routed": sum(1 for r in rows if not r["met_floor"]), "tasks_below_floor_baseline": sum( 1 @@ -761,7 +910,10 @@ def _document(report: dict[str, Any]) -> str: cfg = report["config"] prov = report["provenance"] sampling = ( - "Leave-one-out: each task routes from tier means recomputed with that " + f"Zero-shot: every task routes from columns built on `{cfg['columns_from']}` " + "alone, a different repository, so no pick saw any task of this one." + if cfg.get("columns_from") + else "Leave-one-out: each task routes from tier means recomputed with that " "task excluded, so no pick knows the run it is scored against." if cfg["holdout"] else "In-sample: the router read tier means computed from these same " @@ -836,6 +988,104 @@ def _load_tasks(dataset_path: Path) -> tuple[list[dict[str, Any]], str]: return tasks, data.get("output_scope") or data.get("name") +def _apply_rubric_inputs( + path: Path, + tasks: list[dict[str, Any]], + bucketing: str, + holdout: bool, +) -> tuple[dict[str, float], Callable[[str], dict[str, str]], dict[str, Any]]: + """Route on a rubric pass instead of the skill's floor and tier. + + Each task's required quality becomes its floor and its difficulty bucket its + column, and the columns are rebuilt to match: every model's existing score + on a task is filed under that task's bucket rather than its dataset tier. + No run is repeated; only the grouping changes. + + ``absolute`` bucketing uses the bucket the rubric pass recorded. + ``relative`` cuts the tasks' difficulty sums into quarters -- with + ``holdout``, quarters of the OTHER tasks, so the routed task never shapes + the boundaries it is placed by. route.py still knows four tier names, so + the buckets travel as those keys (``BUCKET_TO_ROUTE_TIER``). + + Args: + path: A rubric-inputs JSON from ``run-swe-router-headless.py --mode rubric``. + tasks: Dataset tasks. + bucketing: ``absolute`` or ``relative``. + holdout: Leave the routed task out of the quarters (relative only). + + Returns: + Per-task floors, the per-task column function for ``_evaluate``, and the + rubric pass's metadata. + + Raises: + SystemExit: If the file is not a rubric pass or misses a task. + """ + data = json.loads(path.read_text(encoding="utf-8")) + if data.get("mode") != "rubric": + raise SystemExit(f"{path} is not a rubric pass (mode={data.get('mode')!r})") + scored = data.get("tasks") or {} + missing = {t["id"] for t in tasks} - set(scored) + if missing: + raise SystemExit( + f"--rubric-inputs is missing {len(missing)} task(s): {', '.join(sorted(missing))}" + ) + floors = {tid: float(v["required_quality"]) for tid, v in scored.items()} + sums = {tid: int(v["difficulty_sum"]) for tid, v in scored.items()} + absolute = {tid: BUCKET_TO_ROUTE_TIER[v["bucket"]] for tid, v in scored.items()} + + def column_for(routed: str) -> dict[str, str]: + if bucketing == "absolute": + return absolute + reference = [s for tid, s in sums.items() if not (holdout and tid == routed)] + edges = relative_edges(reference) + return { + tid: BUCKET_TO_ROUTE_TIER[relative_bucket(s, edges)] + for tid, s in sums.items() + } + + meta = dict(data.get("judged_by") or {}) + meta.update(mode="rubric", bucketing=bucketing) + return floors, column_for, meta + + +def _columns_from_other_dataset( + dataset_path: Path, + rubric_inputs: Path, + harness: str, + skill: str, +) -> tuple[dict[str, dict[str, Any]], str]: + """Every model's bucket columns, built from a different dataset. + + This is the zero-shot case: the router has only another repository's + benchmark to go on. Each of that dataset's tasks is filed under the bucket + its own rubric pass gave it, and the columns are its models' means there. + Buckets are absolute, so the routed tasks are placed by the same fixed + ranges. + + Args: + dataset_path: The dataset whose results build the columns. + rubric_inputs: That dataset's rubric pass. + harness: Harness folder. + skill: Skill folder. + + Returns: + ``_tier_stats`` output for that dataset, and its scope. + + Raises: + SystemExit: If the dataset has no results or the rubric pass misses a task. + """ + tasks, scope = _load_tasks(dataset_path) + results = _load_results(harness, skill, scope) + if not results: + raise SystemExit( + f"no results for --columns-from under {harness}/{skill}/{scope}" + ) + _, column_for, _ = _apply_rubric_inputs(rubric_inputs, tasks, "absolute", False) + known = {t["id"] for t in tasks} + columns = {tid: c for tid, c in column_for("").items() if tid in known} + return _tier_stats(_regrouped(results, columns)), scope + + def _parse_args() -> argparse.Namespace: """Parse command-line arguments.""" parser = argparse.ArgumentParser( @@ -878,6 +1128,45 @@ def _parse_args() -> argparse.Namespace: "tier}}}. Supplies BOTH the floor and the tier per task, replacing " "--floor/--floor-sweep and the dataset's complexity label.", ) + parser.add_argument( + "--rubric-inputs", + type=Path, + default=None, + help="JSON from run-swe-router-headless.py --mode rubric. Each task's " + "required quality is its floor and its difficulty bucket its column; " + "every model's existing scores are regrouped by bucket. Replaces " + "--judged-inputs, --floor and the dataset's complexity label.", + ) + parser.add_argument( + "--bucketing", + choices=("absolute", "relative"), + default=None, + help="Difficulty buckets for --rubric-inputs: absolute ranges, or " + "quarters of the tasks' own difficulty sums. Default: the routing " + "policy's difficulty.bucketing.", + ) + parser.add_argument( + "--column-model", + choices=("buckets", "regression"), + default="buckets", + help="With --rubric-inputs: compare the required quality against bucket " + "column means, or against each model's score predicted from the task's " + "difficulty sum by a line fitted over the other tasks. Default: %(default)s.", + ) + parser.add_argument( + "--columns-from", + type=Path, + default=None, + help="Zero-shot replay: build the bucket columns from THIS other dataset's " + "results instead of --dataset's, and route --dataset's tasks against them. " + "Needs --rubric-inputs and --columns-rubric-inputs; absolute buckets only.", + ) + parser.add_argument( + "--columns-rubric-inputs", + type=Path, + default=None, + help="The rubric pass for --columns-from's dataset.", + ) parser.add_argument( "--floors-file", type=Path, @@ -904,6 +1193,14 @@ def _parse_args() -> argparse.Namespace: default=DEFAULT_TIE_BAND, help="Points below which two models count as tied. Default: %(default)s.", ) + parser.add_argument( + "--min-completion", + type=float, + default=0.0, + help="Skip a model that finished less than this share of the tasks in " + "the column it is read from, as the shipped policy's " + "min_completion_in_bucket does (1.0). Default: %(default)s, the old rule.", + ) parser.add_argument( "--holdout", action="store_true", @@ -957,6 +1254,41 @@ def main() -> None: judged_tiers: dict[str, str] = {} judged_meta: dict[str, Any] = {} file_floors: dict[str, float] = {} + if args.rubric_inputs and args.judged_inputs: + raise SystemExit("use --rubric-inputs or --judged-inputs, not both") + column_for = None + stats_for = None + if args.column_model == "regression" and not args.rubric_inputs: + raise SystemExit("--column-model regression needs --rubric-inputs") + if args.rubric_inputs: + bucketing = args.bucketing or load_policy().difficulty.bucketing + file_floors, column_for, judged_meta = _apply_rubric_inputs( + args.rubric_inputs, tasks, bucketing, args.holdout + ) + if args.column_model == "regression": + scored = json.loads(args.rubric_inputs.read_text(encoding="utf-8"))["tasks"] + sums = {tid: int(v["difficulty_sum"]) for tid, v in scored.items()} + + stats_for = partial(_predicted_stats, results, sums, holdout=args.holdout) + judged_meta["column_model"] = "regression" + fixed_stats = None + if args.columns_from: + if not (args.rubric_inputs and args.columns_rubric_inputs): + raise SystemExit( + "--columns-from needs --rubric-inputs and --columns-rubric-inputs" + ) + if bucketing != "absolute" or args.column_model != "buckets" or args.holdout: + raise SystemExit( + "--columns-from supports absolute buckets only, without --holdout " + "(the routed tasks are never in the columns)" + ) + columns_path = args.columns_from + if not columns_path.is_absolute(): + columns_path = BENCHMARKS_DIR / columns_path + fixed_stats, columns_scope = _columns_from_other_dataset( + columns_path, args.columns_rubric_inputs, args.harness, args.skill + ) + judged_meta["columns_from"] = columns_scope if args.judged_inputs: judged = json.loads(args.judged_inputs.read_text(encoding="utf-8")) judged_meta = judged.get("judged_by") or {} @@ -998,21 +1330,31 @@ def main() -> None: holdout=args.holdout, provenance=provenance, judged_tiers=judged_tiers, + column_for=column_for, + stats_for=stats_for, + fixed_stats=fixed_stats, + min_completion=args.min_completion, ) except RouteError as exc: raise SystemExit(f"routing failed at floor {floor}: {exc}") from exc totals = evaluated["totals"] logger.info( "floor %.0f: %s -> $%.2f vs $%.2f baseline (%.1f%% saved), " - "mean score %.2f vs %.2f (%+.2f), %d task(s) under floor", + "mean score %.2f vs %.2f (%+.2f; %+.2f with failures as 0), " + "%d task(s) under floor", floor, - "leave-one-out" if args.holdout else "in-sample", + "zero-shot" + if fixed_stats + else "leave-one-out" + if args.holdout + else "in-sample", totals["routed_total_cost_usd"], totals["baseline_total_cost_usd"], totals["cost_saving_pct"], totals["routed_mean_score"], totals["baseline_mean_score"], totals["mean_score_delta"], + totals["mean_score_delta_failures_as_zero"], totals["tasks_below_floor_routed"], ) runs.append({"floor": None if file_floors else floor, **evaluated}) @@ -1030,9 +1372,15 @@ def main() -> None: "no_allow_list": args.no_allow_list, "tie_band": args.tie_band, "holdout": args.holdout, + "columns_from": judged_meta.get("columns_from"), + "min_completion": args.min_completion, "judged_by": judged_meta, "floor_note": ( - "Judged per task by " + "Required quality per task, from the rubric pass by " + f"{judged_meta.get('harness')} + {judged_meta.get('model')}: blast " + "radius and verification, priced by routing-policy.json." + if judged_meta.get("mode") == "rubric" + else "Judged per task by " f"{judged_meta.get('harness')} + {judged_meta.get('model')} " "running the skill's step 1 against the cloned repo -- the real " "judgment the skill asks for, not a policy constant. See " @@ -1049,7 +1397,13 @@ def main() -> None: + "." ), "tier_note": ( - "Classified per task by the same judged run, NOT read from the " + f"Difficulty bucket per task from the rubric pass " + f"({judged_meta.get('bucketing')} bucketing); every model's " + "existing scores regrouped into bucket columns (easy, moderate, " + "hard, very hard shown as trivial, low, medium, high). The " + "dataset's own label is kept beside each row." + if judged_meta.get("mode") == "rubric" + else "Classified per task by the same judged run, NOT read from the " "dataset. Each row carries the dataset's own `complexity` label " "beside it so disagreement is visible." if judged_meta diff --git a/benchmarks/scripts/run-swe-router-headless.py b/benchmarks/scripts/run-swe-router-headless.py index a50e0506..3251d950 100644 --- a/benchmarks/scripts/run-swe-router-headless.py +++ b/benchmarks/scripts/run-swe-router-headless.py @@ -46,6 +46,7 @@ import importlib.util import json import logging +import os import re import shutil import statistics @@ -77,6 +78,7 @@ RunnerConfigError, load_runner_config, ) +import swe_router_rubric as RUBRIC # noqa: E402 # The router skill lives with the repo's other skills. Its SKILL.md is inlined # into the prompt: omp has no --skill flag, and Claude Code's slash command is @@ -91,6 +93,18 @@ MIN_FLOOR = 55.0 MAX_FLOOR = 80.0 +# What the agent is asked to judge. "floor" is the shipped skill's step 1 (a +# consequence floor and a complexity tier); "rubric" scores the six rubric +# dimensions from vend/swe-router/rubric.md, which code then turns +# into a required quality and a difficulty bucket (swe_router_rubric.py). +MODE_FLOOR = "floor" +MODE_RUBRIC = "rubric" +MODES = (MODE_FLOOR, MODE_RUBRIC) +DEFAULT_FLOOR_OUT = REPO_ROOT / "docs" / "metrics" / "swe-router-judged-inputs.json" +# A rubric pass describes one dataset's tasks, so its default file name carries +# the dataset scope and the judging model: one file per dataset and judge. +RUBRIC_OUT_TEMPLATE = "swe-router-rubric-{scope}-{model}.json" + DEFAULT_DATASET = "dataset/mcp-gateway-registry-v2.yaml" DEFAULT_TIMEOUT_SECONDS = 900 DEFAULT_REPEATS = 3 @@ -136,6 +150,24 @@ def _utc_now_iso() -> str: return datetime.now(timezone.utc).isoformat().replace("+00:00", "Z") +def _default_out( + mode: str, + dataset: Dataset, + dataset_path: Path, + config: RunnerConfig, +) -> Path: + """Where to write the judged inputs when --out is not given. + + A rubric pass is named after its dataset and judge, so passes over two + datasets, or by two judges, never overwrite each other. + """ + if mode != MODE_RUBRIC: + return DEFAULT_FLOOR_OUT + scope = dataset.output_scope or dataset_path.stem + name = RUBRIC_OUT_TEMPLATE.format(scope=scope, model=config.model_slug) + return REPO_ROOT / "docs" / "metrics" / name + + def _build_prompt(task: Task, clone_path: Path) -> str: """Build the step-1 prompt: the skill, the task, and the output contract. @@ -271,7 +303,7 @@ def _omp_cmd(config: RunnerConfig, prompt: str) -> list[str]: The command argument vector. """ if config.is_bedrock: - model = f"{HARNESS.OMP_PROVIDER_BEDROCK}/{config.model}" + model = f"{HARNESS.OMP_PROVIDER_BEDROCK}/{HARNESS.model_to_wire_id(config.model or '')}" else: model = f"{HARNESS.OMP_PROVIDER_VLLM}/{config.model}" cmd = ["omp", "-p", "--mode", "json", "--no-session", "--model", model] @@ -336,6 +368,10 @@ def _run_omp_judgment( else: agent_dir.mkdir(parents=True, exist_ok=True) env = HARNESS._build_omp_env(config, agent_dir) + if config.is_bedrock: + # omp fuzzy-matches --model; warm this dir's catalog and prove the exact + # id is in it (see HARNESS._omp_bedrock_catalog). + HARNESS._require_omp_bedrock_model(config, env) start = time.time() events: list[dict[str, Any]] = [] # stdin=DEVNULL is required, not cosmetic: omp treats an inherited stdin as a @@ -371,6 +407,17 @@ def _run_omp_judgment( elapsed = time.time() - start result = HARNESS._pi_result_from_events(events, elapsed) result["_elapsed_seconds"] = round(elapsed, 1) + expected = ( + HARNESS.model_to_wire_id(config.model or "") + if config.is_bedrock + else config.model + ) + models_used = HARNESS._omp_models_used(events) + if expected and any(m != expected for m in models_used): + raise RuntimeError( + f"omp answered with {models_used} instead of the requested " + f"'{expected}': its --model fuzzy-matched a different model" + ) return _omp_final_text(events), result @@ -440,6 +487,8 @@ def _judge_task( task: Task, attempt: int, label: str, + mode: str = MODE_FLOOR, + rubric_text: str = "", ) -> dict[str, Any]: """Clone a task's repo, run one judgment in it, and return the result. @@ -452,6 +501,8 @@ def _judge_task( task: The task to judge. attempt: 1-based repeat index, recorded on the judgment. label: Log prefix. + mode: ``floor`` for the skill's step 1, ``rubric`` for the six scores. + rubric_text: The rubric, inlined into the prompt in rubric mode. Returns: A judgment record: the tuple plus provenance and cost, or an ``error``. @@ -461,8 +512,11 @@ def _judge_task( # cloning -- safe there (one run per task), unsafe here: two repeats of the # SAME task in flight would delete each other's checkout mid-run, and the # cleanup below would remove the survivor. Giving each attempt its own parent - # makes the collision impossible rather than merely unlikely. - clone_dir = str(Path(config.clone_dir) / f"router-attempt-{attempt}") + # makes the collision impossible rather than merely unlikely. The process id + # extends that to two passes run side by side (one judge model each), which + # otherwise clone the same task into the same folder and collide on git's + # lock file. + clone_dir = str(Path(config.clone_dir) / f"router-{os.getpid()}-attempt-{attempt}") clone_path = HARNESS._clone_repo(task, ref, clone_dir, log_prefix=label) record: dict[str, Any] = { "task": task.id, @@ -471,7 +525,12 @@ def _judge_task( "judged_at": _utc_now_iso(), } try: - prompt = _build_prompt(task, clone_path) + if mode == MODE_RUBRIC: + prompt = RUBRIC.build_prompt( + task.id, task.problem_statement or "", clone_path, rubric_text + ) + else: + prompt = _build_prompt(task, clone_path) if config.agent == AGENT_OMP: text, result = _run_omp_judgment( config, prompt, clone_path, clone_path.parent / "omp-agent" @@ -490,12 +549,16 @@ def _judge_task( "latency_seconds": metrics.get("latency_seconds"), "total_cost_usd": metrics.get("total_cost_usd"), } - record.update(_extract_judgment(text)) + if mode == MODE_RUBRIC: + record.update(RUBRIC.extract_judgment(text)) + verdict = " ".join(f"{d}={record['scores'][d]}" for d in RUBRIC.DIMENSIONS) + else: + record.update(_extract_judgment(text)) + verdict = f"floor={record['floor']} tier={record['tier']}" logger.info( - " %s judged floor=%s tier=%s (%ss, $%s)", + " %s judged %s (%ss, $%s)", label, - record["floor"], - record["tier"], + verdict, record["metrics"]["latency_seconds"], record["metrics"]["total_cost_usd"], ) @@ -556,6 +619,9 @@ def _collect( tasks: list[Task], repeats: int, concurrency: int, + mode: str = MODE_FLOOR, + rubric_text: str = "", + policy: RUBRIC.RoutingPolicy | None = None, ) -> tuple[list[dict[str, Any]], dict[str, Any]]: """Run every (task, repeat) judgment and consolidate the successful ones. @@ -565,6 +631,9 @@ def _collect( tasks: Tasks to judge. repeats: How many independent judgments per task. concurrency: How many judgments to run at once. + mode: ``floor`` or ``rubric``; picks the prompt and the consolidation. + rubric_text: The rubric, for rubric mode. + policy: The routing policy, for rubric mode's consolidation. Returns: Every judgment record, and the consolidated ``{task_id: tuple}`` map. @@ -588,6 +657,8 @@ def _collect( task, attempt, f"[{task.id} #{attempt}] {i} of {total}", + mode, + rubric_text, ): (task, attempt) for i, (task, attempt) in enumerate(jobs, start=1) } @@ -601,7 +672,12 @@ def _collect( if not good: logger.error("task %s produced no usable judgment", task.id) continue - consolidated[task.id] = _consolidate(good) + if mode == MODE_RUBRIC: + if policy is None: + raise ValueError("rubric mode needs a routing policy to consolidate") + consolidated[task.id] = RUBRIC.consolidate(good, policy) + else: + consolidated[task.id] = _consolidate(good) return records, consolidated @@ -688,6 +764,70 @@ def _judgments_markdown(payload: dict[str, Any]) -> str: return "\n".join(lines) +def _rubric_markdown(payload: dict[str, Any]) -> str: + """Render rubric-mode judgments as a markdown document. + + One row per task: the six consolidated scores, the required quality and + bucket code derives from them, and whether identical repeats agreed. + + Args: + payload: The full rubric-inputs mapping. + + Returns: + The markdown source. + """ + meta, tasks = payload["judged_by"], payload["tasks"] + short = { + "hidden_contract": "HC", + "silent_invariant": "SI", + "coordinated_surfaces": "CS", + "ambiguity": "AM", + "verification": "VE", + "blast_radius": "BR", + } + stable_q = sum(1 for v in tasks.values() if v["quality_unanimous"]) + stable_b = sum(1 for v in tasks.values() if v["bucket_unanimous"]) + lines = [ + f"# How {meta['harness']} + {meta['model']} scores each task on the rubric", + "", + f"The rubric pass over `{meta['dataset']}`: each task scored on six " + f"dimensions, 0 to 2, {meta['repeats']} independent time(s), in a clone " + "of its repository at the task's pinned ref. No model output was read; " + "this describes the tasks only. Code turns the scores into a required " + "quality and a difficulty bucket (`routing-policy.json`).", + "", + f"- **Stability.** Required quality agreed across repeats on " + f"{stable_q}/{len(tasks)} tasks; the bucket on {stable_b}/{len(tasks)}.", + f"- **Runs.** {meta['runs_ok']} succeeded, {meta['runs_failed']} failed. " + f"Cost ${meta.get('judging_cost_usd')}.", + f"- **Judged.** {meta['started_at']} to {meta['finished_at']}.", + "", + "Columns: " + + ", ".join(f"{s} {d.replace('_', ' ')}" for d, s in short.items()) + + ". Each score is the lower median of the repeats.", + "", + "| Task | " + + " | ".join(short.values()) + + " | Quality | Bucket | Confidence | Repeats agree |", + "|---|" + "---:|" * len(short) + "---:|---|---|---|", + ] + for task_id, v in tasks.items(): + agree = ( + "yes" + if v["quality_unanimous"] and v["bucket_unanimous"] + else f"quality {v['quality_seen']}; buckets {v['buckets_seen']}" + ) + cells = " | ".join(str(v["scores"][d]) for d in short) + lines.append( + f"| {task_id} | {cells} | **{v['required_quality']:g}** " + f"| {v['bucket']} | {v['confidence']} | {agree} |" + ) + lines += ["", "## What each task needs", ""] + for task_id, v in tasks.items(): + lines += [f"### {task_id}", "", v.get("summary") or "(no summary recorded)", ""] + return "\n".join(lines) + + def _select_tasks(dataset: Dataset, task_ids: list[str]) -> list[Task]: """Select tasks to judge, preserving dataset order. @@ -721,10 +861,30 @@ def _parse_args() -> argparse.Namespace: " --model us.anthropic.claude-opus-5 --repeats 3\n" " uv run scripts/run-swe-router-headless.py --tasks configurable-ui-title " "--repeats 1\n" + " uv run scripts/run-swe-router-headless.py --mode rubric --repeats 3\n" ), formatter_class=argparse.RawDescriptionHelpFormatter, ) parser.add_argument("--config", default=None, help="Runner config YAML path.") + parser.add_argument( + "--mode", + choices=MODES, + default=MODE_FLOOR, + help="floor: the skill's step 1 (floor and tier). rubric: the six " + "rubric dimensions. Default: %(default)s.", + ) + parser.add_argument( + "--rubric", + type=Path, + default=RUBRIC.DEFAULT_RUBRIC, + help="Rubric text for --mode rubric. Default: %(default)s.", + ) + parser.add_argument( + "--policy", + type=Path, + default=RUBRIC.DEFAULT_POLICY, + help="Routing policy for --mode rubric. Default: %(default)s.", + ) parser.add_argument( "--agent", default=AGENT_OMP, help="Coding agent: omp | claude." ) @@ -763,8 +923,10 @@ def _parse_args() -> argparse.Namespace: parser.add_argument( "--out", type=Path, - default=REPO_ROOT / "docs" / "metrics" / "swe-router-judged-inputs.json", - help="Where to write the judged inputs. Default: %(default)s.", + default=None, + help="Where to write the judged inputs. Default: " + "docs/metrics/swe-router-judged-inputs.json, or in rubric mode " + "docs/metrics/swe-router-rubric--.json.", ) parser.add_argument( "--out-md", @@ -793,7 +955,12 @@ def main() -> None: payload = json.loads(args.render.read_text(encoding="utf-8")) out_md = args.out_md or (REPO_ROOT / "docs" / f"{args.render.stem}.md") out_md.parent.mkdir(parents=True, exist_ok=True) - out_md.write_text(_judgments_markdown(payload), encoding="utf-8") + render = ( + _rubric_markdown + if payload.get("mode") == MODE_RUBRIC + else _judgments_markdown + ) + out_md.write_text(render(payload), encoding="utf-8") logger.info("wrote %s", out_md) return overrides = { @@ -823,15 +990,37 @@ def main() -> None: except DatasetError as exc: raise SystemExit(f"dataset error: {exc}") from exc tasks = _select_tasks(dataset, args.tasks.split(",") if args.tasks else []) + args.out = args.out or _default_out(args.mode, dataset, dataset_path, config) + rubric_text, policy = "", None + if args.mode == MODE_RUBRIC: + try: + rubric_text = args.rubric.read_text(encoding="utf-8") + policy = RUBRIC.load_policy(args.policy) + except (OSError, RUBRIC.RubricError) as exc: + raise SystemExit(f"cannot load the rubric or policy: {exc}") from exc if args.dry_run: clone = Path("/tmp/example-clone") # nosec B108 - illustrative path only - print(_build_prompt(tasks[0], clone)) + if args.mode == MODE_RUBRIC: + print( + RUBRIC.build_prompt( + tasks[0].id, tasks[0].problem_statement or "", clone, rubric_text + ) + ) + else: + print(_build_prompt(tasks[0], clone)) return started = _utc_now_iso() records, consolidated = _collect( - config, dataset, tasks, args.repeats, args.concurrency + config, + dataset, + tasks, + args.repeats, + args.concurrency, + args.mode, + rubric_text, + policy, ) costs = [ r["metrics"]["total_cost_usd"] @@ -844,8 +1033,13 @@ def main() -> None: "model": config.model, "provider": config.provider, "skill": "swe-router", - "step": "1 and 1b (consequence floor + complexity tier) only; " - "route.py is run separately by eval_swe_router.py", + "step": ( + "rubric pass: six dimensions scored 0-2; code derives required " + "quality and difficulty bucket" + if args.mode == MODE_RUBRIC + else "1 and 1b (consequence floor + complexity tier) only; " + "route.py is run separately by eval_swe_router.py" + ), "dataset": str(dataset_path.relative_to(REPO_ROOT)), "repeats": args.repeats, "started_at": started, @@ -854,8 +1048,14 @@ def main() -> None: "runs_ok": sum(1 for r in records if "error" not in r), "runs_failed": sum(1 for r in records if "error" in r), }, - "consolidation": "median floor, modal tier across repeats; per-task " - "spread kept in floors_seen / tiers_seen", + "mode": args.mode, + "consolidation": ( + "lower median per dimension across repeats; per-repeat quality and " + "bucket kept in quality_seen / buckets_seen" + if args.mode == MODE_RUBRIC + else "median floor, modal tier across repeats; per-task spread kept " + "in floors_seen / tiers_seen" + ), "tasks": consolidated, "judgments": records, } @@ -863,6 +1063,24 @@ def main() -> None: args.out.write_text(json.dumps(payload, indent=2) + "\n", encoding="utf-8") out_md = args.out_md or (REPO_ROOT / "docs" / f"{args.out.stem}.md") out_md.parent.mkdir(parents=True, exist_ok=True) + if args.mode == MODE_RUBRIC: + payload["rubric"] = str(args.rubric) + payload["policy"] = policy.model_dump(mode="json") if policy else None + args.out.write_text(json.dumps(payload, indent=2) + "\n", encoding="utf-8") + out_md.write_text(_rubric_markdown(payload), encoding="utf-8") + stable = sum(1 for v in consolidated.values() if v["bucket_unanimous"]) + logger.info( + "scored %d/%d task(s); %d run(s) failed; bucket unanimous on %d/%d; cost $%s", + len(consolidated), + len(tasks), + payload["judged_by"]["runs_failed"], + stable, + len(consolidated), + payload["judged_by"]["judging_cost_usd"], + ) + logger.info("wrote %s", args.out) + logger.info("wrote %s", out_md) + return out_md.write_text(_judgments_markdown(payload), encoding="utf-8") unstable = [t for t, v in consolidated.items() if not v["floor_unanimous"]] diff --git a/benchmarks/scripts/swe_router_rubric.py b/benchmarks/scripts/swe_router_rubric.py new file mode 100644 index 00000000..a59b01ae --- /dev/null +++ b/benchmarks/scripts/swe_router_rubric.py @@ -0,0 +1,328 @@ +"""The swe-router task rubric: scoring contract, policy and consolidation. + +The rubric replaces the skill's two free-form judgments (a consequence floor +and a complexity tier) with six dimensions scored 0, 1 or 2. A model scores the +dimensions; this module turns the scores into the two numbers routing needs: + +* **required quality**: the score a result must reach, from blast radius and + verification (``routing-policy.json``, ``required_quality``); +* **difficulty bucket**: which column of measured scores to compare against + it, from the other four dimensions (``difficulty``). + +The model never sees the numbers. It describes the task; the policy prices it. + +Everything here is a pure function so the judgment runner +(``run-swe-router-headless.py --mode rubric``) and the replay +(``eval_swe_router.py --rubric-inputs``) share one implementation. +""" + +from __future__ import annotations + +import json +import re +import statistics +from collections import Counter +from pathlib import Path +from typing import Any, Literal + +from pydantic import BaseModel, Field, field_validator, model_validator + +# The rubric and policy ship with the skill, so the skill's copy is the only one. +_SKILL_DIR = Path(__file__).resolve().parents[2] / "vend" / "swe-router" +DEFAULT_RUBRIC = _SKILL_DIR / "rubric.md" +DEFAULT_POLICY = _SKILL_DIR / "routing-policy.json" + +DIMENSIONS = ( + "hidden_contract", + "silent_invariant", + "coordinated_surfaces", + "ambiguity", + "verification", + "blast_radius", +) +SCORES = (0, 1, 2) +CONFIDENCE_LEVELS = ("high", "medium", "low") + +# Phase 0 replays through the existing route.py, which knows only its four tier +# names. The buckets are mapped onto those column keys, in order; the report +# names them by bucket. Phase 1 teaches route.py the bucket names directly. +BUCKET_ORDER = ("easy", "moderate", "hard", "very_hard") +BUCKET_TO_ROUTE_TIER = dict(zip(BUCKET_ORDER, ("trivial", "low", "medium", "high"))) + +_JSON_FENCE_RE = re.compile(r"```(?:json)?\s*(\{.*?\})\s*```", re.DOTALL) + + +class RubricError(ValueError): + """A rubric judgment or policy that cannot be used.""" + + +class RequiredQuality(BaseModel): + """How blast radius and verification set the score a result must reach.""" + + by_blast_radius: dict[int, float] + add_when_verification_is_2: float = Field(default=0, ge=0, le=20) + meaning: dict[str, str] = Field(default_factory=dict) + + @field_validator("by_blast_radius") + @classmethod + def _covers_every_score(cls, v: dict[int, float]) -> dict[int, float]: + if sorted(v) != list(SCORES): + raise ValueError( + f"by_blast_radius must define {list(SCORES)}, got {sorted(v)}" + ) + if any(not 0 < q <= 100 for q in v.values()): + raise ValueError("required quality values must be in (0, 100]") + return v + + +class Difficulty(BaseModel): + """Which dimensions add up to difficulty, and how the sum is bucketed. + + ``absolute`` uses the fixed ``buckets`` ranges. ``relative`` splits the + difficulty sums of the benchmark's own tasks into quarters, so "hard" means + harder than most of the work that was measured. Models score the rubric + generously, so fixed ranges put almost every task in the top two buckets; + quarters keep the ranking whatever the overall level. ``buckets`` stays the + fallback when no benchmark tasks are available to take quarters of. + + On the 21-task v2 set, quarters sorted tasks better but left columns of 3-12 + tasks, too few for steady per-model means, and routed worse than absolute + buckets. Prefer ``relative`` only for a task set large enough to fill four + columns. + """ + + dimensions: list[str] + bucketing: Literal["absolute", "relative"] = "absolute" + buckets: dict[str, tuple[int, int]] + + @model_validator(mode="after") + def _check(self) -> Difficulty: + unknown = set(self.dimensions) - set(DIMENSIONS) + if unknown: + raise ValueError(f"unknown difficulty dimension(s): {sorted(unknown)}") + if list(self.buckets) != list(BUCKET_ORDER): + raise ValueError(f"buckets must be {list(BUCKET_ORDER)} in that order") + expected = 0 + for name, (lo, hi) in self.buckets.items(): + if lo != expected or hi < lo: + raise ValueError( + f"bucket {name} must start at {expected} and not be empty" + ) + expected = hi + 1 + if expected - 1 != 2 * len(self.dimensions): + raise ValueError( + f"buckets must cover 0..{2 * len(self.dimensions)}, end at {expected - 1}" + ) + return self + + +class RoutingPolicy(BaseModel): + """The numbers a platform team owns: required quality and difficulty buckets.""" + + schema_version: str + required_quality: RequiredQuality + difficulty: Difficulty + tie_band_points: float = Field(default=3.0, ge=0) + min_completion_in_bucket: float = Field(default=1.0, ge=0, le=1) + when_unsure: str = "stay_on_current_model" + + +def load_policy(path: Path = DEFAULT_POLICY) -> RoutingPolicy: + """Load and validate a routing policy. + + Raises: + RubricError: If the file is missing, not JSON, or fails validation. + """ + try: + return RoutingPolicy.model_validate_json(path.read_text(encoding="utf-8")) + except FileNotFoundError as exc: + raise RubricError(f"no routing policy at {path}") from exc + except ValueError as exc: + raise RubricError(f"invalid routing policy {path}: {exc}") from exc + + +def required_quality(scores: dict[str, int], policy: RoutingPolicy) -> float: + """The score a result must reach for this task.""" + rq = policy.required_quality + base = rq.by_blast_radius[scores["blast_radius"]] + return base + (rq.add_when_verification_is_2 if scores["verification"] == 2 else 0) + + +def difficulty_sum(scores: dict[str, int], policy: RoutingPolicy) -> int: + """Sum of the difficulty dimensions.""" + return sum(scores[d] for d in policy.difficulty.dimensions) + + +def bucket(scores: dict[str, int], policy: RoutingPolicy) -> str: + """The difficulty bucket for these scores.""" + total = difficulty_sum(scores, policy) + for name, (lo, hi) in policy.difficulty.buckets.items(): + if lo <= total <= hi: + return name + raise RubricError(f"difficulty sum {total} falls in no bucket") + + +def relative_edges(sums: list[int]) -> tuple[float, float, float]: + """Quartile cut points of the benchmark tasks' difficulty sums. + + Raises: + RubricError: With fewer than four sums, where quarters mean nothing. + """ + if len(sums) < 4: + raise RubricError(f"relative buckets need at least 4 tasks, got {len(sums)}") + q1, q2, q3 = statistics.quantiles(sums, n=4, method="inclusive") + return q1, q2, q3 + + +def relative_bucket(total: int, edges: tuple[float, float, float]) -> str: + """Place a difficulty sum among quartile cut points. + + A sum equal to a cut point falls in the lower bucket, so a tie never + promotes a task. + """ + for name, edge in zip(BUCKET_ORDER, edges): + if total <= edge: + return name + return BUCKET_ORDER[-1] + + +def build_prompt(task_id: str, problem: str, clone_path: Path, rubric_text: str) -> str: + """The rubric-pass prompt: the rubric, where the repo is, and the output contract.""" + fields = ",\n".join( + f' "{d}": {{"score": <0, 1 or 2>, "evidence": ""}}' + for d in DIMENSIONS + ) + return "\n".join( + [ + rubric_text, + "", + "===TASK===", + "", + "Score the coding task below with the rubric above. Do NOT recommend a " + "model and do NOT write, edit or run any code in the repository.", + "", + f"The repository the change would land in is cloned at {clone_path} and " + "is your working directory. Read its map and whatever else the rubric " + "asks for; do not modify anything.", + "", + "Finish by printing EXACTLY one fenced JSON block and nothing after it:", + "", + "```json", + "{", + ' "scores": {', + fields, + " },", + f' "confidence": "",', + ' "summary": ""', + "}", + "```", + "", + f"Task id: {task_id}", + "", + "Task description:", + problem or "(see reference issue)", + ] + ) + + +def _candidates(text: str) -> list[str]: + """Fenced JSON blocks in the text, else bare objects that parse.""" + fenced = [m.group(1) for m in _JSON_FENCE_RE.finditer(text)] + if fenced: + return fenced + found = [] + for start in (i for i, ch in enumerate(text) if ch == "{"): + for end in range(len(text), start, -1): + try: + if isinstance(json.loads(text[start:end]), dict): + found.append(text[start:end]) + break + except json.JSONDecodeError: + continue + return found + + +def _validate(parsed: dict[str, Any]) -> dict[str, Any]: + """Check one parsed judgment; return it with scores flattened to ints.""" + raw = parsed.get("scores") + if not isinstance(raw, dict) or set(raw) != set(DIMENSIONS): + raise RubricError(f"scores must name exactly {list(DIMENSIONS)}") + scores, evidence = {}, {} + for dim in DIMENSIONS: + entry = raw[dim] + value = entry.get("score") if isinstance(entry, dict) else entry + if isinstance(value, bool) or value not in SCORES: + raise RubricError(f"{dim} score must be 0, 1 or 2, got {value!r}") + scores[dim] = int(value) + evidence[dim] = ( + str(entry.get("evidence", "")) if isinstance(entry, dict) else "" + ) + confidence = parsed.get("confidence") + if confidence not in CONFIDENCE_LEVELS: + raise RubricError( + f"confidence must be one of {CONFIDENCE_LEVELS}, got {confidence!r}" + ) + return { + "scores": scores, + "evidence": evidence, + "confidence": confidence, + "summary": str(parsed.get("summary", "")), + } + + +def extract_judgment(text: str) -> dict[str, Any]: + """Pull the rubric judgment out of the agent's final message. + + Takes the last block that validates, so a model that thinks aloud in JSON + before its final answer is still read correctly. + + Raises: + RubricError: If no block holds six valid scores and a confidence. + """ + errors = [] + for chunk in reversed(_candidates(text)): + try: + return _validate(json.loads(chunk)) + except (json.JSONDecodeError, RubricError) as exc: + errors.append(str(exc)) + detail = errors[0] if errors else "no JSON object found" + raise RubricError(f"no usable rubric judgment ({detail}): {text.strip()[-400:]!r}") + + +def consolidate( + judgments: list[dict[str, Any]], policy: RoutingPolicy +) -> dict[str, Any]: + """Reduce a task's repeated judgments to the scores routing will use. + + Each dimension takes the lower median of its repeats, so it stays one of + 0, 1 or 2. Required quality and bucket are computed from those scores, and + beside them the values each repeat produced on its own: a task whose + bucket moves between identical runs is a finding about the rubric. + + Raises: + RubricError: If there are no judgments. + """ + if not judgments: + raise RubricError("no successful judgments to consolidate") + scores = { + d: statistics.median_low(j["scores"][d] for j in judgments) for d in DIMENSIONS + } + per_run_quality = [required_quality(j["scores"], policy) for j in judgments] + per_run_bucket = [bucket(j["scores"], policy) for j in judgments] + confidence = Counter(j["confidence"] for j in judgments).most_common(1)[0][0] + return { + "scores": scores, + "required_quality": required_quality(scores, policy), + "difficulty_sum": difficulty_sum(scores, policy), + "bucket": bucket(scores, policy), + "confidence": confidence, + "summary": judgments[-1].get("summary", ""), + "attempts": len(judgments), + "scores_seen": { + d: sorted(j["scores"][d] for j in judgments) for d in DIMENSIONS + }, + "quality_seen": sorted(per_run_quality), + "buckets_seen": dict(Counter(per_run_bucket)), + "quality_unanimous": len(set(per_run_quality)) == 1, + "bucket_unanimous": len(set(per_run_bucket)) == 1, + } diff --git a/benchmarks/tests/test_build_vended_models.py b/benchmarks/tests/test_build_vended_models.py index 2d35766b..4df24461 100644 --- a/benchmarks/tests/test_build_vended_models.py +++ b/benchmarks/tests/test_build_vended_models.py @@ -2,9 +2,9 @@ from __future__ import annotations +import hashlib import importlib.util import json -import subprocess import sys import tempfile import unittest @@ -22,8 +22,12 @@ _VEND_DIR = _REPO_ROOT / "vend" / "swe-router" +# The generator's inputs -- the frontier JSON and the per-task run artifacts -- +# are not committed in this repository (see .gitignore), so the tests that read +# real data instead of a fixture have nothing to read. They stay in the file so +# they run wherever the data IS present. _SKIP_COMMITTED = not ( - _SCRIPTS_DIR.parent.parent / "docs" / "metrics" / "pareto-frontier-omp-swe3.json" + _REPO_ROOT / "docs" / "metrics" / "pareto-frontier-omp-swe3.json" ).exists() _SKIP_REASON = "docs/metrics/ frontier data not present in this repo" @@ -72,6 +76,19 @@ def _write(data: dict) -> Path: return Path(tmp.name) +def _runs(dates: dict[str, str | None]) -> Path: + """Write one omp/swe3/v2 run summary per model, with the given run_date.""" + root = Path(tempfile.mkdtemp()) + for model, run_date in dates.items(): + folder = root / model / "omp" / "swe3" / "mcp-gateway-registry-v2" + folder.mkdir(parents=True) + summary: dict = {"model_slug": model, "tasks": []} + if run_date: + summary["run_date"] = run_date + (folder / "run-summary.json").write_text(json.dumps(summary), "utf-8") + return root + + class BuildTest(unittest.TestCase): def test_includes_every_model_not_only_the_frontier(self) -> None: # The skill filters to what the user's assistant offers before ranking, @@ -94,7 +111,7 @@ def test_frontier_flags_are_documented_as_context_not_a_key(self) -> None: out = bvm.build(_write(_frontier()), _REPO_ROOT) note = out["measurement_basis"]["frontier_flags"] self.assertIn("not a selection key", note) - self.assertIn("score_by_complexity", note) + self.assertIn("score_by_difficulty", note) @unittest.skipIf(_SKIP_COMMITTED, _SKIP_REASON) def test_a_frontier_model_can_still_lose_at_a_tier(self) -> None: @@ -125,7 +142,8 @@ def test_incomplete_runs_stay_visible(self) -> None: def test_provenance_carries_the_measurement_context(self) -> None: # A vended file has no reader who knows this repo, so the context # travels with the data. - out = bvm.build(_write(_frontier()), _REPO_ROOT) + runs = _runs({"big": "2026-10-01", "small": "2026-09-01"}) + out = bvm.build(_write(_frontier()), _REPO_ROOT, runs_dir=runs) p = out["provenance"] self.assertEqual(p["harness"], "omp") self.assertEqual(p["skill"], "swe3") @@ -147,6 +165,97 @@ def test_empty_source_is_a_clear_error(self) -> None: with self.assertRaisesRegex(SystemExit, "no all_models"): bvm.build(_write(_frontier(all_models=[])), _REPO_ROOT) + def test_provenance_ignores_where_the_file_lives(self) -> None: + # The squash-merge case: the same bytes must stamp the same provenance, + # whatever commit or path they arrived under. + runs = _runs({"big": "2026-10-01"}) + first = bvm.build(_write(_frontier()), _REPO_ROOT, runs_dir=runs) + second = bvm.build(_write(_frontier()), _REPO_ROOT, runs_dir=runs) + keys = ("measured_on", "source_sha256") + self.assertEqual( + [first["provenance"][k] for k in keys], + [second["provenance"][k] for k in keys], + ) + + def test_source_sha256_changes_with_the_data(self) -> None: + runs = _runs({}) + before = bvm.build(_write(_frontier()), _REPO_ROOT, runs_dir=runs) + after = bvm.build(_write(_frontier(note="new")), _REPO_ROOT, runs_dir=runs) + self.assertNotEqual( + before["provenance"]["source_sha256"], + after["provenance"]["source_sha256"], + ) + + def test_measured_on_is_the_latest_run_of_a_frontier_model(self) -> None: + # "other" is not in the frontier JSON, so its later date is ignored. + runs = _runs( + {"big": "2026-10-01", "small": "2026-09-15", "other": "2026-12-31"} + ) + out = bvm.build(_write(_frontier()), _REPO_ROOT, runs_dir=runs) + self.assertEqual(out["provenance"]["measured_on"], "2026-10-01") + + def test_measured_on_is_null_without_any_run_date(self) -> None: + runs = _runs({"big": None, "small": None}) + out = bvm.build(_write(_frontier()), _REPO_ROOT, runs_dir=runs) + self.assertIsNone(out["provenance"]["measured_on"]) + + +def _rubric_file(tasks: dict[str, dict[str, int]], stored_bucket: str = "easy") -> Path: + """A rubric pass whose stored bucket deliberately disagrees with its scores.""" + body = { + "mode": "rubric", + "judged_by": {"model": "judge", "repeats": 3}, + "tasks": { + t: {"scores": scores, "bucket": stored_bucket} + for t, scores in tasks.items() + }, + } + return _write(body) + + +def _scores(difficulty: int) -> dict[str, int]: + """Six rubric scores whose four difficulty scores sum to ``difficulty``.""" + dims = ["hidden_contract", "silent_invariant", "coordinated_surfaces", "ambiguity"] + scores = {d: 0 for d in dims} | {"verification": 1, "blast_radius": 1} + for i in range(difficulty): + scores[dims[i % 4]] += 1 + return scores + + +class BucketColumnsTest(unittest.TestCase): + def _runs(self, rows: list[dict]) -> Path: + root = Path(tempfile.mkdtemp()) + folder = root / "m1" / "omp" / "swe3" / "ds" + folder.mkdir(parents=True) + summary = {"model_slug": "m1", "tasks": rows} + (folder / "run-summary.json").write_text(json.dumps(summary), "utf-8") + return root + + def test_buckets_are_recomputed_from_scores_not_read(self) -> None: + path = _rubric_file({"a": _scores(5), "b": _scores(1)}) + buckets = bvm.task_buckets(path, bvm.rubric.DEFAULT_POLICY) + self.assertEqual(buckets, {"a": "hard", "b": "easy"}) + + def test_columns_average_by_bucket_and_count_failures(self) -> None: + runs = self._runs( + [ + {"task": "a", "task_score": 80.0}, + {"task": "b", "task_score": 70.0}, + {"task": "c", "task_score": 0, "failed": True}, + ] + ) + stats = bvm.per_bucket_stats( + runs, "omp", "swe3", "ds", {"a": "hard", "b": "hard", "c": "hard"} + ) + self.assertEqual( + stats["m1"], {"score": {"hard": 75.0}, "completion": {"hard": "2/3"}} + ) + + def test_a_task_without_rubric_scores_is_an_error(self) -> None: + runs = self._runs([{"task": "a", "task_score": 80.0}]) + with self.assertRaisesRegex(SystemExit, "no rubric scores"): + bvm.per_bucket_stats(runs, "omp", "swe3", "ds", {}) + @unittest.skipIf(_SKIP_COMMITTED, _SKIP_REASON) class CommittedArtifactTest(unittest.TestCase): @@ -163,6 +272,18 @@ def test_committed_models_json_matches_the_generator(self) -> None: "vend/swe-router/models.json is stale; run build_vended_models.py", ) + def test_score_by_difficulty_is_present_for_every_model(self) -> None: + # The skill selects on this column; a model without it cannot be routed. + payload = json.loads(bvm.DEFAULT_OUT.read_text(encoding="utf-8")) + missing = [ + m["model"] for m in payload["models"] if not m["score_by_difficulty"] + ] + self.assertEqual(missing, []) + + def test_provenance_names_the_rubric_pass(self) -> None: + payload = json.loads(bvm.DEFAULT_OUT.read_text(encoding="utf-8")) + self.assertIn(bvm.RUBRIC_JUDGE, payload["provenance"]["rubric_scores"]["file"]) + def test_score_by_complexity_is_present_for_every_model(self) -> None: # The skill compares a floor against the tier, not the overall mean. # A model without tier scores silently falls back to the mean, which is @@ -201,35 +322,11 @@ def test_tier_scores_bracket_the_overall_mean(self) -> None: self.assertLessEqual(min(tiers), m["score"] + 0.01, m["model"]) self.assertGreaterEqual(max(tiers), m["score"] - 0.01, m["model"]) - def test_source_commit_tracks_the_frontier_not_head(self) -> None: - # Stamping HEAD would make the file differ after every unrelated commit - # and turn --check into noise. It must identify the data's version. + def test_source_sha256_matches_the_committed_frontier_bytes(self) -> None: + # What `sha256sum | cut -c1-12` prints, so anyone can check it. payload = bvm.build(bvm.DEFAULT_SOURCE, _REPO_ROOT) - stamped = payload["provenance"]["source_commit"] - head = subprocess.run( - ["git", "-C", str(_REPO_ROOT), "rev-parse", "--short", "HEAD"], - capture_output=True, - text=True, - check=False, - ).stdout.strip() - last_touch = subprocess.run( - [ - "git", - "-C", - str(_REPO_ROOT), - "log", - "-1", - "--format=%h", - "--", - str(bvm.DEFAULT_SOURCE.relative_to(_REPO_ROOT)), - ], - capture_output=True, - text=True, - check=False, - ).stdout.strip() - self.assertEqual(stamped, last_touch) - if head != last_touch: - self.assertNotEqual(stamped, head) + digest = hashlib.sha256(bvm.DEFAULT_SOURCE.read_bytes()).hexdigest()[:12] + self.assertEqual(payload["provenance"]["source_sha256"], digest) def test_vend_dir_holds_exactly_the_portable_files(self) -> None: # The portability contract: the skill must work in a directory holding @@ -237,7 +334,7 @@ def test_vend_dir_holds_exactly_the_portable_files(self) -> None: # Importing route.py in tests leaves a __pycache__; it is a build # artifact of running the checks, not something anyone installs. # install.sh is the installer, not part of the skill: it fetches the - # five installed files and never copies itself, so it cannot become a + # seven installed files and never copies itself, so it cannot become a # runtime dependency. test_installer_installs_exactly_the_skill_files # holds that line. self.assertEqual( @@ -250,6 +347,8 @@ def test_vend_dir_holds_exactly_the_portable_files(self) -> None: "model-aliases.json", "models.json", "route.py", + "routing-policy.json", + "rubric.md", ], ) @@ -272,40 +371,39 @@ def test_installer_installs_exactly_the_skill_files(self) -> None: "model-aliases.json", "models.json", "route.py", + "routing-policy.json", + "rubric.md", ], ) self.assertNotIn("install.sh", declared) - def test_readme_tier_tables_match_the_data(self) -> None: - # The README prints the four tier tables so a reader can see the scan. - # Hand-copied numbers drift; a regenerated models.json must not leave - # the documented tables quietly wrong. + def test_readme_bucket_table_matches_the_data(self) -> None: + # The README prints every model's per-bucket columns. Hand-copied + # numbers drift; a regenerated models.json must not leave the + # documented table quietly wrong. import re payload = json.loads(bvm.DEFAULT_OUT.read_text()) by = {m["model"]: m for m in payload["models"]} text = (_VEND_DIR / "README.md").read_text(encoding="utf-8") - checked = 0 - for tier in ("trivial", "low", "medium", "high"): - block = text.split(f"**`{tier}`** —", 1) - if len(block) < 2: - continue - table = block[1].split("\n\n", 2)[1] - rows = re.findall( - r"^\| \$([\d.]+) \| `([^`]+)` \| ([\d.]+) \|", table, re.M - ) - self.assertTrue(rows, f"no rows parsed for {tier}") - costs = [float(c) for c, _, _ in rows] - self.assertEqual(costs, sorted(costs), f"{tier} table not cheapest-first") - for cost, model, score in rows: - self.assertAlmostEqual( - float(score), by[model]["score_by_complexity"][tier], places=1 - ) - self.assertAlmostEqual( - float(cost), by[model]["cost_per_task_usd"], places=2 - ) - checked += 1 - self.assertGreaterEqual(checked, 60, "expected four full tier tables") + rows = re.findall( + r"^\| `([^`]+)` \| ([\d.]+) \| ([^|]+) \| ([^|]+) \| ([^|]+) \|$", + text, + re.M, + ) + self.assertEqual(sorted(r[0] for r in rows), sorted(by)) + costs = [float(r[1]) for r in rows] + self.assertEqual(costs, sorted(costs), "bucket table not cheapest-first") + for model, cost, *cells in rows: + m = by[model] + self.assertAlmostEqual(float(cost), m["cost_per_task_usd"], places=2) + for bucket, cell in zip(("moderate", "hard", "very_hard"), cells): + score = m["score_by_difficulty"].get(bucket) + done = m["completion_by_difficulty"].get(bucket, "") + expected = "--" if score is None else f"{score:.1f}" + if done and done.split("/")[0] != done.split("/")[1]: + expected += f" ({done})" + self.assertEqual(cell.strip(), expected, f"{model} {bucket}") def test_every_model_has_an_alias_entry(self) -> None: # A model with no aliases can never be matched against an assistant's diff --git a/benchmarks/tests/test_route.py b/benchmarks/tests/test_route.py index 91749047..81028585 100644 --- a/benchmarks/tests/test_route.py +++ b/benchmarks/tests/test_route.py @@ -82,12 +82,13 @@ def test_the_pick_is_never_dominated(self) -> None: ) def test_reads_the_tier_not_the_overall_mean(self) -> None: - # qwen3.8-27b averages 78.48 overall and 71.45 on high. A floor of 75 - # must reject it for hard work. + # claude-haiku-5-5 averages 72.69 overall but 79.16 on low and 60.00 + # on high. A floor of 75 must take it for low work and reject it for + # hard work, as it must qwen3.8-27b (78.48 overall, 71.45 on high). r = _route(tier="high", floor=75) - self.assertNotEqual(r["recommended"]["model"], "qwen3.8-27b") + self.assertNotIn(r["recommended"]["model"], {"claude-haiku-5-5", "qwen3.8-27b"}) r = _route(tier="low", floor=75) - self.assertEqual(r["recommended"]["model"], "qwen3.8-27b") + self.assertEqual(r["recommended"]["model"], "claude-haiku-5-5") def test_flags_a_margin_inside_the_noise(self) -> None: # 71.45 against a floor of 70 is not a reliable pass. @@ -189,21 +190,21 @@ def test_an_allow_list_with_only_comments_is_an_error(self) -> None: _route(tier="low", floor=50, allowed_file=p, no_allow_list=False) def test_the_shipped_lists_commented_suggestions_are_not_policy(self) -> None: - # The file suggests adding claude-sonnet-5 and claude-haiku-4-5 as - # commented lines. A comment is a comment. + # The file suggests the self-hosted kimi-k3, gemma-4-31b and + # qwen3.6-35b as commented lines. A comment is a comment. path = _VEND / "allowed-models.txt" names = route.parse_allow_list(path) - self.assertNotIn("claude-sonnet-5", names) - self.assertNotIn("claude-haiku-4-5", names) + self.assertNotIn("kimi-k3", names) + self.assertNotIn("qwen3.6-35b", names) self.assertIn( - "claude-sonnet-5", + "# kimi-k3", path.read_text(encoding="utf-8"), - "the file should still suggest sonnet as an addition", + "the file should still suggest kimi-k3 as an addition", ) def test_the_shipped_allow_list_parses(self) -> None: names = route.parse_allow_list(_VEND / "allowed-models.txt") - self.assertEqual(len(names), 6) + self.assertEqual(len(names), 5) self.assertIn("claude-opus-5", names) def test_the_shipped_allow_list_is_the_frontier(self) -> None: @@ -215,16 +216,15 @@ def test_the_shipped_allow_list_is_the_frontier(self) -> None: self.assertEqual(names, frontier) def test_the_shipped_allow_list_says_what_it_excludes(self) -> None: - # Four of the five are self-hosted, so the list as written leaves a - # hosted-API developer with one option. Shipping that silently would be - # a trap. + # qwen3.8-27b is self-hosted, so a hosted-API developer who keeps it + # would be sent to a model they cannot select. The file must say so. text = (_VEND / "allowed-models.txt").read_text(encoding="utf-8") - self.assertIn("claude-sonnet-5", text) - self.assertIn("self-hosted", text) + self.assertIn("qwen3.8-27b, is self-hosted", text) def test_the_shipped_allow_list_forces_opus_for_bedrock_users(self) -> None: # Recorded because it is the argument for editing the shipped list: - # four of its five models are self-hosted. + # an available model that is not on it (claude-sonnet-5, + # claude-haiku-4-5) is never recommended, however well it scores. r = _route( tier="high", floor=70, @@ -265,6 +265,182 @@ def test_skill_has_an_early_bail_out(self) -> None: def test_skill_states_it_runs_once_per_task(self) -> None: self.assertIn("Once per task", self.text) + def test_skill_names_no_measured_model(self) -> None: + # Model names and their scores belong in models.json, which the build + # regenerates. A name in SKILL.md goes stale the next time it does. + models = json.loads(MODELS.read_text(encoding="utf-8"))["models"] + named = [m["model"] for m in models if m["model"] in self.text] + self.assertEqual(named, []) + + def test_skill_runs_route_py_with_the_rubric(self) -> None: + self.assertIn("python3 route.py --confidence", self.text) + self.assertIn("rubric.md", self.text) + + +def _rubric(**overrides: int) -> dict[str, int]: + """Six rubric scores, all 1 unless overridden.""" + scores = {d: 1 for d in route.DIMENSIONS} + scores.update(overrides) + return scores + + +def _models_file(models: list[dict]) -> Path: + """A schema-2 models.json holding just these models.""" + tmp = tempfile.NamedTemporaryFile(mode="w", suffix=".json", delete=False) + json.dump({"schema_version": "2.0", "models": models}, tmp) + tmp.close() + return Path(tmp.name) + + +def _model(name: str, cost: float, scores: dict, completion: dict) -> dict: + return { + "model": name, + "score": 80.0, + "cost_per_task_usd": cost, + "score_by_difficulty": scores, + "completion_by_difficulty": completion, + } + + +class RubricParseTest(unittest.TestCase): + def test_accepts_plain_nested_and_wrapped_scores(self) -> None: + plain = _rubric() + nested = {d: {"score": v, "evidence": "x"} for d, v in plain.items()} + for text in (plain, nested, {"scores": nested}): + self.assertEqual(route.parse_rubric(json.dumps(text)), plain) + + def test_rejects_a_missing_dimension(self) -> None: + scores = _rubric() + del scores["ambiguity"] + with self.assertRaisesRegex(route.RouteError, "must score exactly"): + route.parse_rubric(json.dumps(scores)) + + def test_rejects_a_score_outside_0_to_2(self) -> None: + for bad in (3, -1, True, "1"): + with self.assertRaisesRegex(route.RouteError, "must be 0, 1 or 2"): + route.parse_rubric(json.dumps(_rubric(blast_radius=bad))) + + +class RubricPolicyTest(unittest.TestCase): + def setUp(self) -> None: + self.policy = route.load_policy(route.DEFAULT_POLICY) + + def test_shipped_policy_loads(self) -> None: + self.assertEqual( + list(self.policy["difficulty"]["buckets"]), list(route.BUCKETS) + ) + + def test_matches_the_benchmark_implementation_for_every_score(self) -> None: + # The build buckets benchmark tasks with swe_router_rubric.py; the skill + # buckets the developer's task with route.py. They must never disagree. + sys.path.insert(0, str(_REPO_ROOT / "benchmarks" / "scripts")) + import itertools + + import swe_router_rubric as rubric + + typed = rubric.load_policy(route.DEFAULT_POLICY) + for combo in itertools.product(route.SCORES, repeat=len(route.DIMENSIONS)): + scores = dict(zip(route.DIMENSIONS, combo)) + self.assertEqual( + ( + route.bucket_for(scores, self.policy), + route.required_quality(scores, self.policy), + ), + (rubric.bucket(scores, typed), rubric.required_quality(scores, typed)), + scores, + ) + + def test_blast_radius_and_verification_set_the_required_quality(self) -> None: + self.assertEqual( + route.required_quality(_rubric(blast_radius=0), self.policy), 60 + ) + self.assertEqual( + route.required_quality( + _rubric(blast_radius=2, verification=2), self.policy + ), + 80, + ) + + def test_a_malformed_policy_fails_with_a_message(self) -> None: + policy = json.loads(route.DEFAULT_POLICY.read_text(encoding="utf-8")) + policy["difficulty"]["buckets"]["hard"] = "4-5" + tmp = tempfile.NamedTemporaryFile(mode="w", suffix=".json", delete=False) + json.dump(policy, tmp) + tmp.close() + with self.assertRaisesRegex(route.RouteError, "bucket hard must be"): + route.load_policy(Path(tmp.name)) + + def test_a_missing_policy_is_an_error(self) -> None: + with self.assertRaisesRegex(route.RouteError, "no routing policy"): + route.load_policy(Path("/nonexistent/routing-policy.json")) + + +class RubricRoutingTest(unittest.TestCase): + def _go(self, models: list[dict], **kw) -> dict: + args = dict( + rubric=_rubric(), confidence="high", models_path=_models_file(models) + ) + args.update(kw) + return _route(tie_band=None, **args) + + def test_picks_the_cheapest_model_clearing_the_bucket_column(self) -> None: + # All ones: difficulty sum 4 (hard), blast radius 1 (70). + models = [ + _model("small", 1.0, {"hard": 72.0}, {"hard": "10/10"}), + _model("large", 9.0, {"hard": 85.0}, {"hard": "10/10"}), + ] + r = self._go(models) + self.assertEqual( + (r["bucket"], r["required_quality"], r["recommended"]["model"]), + ("hard", 70.0, "small"), + ) + + def test_skips_a_model_that_failed_a_task_in_the_bucket(self) -> None: + models = [ + _model("small", 1.0, {"hard": 90.0}, {"hard": "9/10"}), + _model("large", 9.0, {"hard": 85.0}, {"hard": "10/10"}), + ] + r = self._go(models) + self.assertEqual( + (r["recommended"]["model"], r["skipped_for_failures"][0]["model"]), + ("large", "small"), + ) + + def test_reads_the_next_harder_bucket_when_one_is_missing(self) -> None: + models = [_model("only", 1.0, {"hard": 75.0}, {"hard": "5/5"})] + r = self._go(models, rubric=_rubric(hidden_contract=0, ambiguity=0)) + self.assertEqual( + (r["bucket"], r["scored_from_harder_bucket"]), + ("moderate", {"only": "hard"}), + ) + + def test_low_confidence_stays_on_the_current_model(self) -> None: + r = self._go([], confidence="low") + self.assertEqual((r["status"], r["recommended"]), ("unsure", None)) + + def test_schema_1_models_cannot_serve_the_rubric(self) -> None: + tmp = tempfile.NamedTemporaryFile(mode="w", suffix=".json", delete=False) + json.dump( + {"schema_version": "1.1", "models": [{"model": "m", "score": 1}]}, tmp + ) + tmp.close() + with self.assertRaisesRegex(route.RouteError, "no score_by_difficulty"): + self._go([], models_path=Path(tmp.name)) + + def test_the_shipped_data_pick_is_never_dominated(self) -> None: + for combo in ((0, 0, 0, 0), (1, 1, 1, 1), (2, 2, 1, 1), (2, 2, 2, 2)): + for radius in (0, 1, 2): + dims = dict(zip(route.DIMENSIONS[:4], combo), blast_radius=radius) + r = _route(tie_band=None, rubric=_rubric(**dims), confidence="high") + pick = r["recommended"] + if not pick: + continue + for other in r["cleared_floor"]: + self.assertFalse( + other["score"] > pick["score"] + and other["cost_per_task_usd"] < pick["cost_per_task_usd"] + ) + class CliTest(unittest.TestCase): def test_runs_as_a_script_on_stdlib_alone(self) -> None: @@ -306,6 +482,66 @@ def test_exits_nonzero_when_nothing_is_recommended(self) -> None: self.assertEqual(out.returncode, 1) self.assertEqual(json.loads(out.stdout)["status"], "nothing_clears_floor") + def _cli(self, *argv: str) -> subprocess.CompletedProcess: + return subprocess.run( + [sys.executable, str(_VEND / "route.py"), "--no-allow-list", *argv], + capture_output=True, + text=True, + check=False, + ) + + def test_rubric_runs_as_a_script(self) -> None: + out = self._cli("--confidence", "high", "--rubric", json.dumps(_rubric())) + self.assertEqual( + (out.returncode, json.loads(out.stdout)["bucket"]), (0, "hard") + ) + + def test_needs_the_rubric_or_the_legacy_pair(self) -> None: + self.assertEqual(self._cli("--tier", "low").returncode, 2) + + def test_rejects_the_rubric_with_a_tier(self) -> None: + out = self._cli("--rubric", json.dumps(_rubric()), "--tier", "low") + self.assertEqual(out.returncode, 2) + + def test_an_old_python_gets_a_message_not_a_traceback(self) -> None: + code = ( + "import runpy, sys; sys.version_info = (3, 7, 0); " + f"sys.argv = ['route.py', '--tier', 'low', '--floor', '60']; " + f"runpy.run_path({str(_VEND / 'route.py')!r}, run_name='__main__')" + ) + out = subprocess.run( + [sys.executable, "-c", code], capture_output=True, text=True, check=False + ) + self.assertEqual( + (out.returncode, json.loads(out.stdout)["error"]), + (2, "route.py needs Python 3.8 or later"), + ) + + def test_installer_refuses_without_python3_and_writes_nothing(self) -> None: + # A stub python3 that fails stands in for a missing or too-old one. The + # check runs before any download, so this needs no network. + tmp = Path(tempfile.mkdtemp()) + (tmp / "bin").mkdir() + stub = tmp / "bin" / "python3" + stub.write_text("#!/bin/sh\nexit 1\n", encoding="utf-8") + stub.chmod(0o755) + env = {"PATH": f"{tmp / 'bin'}:/usr/bin:/bin", "HOME": str(tmp)} + out = subprocess.run( + ["bash", str(_VEND / "install.sh"), "--dir", str(tmp / "skills")], + capture_output=True, + text=True, + check=False, + env=env, + ) + self.assertEqual( + ( + out.returncode, + "Nothing was written" in out.stderr, + (tmp / "skills").exists(), + ), + (1, True, False), + ) + def test_route_py_imports_nothing_outside_the_standard_library(self) -> None: src = (_VEND / "route.py").read_text(encoding="utf-8") imports = set(re.findall(r"^(?:from|import)\s+([a-zA-Z_][\w.]*)", src, re.M)) diff --git a/benchmarks/tests/test_run_swe_router_headless.py b/benchmarks/tests/test_run_swe_router_headless.py index 15da1007..98dba4b5 100644 --- a/benchmarks/tests/test_run_swe_router_headless.py +++ b/benchmarks/tests/test_run_swe_router_headless.py @@ -11,6 +11,7 @@ import importlib.util import sys import unittest +from unittest import mock from pathlib import Path _SCRIPTS_DIR = Path(__file__).resolve().parent.parent / "scripts" @@ -201,6 +202,59 @@ class _Config: self.assertEqual(len(seen), 3) self.assertEqual(len(set(seen)), 3, f"clone dirs collided: {seen}") + def test_two_passes_side_by_side_get_different_clone_parents(self) -> None: + # Two processes judging with different models clone the same task at + # the same attempt number; the process id keeps their folders apart. + seen: list[str] = [] + + class _FakeHarness: + @staticmethod + def _clone_repo(task, ref, clone_dir, log_prefix=""): + seen.append(clone_dir) + raise RuntimeError("stop here; the clone dir is what is under test") + + class _Task: + id = "same-task" + problem_statement = "x" + repo = "https://example.com/r" + + class _Dataset: + @staticmethod + def resolved_ref(task): + return "main" + + class _Config: + clone_dir = "/tmp/router-test" # nosec B108 - test fixture path + + real = rr.HARNESS + rr.HARNESS = _FakeHarness + try: + for pid in (1111, 2222): + with mock.patch.object(rr.os, "getpid", return_value=pid): + with self.assertRaises(RuntimeError): + rr._judge_task(_Config(), _Dataset(), _Task(), 1, "l") + finally: + rr.HARNESS = real + self.assertEqual(len(set(seen)), 2, f"clone dirs collided: {seen}") + if __name__ == "__main__": unittest.main() + + +class DefaultOutTest(unittest.TestCase): + def _out(self, mode: str, scope: str | None) -> Path: + dataset = mock.Mock(output_scope=scope) + config = mock.Mock(model_slug="claude-opus-5-5") + return rr._default_out(mode, dataset, Path("dataset/some-set.yaml"), config) + + def test_rubric_file_names_the_dataset_scope_and_judge(self) -> None: + out = self._out(rr.MODE_RUBRIC, "cli-cli-v1") + self.assertEqual(out.name, "swe-router-rubric-cli-cli-v1-claude-opus-5-5.json") + + def test_rubric_file_falls_back_to_the_dataset_file_name(self) -> None: + out = self._out(rr.MODE_RUBRIC, None) + self.assertEqual(out.name, "swe-router-rubric-some-set-claude-opus-5-5.json") + + def test_floor_mode_keeps_its_fixed_name(self) -> None: + self.assertEqual(self._out(rr.MODE_FLOOR, "cli-cli-v1"), rr.DEFAULT_FLOOR_OUT) diff --git a/benchmarks/tests/test_swe_router_rubric.py b/benchmarks/tests/test_swe_router_rubric.py new file mode 100644 index 00000000..3d226003 --- /dev/null +++ b/benchmarks/tests/test_swe_router_rubric.py @@ -0,0 +1,383 @@ +"""Tests for the swe-router rubric: policy, mapping, parsing and consolidation.""" + +from __future__ import annotations + +import json +import sys +import tempfile +import unittest +from unittest import mock +from pathlib import Path + +_SCRIPTS_DIR = Path(__file__).resolve().parent.parent / "scripts" +sys.path.insert(0, str(_SCRIPTS_DIR)) + +import eval_swe_router as ev # noqa: E402 +import swe_router_rubric as r # noqa: E402 + + +def _scores(**overrides: int) -> dict[str, int]: + """All six dimensions at 0, with overrides.""" + base = {d: 0 for d in r.DIMENSIONS} + base.update(overrides) + return base + + +def _judgment(confidence: str = "high", **overrides: int) -> dict: + """A parsed judgment as extract_judgment returns it.""" + return { + "scores": _scores(**overrides), + "evidence": {}, + "confidence": confidence, + "summary": "s", + } + + +def _reply(scores: dict[str, int], confidence: str = "high") -> str: + """An agent reply carrying one fenced rubric block.""" + body = { + "scores": {d: {"score": v, "evidence": f"e-{d}"} for d, v in scores.items()}, + "confidence": confidence, + "summary": "needs a passthrough", + } + return f"I read the repo.\n```json\n{json.dumps(body)}\n```" + + +def _write_policy(data: dict) -> Path: + """Write a policy dict to a temp file.""" + tmp = tempfile.NamedTemporaryFile(mode="w", suffix=".json", delete=False) + json.dump(data, tmp) + tmp.close() + return Path(tmp.name) + + +class PolicyTest(unittest.TestCase): + def setUp(self) -> None: + self.policy = r.load_policy() + self.raw = json.loads(r.DEFAULT_POLICY.read_text(encoding="utf-8")) + + def test_shipped_policy_loads(self) -> None: + self.assertEqual(self.policy.required_quality.by_blast_radius[1], 70) + + def test_every_reachable_bar_has_a_meaning(self) -> None: + rq = self.policy.required_quality + bars = { + q + bump + for q in rq.by_blast_radius.values() + for bump in (0, rq.add_when_verification_is_2) + } + self.assertEqual({str(int(b)) for b in bars} - set(rq.meaning), set()) + + def test_missing_blast_radius_level_is_rejected(self) -> None: + self.raw["required_quality"]["by_blast_radius"].pop("2") + with self.assertRaisesRegex(r.RubricError, "by_blast_radius"): + r.load_policy(_write_policy(self.raw)) + + def test_gap_between_buckets_is_rejected(self) -> None: + self.raw["difficulty"]["buckets"]["moderate"] = [3, 3] + with self.assertRaisesRegex(r.RubricError, "must start at"): + r.load_policy(_write_policy(self.raw)) + + def test_buckets_must_cover_the_full_sum(self) -> None: + self.raw["difficulty"]["buckets"]["very_hard"] = [6, 7] + with self.assertRaisesRegex(r.RubricError, "cover"): + r.load_policy(_write_policy(self.raw)) + + def test_missing_policy_file_is_a_clear_error(self) -> None: + with self.assertRaisesRegex(r.RubricError, "no routing policy"): + r.load_policy(Path("/nonexistent/routing-policy.json")) + + +class MappingTest(unittest.TestCase): + def setUp(self) -> None: + self.policy = r.load_policy() + + def test_blast_radius_sets_the_bar(self) -> None: + self.assertEqual(r.required_quality(_scores(blast_radius=2), self.policy), 75) + + def test_unverifiable_work_raises_the_bar(self) -> None: + scores = _scores(blast_radius=1, verification=2) + self.assertEqual(r.required_quality(scores, self.policy), 75) + + def test_difficulty_dimensions_never_move_the_bar(self) -> None: + hard = _scores(hidden_contract=2, silent_invariant=2, ambiguity=2) + self.assertEqual(r.required_quality(hard, self.policy), 60) + + def test_bar_dimensions_never_move_the_bucket(self) -> None: + scores = _scores(blast_radius=2, verification=2) + self.assertEqual(r.bucket(scores, self.policy), "easy") + + def test_bucket_edges(self) -> None: + cases = {1: "easy", 2: "moderate", 5: "hard", 6: "very_hard"} + got = {} + for total in cases: + dims = dict.fromkeys(self.policy.difficulty.dimensions, 0) + for d in list(dims)[: total // 2]: + dims[d] = 2 + if total % 2: + dims[list(dims)[total // 2]] = 1 + got[total] = r.bucket(_scores(**dims), self.policy) + self.assertEqual(got, cases) + + +class ExtractTest(unittest.TestCase): + def test_reads_a_fenced_block(self) -> None: + out = r.extract_judgment(_reply(_scores(blast_radius=2))) + self.assertEqual( + (out["scores"]["blast_radius"], out["evidence"]["ambiguity"]), + (2, "e-ambiguity"), + ) + + def test_takes_the_last_valid_block(self) -> None: + text = _reply(_scores(ambiguity=1)) + "\n" + _reply(_scores(ambiguity=2)) + self.assertEqual(r.extract_judgment(text)["scores"]["ambiguity"], 2) + + def test_reads_a_bare_object_without_a_fence(self) -> None: + body = _reply(_scores(verification=1)).split("```json\n")[1].split("\n```")[0] + self.assertEqual( + r.extract_judgment("Answer: " + body)["scores"]["verification"], 1 + ) + + def test_rejects_a_score_off_the_scale(self) -> None: + with self.assertRaisesRegex(r.RubricError, "0, 1 or 2"): + r.extract_judgment(_reply(_scores(ambiguity=3))) + + def test_rejects_a_boolean_score(self) -> None: + with self.assertRaisesRegex(r.RubricError, "0, 1 or 2"): + r.extract_judgment(_reply({**_scores(), "ambiguity": True})) + + def test_rejects_a_missing_dimension(self) -> None: + scores = _scores() + scores.pop("verification") + with self.assertRaisesRegex(r.RubricError, "exactly"): + r.extract_judgment(_reply(scores)) + + def test_rejects_an_unknown_confidence(self) -> None: + with self.assertRaisesRegex(r.RubricError, "confidence"): + r.extract_judgment(_reply(_scores(), confidence="certain")) + + +class ConsolidateTest(unittest.TestCase): + def setUp(self) -> None: + self.policy = r.load_policy() + + def test_lower_median_keeps_scores_on_the_scale(self) -> None: + runs = [_judgment(ambiguity=1), _judgment(ambiguity=2)] + out = r.consolidate(runs, self.policy) + self.assertEqual(out["scores"]["ambiguity"], 1) + + def test_disagreeing_repeats_are_flagged(self) -> None: + runs = [_judgment(blast_radius=1), _judgment(blast_radius=2)] + out = r.consolidate(runs, self.policy) + self.assertEqual( + (out["quality_unanimous"], out["quality_seen"]), (False, [70.0, 75.0]) + ) + + def test_quality_and_bucket_come_from_the_consolidated_scores(self) -> None: + runs = [_judgment(blast_radius=2, hidden_contract=2, ambiguity=2)] * 3 + out = r.consolidate(runs, self.policy) + self.assertEqual((out["required_quality"], out["bucket"]), (75, "hard")) + + def test_no_judgments_is_an_error(self) -> None: + with self.assertRaisesRegex(r.RubricError, "no successful"): + r.consolidate([], self.policy) + + +class PromptTest(unittest.TestCase): + def test_prompt_carries_rubric_task_and_every_dimension(self) -> None: + prompt = r.build_prompt("t-1", "do it", Path("/tmp/clone"), "RUBRIC-TEXT") # nosec B108 + self.assertTrue( + all( + s in prompt for s in ("RUBRIC-TEXT", "t-1", "/tmp/clone", *r.DIMENSIONS) + ) # nosec B108 + ) + + def test_prompt_never_names_a_price_or_model(self) -> None: + # The model describes the task; the policy prices it. + prompt = r.build_prompt("t", "x", Path("/c"), r.DEFAULT_RUBRIC.read_text()) + self.assertFalse(any(w in prompt for w in ("required_quality", "$", "claude-"))) + + +class ReplayRubricInputsTest(unittest.TestCase): + def _inputs(self, tasks: dict, mode: str = "rubric") -> Path: + tmp = tempfile.NamedTemporaryFile(mode="w", suffix=".json", delete=False) + json.dump({"mode": mode, "judged_by": {"model": "m"}, "tasks": tasks}, tmp) + tmp.close() + return Path(tmp.name) + + def _five(self) -> dict: + sums = {"a": 2, "b": 4, "c": 5, "d": 6, "e": 8} + return { + t: {"required_quality": 70, "bucket": "hard", "difficulty_sum": s} + for t, s in sums.items() + } + + def test_absolute_uses_the_recorded_bucket_and_sets_floors(self) -> None: + path = self._inputs( + {"a": {"required_quality": 75, "bucket": "very_hard", "difficulty_sum": 7}} + ) + floors, column_for, meta = ev._apply_rubric_inputs( + path, [{"id": "a"}], "absolute", holdout=True + ) + self.assertEqual( + (floors["a"], column_for("a")["a"], meta["bucketing"]), + (75.0, "high", "absolute"), + ) + + def test_relative_quarters_leave_out_the_routed_task(self) -> None: + tasks = [{"id": t} for t in "abcde"] + _, column_for, _ = ev._apply_rubric_inputs( + self._inputs(self._five()), tasks, "relative", holdout=True + ) + # Routing e: quarters of a-d (2, 4, 5, 6), so e's 8 is very hard and a's 2 easy. + cols = column_for("e") + self.assertEqual((cols["a"], cols["e"]), ("trivial", "high")) + + def test_relative_columns_differ_by_which_task_is_held_out(self) -> None: + tasks = [{"id": t} for t in "abcde"] + _, column_for, _ = ev._apply_rubric_inputs( + self._inputs(self._five()), tasks, "relative", holdout=True + ) + self.assertNotEqual(column_for("a"), column_for("e")) + + def test_regrouping_does_not_touch_the_original_results(self) -> None: + results = {"m1": {"a": {"complexity": "trivial", "score": 80}}} + grouped = ev._regrouped(results, {"a": "high"}) + self.assertEqual( + (grouped["m1"]["a"]["complexity"], results["m1"]["a"]["complexity"]), + ("high", "trivial"), + ) + + def test_rejects_a_floor_mode_file(self) -> None: + with self.assertRaisesRegex(SystemExit, "not a rubric pass"): + ev._apply_rubric_inputs( + self._inputs({}, mode="floor"), [], "absolute", False + ) + + def test_rejects_a_file_missing_a_task(self) -> None: + with self.assertRaisesRegex(SystemExit, "missing 1 task"): + ev._apply_rubric_inputs(self._inputs({}), [{"id": "a"}], "absolute", False) + + +class ZeroShotColumnsTest(unittest.TestCase): + def _columns(self, rubric_tasks: dict) -> dict: + tmp = tempfile.NamedTemporaryFile(mode="w", suffix=".json", delete=False) + json.dump({"mode": "rubric", "tasks": rubric_tasks}, tmp) + tmp.close() + rec = lambda s: { # noqa: E731 + "score": s, + "cost_usd": 1.0, + "failed": False, + "complexity": "low", + } + results = {"m1": {"a": rec(90.0), "b": rec(60.0)}} + tasks = [{"id": "a", "complexity": "low"}, {"id": "b", "complexity": "low"}] + with ( + mock.patch.object(ev, "_load_tasks", return_value=(tasks, "other")), + mock.patch.object(ev, "_load_results", return_value=results), + ): + stats, _ = ev._columns_from_other_dataset( + Path("other.yaml"), Path(tmp.name), "omp", "swe3" + ) + return stats["m1"]["score_by_complexity"] + + def test_columns_are_the_other_datasets_bucket_means(self) -> None: + cols = self._columns( + { + "a": {"required_quality": 60, "bucket": "easy", "difficulty_sum": 0}, + "b": { + "required_quality": 70, + "bucket": "very_hard", + "difficulty_sum": 7, + }, + } + ) + self.assertEqual(cols, {"trivial": 90.0, "high": 60.0}) + + def test_rubric_entries_outside_that_dataset_are_ignored(self) -> None: + cols = self._columns( + { + "a": {"required_quality": 60, "bucket": "easy", "difficulty_sum": 0}, + "b": {"required_quality": 70, "bucket": "easy", "difficulty_sum": 1}, + "x": {"required_quality": 70, "bucket": "hard", "difficulty_sum": 4}, + } + ) + self.assertEqual(cols, {"trivial": 75.0}) + + +class RegressionColumnTest(unittest.TestCase): + def _results(self) -> dict: + # Model "steep" loses 10 points per difficulty step; "flat" does not. + sums = {"a": 0, "b": 2, "c": 4, "d": 6, "e": 8} + rec = lambda s, failed=False: { # noqa: E731 + "score": None if failed else s, + "cost_usd": 1.0, + "failed": failed, + "complexity": "low", + } + results = { + "steep": {t: rec(90 - 5 * x) for t, x in sums.items()}, + "flat": {t: rec(80.0) for t in sums}, + } + return results, sums + + def test_fit_recovers_a_clean_line(self) -> None: + a, b = ev._fit_line([(0, 90), (2, 80), (4, 70)]) + self.assertEqual((round(a, 6), round(b, 6)), (90, -5)) + + def test_too_few_points_fall_back_to_the_mean(self) -> None: + self.assertEqual(ev._fit_line([(1, 60), (3, 80)]), (70, 0.0)) + + def test_no_spread_in_difficulty_falls_back_to_the_mean(self) -> None: + self.assertEqual(ev._fit_line([(4, 60), (4, 70), (4, 80)]), (70, 0.0)) + + def test_prediction_falls_with_difficulty_for_a_degrading_model(self) -> None: + results, sums = self._results() + easy = ev._predicted_stats(results, sums, "a", holdout=True) + hard = ev._predicted_stats(results, sums, "d", holdout=True) + key = ev.PREDICTED_TIER + self.assertEqual( + ( + easy["steep"]["score_by_complexity"][key], + hard["steep"]["score_by_complexity"][key], + ), + (90.0, 60.0), + ) + + def test_holdout_leaves_the_routed_task_out_of_the_fit(self) -> None: + results, sums = self._results() + results["steep"]["d"]["score"] = 100.0 # an outlier on the routed task + pred = ev._predicted_stats(results, sums, "d", holdout=True) + self.assertEqual(pred["steep"]["score_by_complexity"][ev.PREDICTED_TIER], 60.0) + + def test_failed_runs_are_left_out_of_the_fit(self) -> None: + # Routing a and dropping failed b leaves c, d, e: still a clean line. + results, sums = self._results() + results["steep"]["b"] = { + "score": None, + "cost_usd": 1.0, + "failed": True, + "complexity": "low", + } + pred = ev._predicted_stats(results, sums, "a", holdout=True) + self.assertEqual(pred["steep"]["score_by_complexity"][ev.PREDICTED_TIER], 90.0) + + +class RelativeBucketTest(unittest.TestCase): + def test_needs_four_tasks(self) -> None: + with self.assertRaisesRegex(r.RubricError, "at least 4"): + r.relative_edges([1, 2, 3]) + + def test_a_tie_falls_to_the_lower_bucket(self) -> None: + edges = r.relative_edges([2, 4, 4, 5, 5, 5, 6, 6, 7, 7, 8]) + self.assertEqual(r.relative_bucket(5, edges), "moderate") + + def test_shipped_policy_is_absolute(self) -> None: + # Relative quarters sorted the 21 v2 tasks better but split them into + # columns of 3-12 tasks, too few for steady per-model means; absolute + # buckets routed better on that data (docs/swe-router-rubric-evaluation.md). + self.assertEqual(r.load_policy().difficulty.bucketing, "absolute") + + +if __name__ == "__main__": + unittest.main() diff --git a/docs/swe-router-evaluation-judged.md b/docs/swe-router-evaluation-judged.md index 77dc4566..64bd3df6 100644 --- a/docs/swe-router-evaluation-judged.md +++ b/docs/swe-router-evaluation-judged.md @@ -1,5 +1,8 @@ # Does the swe-router pay for itself? +> [!NOTE] +> **This measures the previous router.** It scores the version that judged a quality floor and a complexity tier in prose. The skill now routes on the six-question rubric in [vend/swe-router/rubric.md](../vend/swe-router/rubric.md), which replaced that judgment because it was unstable -- the instability this page documents is what motivated the change. The caveats below still describe where the hard part lives; the numbers are not the current router's. + > Across the 21 tasks the router selected **3 different models**: **Open dense 27B** 13x, **Closed frontier (gen 3)** 3x, **Open MoE 2.8T / 104B active** 1x. On 4 further task(s) nothing cleared the floor, so the skill's answer was to stay on the baseline. > > Against running the **Closed frontier (gen 3)** model on everything, that cost **45.4% less** ($137.16 against $251.04) for a quality change of **-4.6%** (78.99 against 82.83 mean task score, -3.84 points). diff --git a/docs/swe-router-evaluation.md b/docs/swe-router-evaluation.md index 580162fe..6ab3381a 100644 --- a/docs/swe-router-evaluation.md +++ b/docs/swe-router-evaluation.md @@ -1,5 +1,8 @@ # Does routing pay for itself? +> [!NOTE] +> **This measures the previous router.** It scores the version that judged a quality floor and a complexity tier in prose. The skill now routes on the six-question rubric in [vend/swe-router/rubric.md](../vend/swe-router/rubric.md), which replaced that judgment because it was unstable -- the instability this page documents is what motivated the change. The caveats below still describe where the hard part lives; the numbers are not the current router's. + Running `swe-router` against its own evidence: what it picked across 21 tasks, what those picks scored and cost, and the three caveats that decide how far to trust the result. The results above rank models on a whole dataset. A developer picks one per task, which is a different question -- and the usual answer is "run the best model for everything", which these numbers say is expensive. The **[`/swe-router`](../.claude/skills/swe-router/SKILL.md)** skill answers it per task: read the repository and the change, decide a quality floor from the consequence of getting it wrong and a complexity tier, then take the cheapest measured model that clears the floor at that tier. It recommends and stops. The developer makes the switch. diff --git a/vend/swe-router/README.md b/vend/swe-router/README.md index f83b616e..f3879560 100644 --- a/vend/swe-router/README.md +++ b/vend/swe-router/README.md @@ -22,7 +22,7 @@ That writes the skill to `.claude/skills/swe-router` in the current directory. T curl -sL https://raw.githubusercontent.com/aws-samples/sample-agentic-coding-harness-benchmarks/main/vend/swe-router/install.sh | bash -s -- --dir ~/.claude/skills ``` -`--ref` pins a tag or commit, and `--force` replaces an `allowed-models.txt` you have already edited. Pinning is also the supply-chain answer: `--ref ` installs a fixed revision you can read on GitHub first, rather than whatever `main` holds on the day you run it. Run `install.sh --help` for the rest. Re-running the installer upgrades the skill and keeps your edited `allowed-models.txt`, so an upgrade never silently replaces your model policy. +`--ref` pins a tag or commit, and `--force` replaces an `allowed-models.txt` or `routing-policy.json` you have already edited. Pinning is also the supply-chain answer: `--ref ` installs a fixed revision you can read on GitHub first, rather than whatever `main` holds on the day you run it. Run `install.sh --help` for the rest. Re-running the installer upgrades the skill and keeps your edited `allowed-models.txt` and `routing-policy.json`, so an upgrade never silently replaces your policy. If you would rather read a script than pipe it into a shell, download it first: @@ -31,21 +31,25 @@ curl -sL -O https://raw.githubusercontent.com/aws-samples/sample-agentic-coding- less install.sh && bash install.sh ``` -Five files land in the skill directory: +Seven files land in the skill directory: | File | | |---|---| | `SKILL.md` | the skill itself | -| `route.py` | the selection, as code. Standard library only — no install step | -| `models.json` | the measurements — 18 models, scores and cost per tier | +| `rubric.md` | the six questions the skill answers about a task | +| `route.py` | the selection, as code. Standard library only, no install step | +| `routing-policy.json` | how answers become a bucket and a required quality. **Yours to edit.** | +| `models.json` | the measurements: 23 models, scores and cost per bucket | | `model-aliases.json` | model names as different assistants spell them | | `allowed-models.txt` | which models your organisation permits. **Edit this one.** | -The path differs by assistant, which is what `--dir` is for. The skill has no dependencies and imports nothing — a directory of files is the whole install, and copying that directory by hand works just as well as the installer. +**It needs `python3`, 3.8 or later, on PATH, and no Python packages.** `route.py` uses only the standard library. The installer checks for it before downloading anything and stops with instructions if it is missing; `route.py` itself reports a too-old Python as an error rather than a traceback. + +The path differs by assistant, which is what `--dir` is for. The skill has no other dependencies and imports nothing outside the standard library, so a directory of files is the whole install, and copying that directory by hand works just as well as the installer. ### Restricting it to models your organisation allows -`allowed-models.txt` ships with the skill, allowing the five models on the measured frontier. **Edit it.** As written it is a starting point, not a policy: four of those five are self-hosted, and if you do not serve them the skill will recommend a model nobody can select. The file says so at the top and lists what to add. +`allowed-models.txt` ships with the skill, allowing the five models on the measured frontier. **Edit it.** As written it is a starting point, not a policy: one of those five, `qwen3.8-27b`, is self-hosted, and if you do not serve it the skill will recommend a model nobody can select. The file says so at the top and lists what to add. The file is also the candidate set, not just a filter. The skill treats every entry as a model the developer can actually select, and does not inspect the agent's model picker or provider config to check — a model reached through a proxy or driven by a second agent is invisible to that kind of check, so second-guessing the list mostly costs developers the cheaper option. That puts the whole promise on whoever maintains the file: **list a model and a developer can select it; if they cannot reach it, it does not belong on the list.** @@ -61,261 +65,143 @@ If it is firing when you do not want it, narrow the `Run it BEFORE…` sentence ## What the numbers mean -`models.json` holds 18 models measured on **21 software-engineering tasks** drawn from real closed issues in one open-source repository. Each task asks the model to take a problem from description to a working patch, producing six artifacts: a GitHub issue spec, a low-level design, an expert review, a testing plan, the patch, and an implementation summary. An LLM judge with access to the repository scores each artifact. +`models.json` holds 23 models measured on **21 software-engineering tasks** drawn from real closed issues in one open-source repository. Each task asks the model to take a problem from description to a working patch, producing six artifacts: a GitHub issue spec, a low-level design, an expert review, a testing plan, the patch, and an implementation summary. An LLM judge with access to the repository scores each artifact. | Field | Meaning | |---|---| -| `score` | Mean 0-100 across the tasks the model completed. Context; the skill selects on `score_by_complexity` | +| `score` | Mean 0-100 across the tasks the model completed. Context; the skill selects on `score_by_difficulty` | | `cost_per_task_usd` | Mean cost of one task | | `hosting` | `Bedrock` or `self-hosted`. Reported, never used to rank | | `tasks_completed` / `tasks_total` | Three models did not finish all 21 | -| `on_combined_frontier` | Nothing beats it on both axes across all 18 models. Context, not a selection key — it comes from overall means and can disagree with the per-tier ranking | +| `on_combined_frontier` | Nothing beats it on both axes across all 23 models. Context, not a selection key: it comes from overall means and can disagree with the per-bucket ranking | | `on_hosting_frontier` | The same, computed within one hosting basis, which is the apples-to-apples version | -| `score_by_complexity` | Mean score per tier — the number the floor is compared against | -| `completion_by_complexity` | How many tasks it finished per tier, where failure shows | +| `score_by_difficulty` | Mean score per difficulty bucket: the number the required quality is compared against | +| `completion_by_difficulty` | How many tasks it finished per bucket, where failure shows | +| `score_by_complexity` / `completion_by_complexity` | The same per hand-set dataset tier, for the legacy `--tier` path | ### Read these caveats before you trust a number **One repository.** Every task comes from a Python/FastAPI and React service with nginx, Terraform, Helm and bash around it. Rankings travel better than absolute scores. If you write Rust game engines, treat the ordering as a starting point and the numbers as indicative. -**One run per model per task, 5 or 6 per tier.** Small differences are noise. Dropping a single task from a tier reverses the order of two models 82% of the time when they sit within 1 point of each other, 53% within 2, and 47% within 3 — but only 5% past 5 points, and never past 8. The skill treats anything under 3 points at a tier as a tie and takes the cheaper model. +**One run per model per task, 2 to 10 per bucket.** Small differences are noise. In the earlier per-tier analysis, dropping a single task from a tier of five or six reversed the order of two models 82% of the time when they sat within 1 point of each other, 53% within 2, and 47% within 3, but only 5% past 5 points and never past 8. The skill treats anything under 3 points as a tie and takes the cheaper model. The moderate bucket holds only 2 tasks; read its column as a rough guide. **Self-hosted costs are measured under load.** Hosted figures come from a metered bill. Self-hosted figures come from the server's hourly price divided by throughput measured at a stated concurrency — a platform team serving a group of developers, not one person with an idle GPU. Both are a cost per task, and the skill ranks them together. -**These numbers move.** A fix to token accounting once changed `claude-opus-5` from $7.63 to $11.95 per task while every score stayed identical. Check `provenance.measured_on` and refetch if it looks old. +**These numbers move.** A fix to token accounting once changed `claude-opus-5` from $7.63 to $11.95 per task while every score stayed identical. Check `provenance.measured_on`, the date of the newest run in the file, and refetch if it looks old. `provenance.source_sha256` identifies the exact data: it is the first 12 hex digits of the SHA-256 of the frontier file it was built from, so two copies with the same value hold the same numbers. ## How it decides -Two questions, asked separately, because they answer different things. - -**How good does this have to be?** That is about consequence. A one-line change to an auth path can ruin someone's week; a fifty-line change to a scratch script cannot. Consequence sets a **floor** — the score a model has to reach before it is worth considering. - -**How hard is this?** That is about the work itself. It does not change the floor. It changes **which measured score** gets compared against the floor, because models fall off at very different rates as tasks get harder. - -Keeping them apart is the whole trick. Collapsing them — treating a big task as a high-stakes one — is how you end up paying for the top model to write a docs page. +The skill asks your assistant's model six questions about the task, from [`rubric.md`](rubric.md), and `route.py` turns the answers into a model. The rubric file carries each question, its scoring guide and the arithmetic; the walk-through below follows the same order. ``` - AGENTS.md ──▶ read the repo's own map - CLAUDE.md (fall back: README.md, CONTRIBUTING.md) + the task + the repository (AGENTS.md / CLAUDE.md / README.md first) + │ + ▼ + six rubric answers, 0-2 each, with evidence + │ + ┌─────────────┴──────────────┐ + ▼ ▼ + hidden_contract verification + silent_invariant blast_radius + coordinated_surfaces │ + ambiguity ▼ + │ REQUIRED QUALITY + ▼ 60 / 70 / 75 by blast radius, + sum 0-8 ──▶ BUCKET +5 when nothing would catch a mistake + easy / moderate / │ + hard / very hard │ + │ │ + ▼ │ + each model's mean score │ + on benchmark tasks in ◀── compare ┘ + that bucket (models.json) + │ + ▼ + allowed-models.txt ── keep permitted, measured models │ ▼ - the task ─────┬──────────────────────┐ - │ │ - ▼ ▼ - what happens if how hard is it? - this is wrong? │ - │ │ - ▼ ▼ - ┌────────────────┐ one table per tier, pick the matching one - │ prototype 55 │ ┌──────┐┌──────┐┌──────┐┌──────┐ - │ internal 65 │ │triv. ││ low ││medium││ high │ - │ production 70 │ │ ││ ││ ││ │ - │ security 75 │ └──────┘└──────┘└──────┘└──────┘ - └────────────────┘ beyond that range ──▶ strongest available, - │ say so, stop - │ │ - +5 if it turns │ picks which table - on one exact fact │ - │ │ - ▼ ▼ - FLOOR ◀── read against ── that tier's table - │ - ▼ - allowed-models.txt ── the candidate set: what you - │ permit AND can select - ┌─────────┴─────────┐ - ▼ ▼ - in models.json not measured - │ (named, excluded) - │ - ├── empty? say which test - │ emptied it, then stop - ▼ - cheapest at or above the floor - gaps under 3 pts count as ties - │ - ┌──────────┼──────────────┐ - ▼ ▼ ▼ - switch to already nothing - model X on it clears it - │ │ │ - say what stay put stay put, and - it costs name the shortfall + skip any that failed a benchmark task in the bucket + │ + ▼ + cheapest at or above the required quality + (gaps under 3 points count as ties) ``` -### Why the floor comes from consequence, not size +A **bucket** is the task's difficulty level. The four difficulty answers add up to a sum from 0 to 8: 0-1 is easy, 2-3 moderate, 4-5 hard, 6-8 very hard. The two consequence answers set the **required quality**, the benchmark score a model must reach. Your platform team owns both mappings in [`routing-policy.json`](routing-policy.json). -The natural instinct is to reach for a stronger model when a task looks big. The measurements do not support it. +### Why two separate numbers -On the benchmark, five tasks all classed as `trivial` — every one a single-file change — ranged from a **0.4-point** gap between the best and second-best model to a **13.8-point** gap. Three were mechanical: write a docs index, add two config passthroughs, narrow a render condition. On those, the expensive model bought nothing. Two turned on knowing one exact thing — that BSD `sed -i` swallows its next argument, that a version check has to compare rather than test for existence. On those it was worth 10 to 14 points. +How hard a task is and how bad a mistake would be are different questions. A one-line change to a password check is easy and dangerous. A twenty-file refactor of an internal dashboard is hard and harmless. Difficulty picks which column of measured scores to read; consequence sets the bar that column has to clear. Mixing them is how a team ends up paying for the top model to write a docs page. -Same size. Opposite answers. What separated them was whether the task hinged on getting one specific thing right, which is why that gets a +5 on the floor and size gets nothing. +### Why difficulty picks the column -### Why difficulty picks the table +Models lose points at different rates as tasks get harder, and some start failing them. On the benchmark, `claude-haiku-5-5` scores 83.2 on moderate tasks and 65.3 on very hard ones, with one very hard task unfinished. `claude-sonnet-5-5` drops from 86.5 to 77.2 and finishes all of them. An overall mean hides both facts. -Models do not degrade in parallel, so the running order changes from one table to the next. Six of the eighteen, ranked within each tier (`!` marks a tier where the model failed at least one task): +Each benchmark task got its bucket from the same six questions, answered three times by Claude Opus 5.5, so "hard" means the same thing in the column and in your request. The rubric scores live in the benchmark repository, in `docs/metrics/swe-router-rubric--.json`; the skill never reads them, only the columns built from them. -``` - trivial low medium high - ───────────────── ───────────────── ───────────────── ───────────────── - glm-5.3 85.0 opus-5 86.2 opus-5 81.8 opus-5 79.8 - opus-5 83.8 glm-5.3 82.6 qwen3.8 78.7 glm-5.3 74.7 - qwen3.8 81.4 qwen3.8 80.9 glm-5.3 78.5 sonnet-5 73.7 - sonnet-5 76.1 sonnet-5 80.5 sonnet-5 77.5 qwen3.8 71.5 ! - kimi-k2.7 72.4 kimi-k2.7 71.7 kimi-k2.7 69.6 kimi-k2.7 58.5 ! - gemma-4 66.5 gemma-4 63.3 gemma-4 59.3 gemma-4 50.0 -``` +### Why a failure skips a model -Read `qwen3.8` across the four. Third on trivial and low, second on medium, then **fourth on high, behind `sonnet-5`**, having failed one hard task. Its whole-dataset average of 78.48 beats sonnet's 76.97, which is why it sits on the published frontier and sonnet does not. On hard work that ordering is reversed. - -`glm-5.3` does the opposite: it tops the `trivial` table at 85.0, above `opus-5`, then slips behind `opus-5` on every harder tier, and behind `qwen3.8` on medium. - -**A recommendation built on the overall average is reading a table that does not exist.** There is no tier where those averages are the ranking. - -The `!` marks carry information no score can. A failure is excluded from the mean rather than averaged in, so `qwen3.8` scoring 71.5 on `high` is its average over the four hard tasks it finished — the fifth is simply absent. `completion_by_complexity` is where that shows, and the skill reports it rather than quoting the average of the ones that worked. - -How much difficulty matters depends on the pair. Between `claude-opus-5` and `claude-sonnet-5` it explains 4% of the gap; between `claude-opus-5` and `gemma-4-31b`, 43%. Two frontier models degrade together. A frontier model and a small one do not. - -### The four tables - -`score_by_complexity` is one table per tier, and this is what they look like. Each is sorted cheapest first, which is the order the skill reads them in: **scan down until the score clears your floor, and stop.** That first hit is the recommendation. - -A bold completion count means the model failed at least one task at that tier, which no score can show — a failure is excluded from the mean rather than averaged into it. - -**`trivial`** — scan down until the score clears your floor. - -| $/task | Model | Score | Finished | -|---:|---|---:|:-:| -| $0.26 | `qwen3.6-35b` | 58.4 | 5/5 | -| $0.47 | `qwen3-coder-30b` | 45.6 | 5/5 | -| $0.48 | `minimax-m2.5` | 54.7 | 5/5 | -| $0.76 | `claude-haiku-4-5` | 55.4 | 5/5 | -| $0.79 | `devstral-2-123b` | 56.5 | **4/5** | -| $0.87 | `gemma-4-31b` | 66.5 | 5/5 | -| $1.47 | `qwen3.8-27b` | 81.4 | 5/5 | -| $2.15 | `deepseek-v3.2` | 64.4 | 5/5 | -| $2.47 | `kimi-k2.7-code` | 72.4 | 5/5 | -| $3.51 | `minimax-m3` | 65.7 | 5/5 | -| $4.18 | `claude-opus-4-5` | 68.8 | 5/5 | -| $4.33 | `kimi-k3` | 81.8 | 5/5 | -| $4.67 | `claude-sonnet-5` | 76.1 | 5/5 | -| $4.95 | `claude-opus-4-6-v1` | 74.2 | 5/5 | -| $5.32 | `claude-opus-4-8` | 73.8 | 5/5 | -| $7.04 | `glm-5.3` | 85.0 | 5/5 | -| $7.35 | `claude-opus-4-7` | 77.8 | 5/5 | -| $11.95 | `claude-opus-5` | 83.8 | 5/5 | - -**`low`** — scan down until the score clears your floor. - -| $/task | Model | Score | Finished | -|---:|---|---:|:-:| -| $0.26 | `qwen3.6-35b` | 62.5 | 5/5 | -| $0.47 | `qwen3-coder-30b` | 48.4 | 5/5 | -| $0.48 | `minimax-m2.5` | 60.2 | 5/5 | -| $0.76 | `claude-haiku-4-5` | 57.8 | 5/5 | -| $0.79 | `devstral-2-123b` | 61.3 | **3/5** | -| $0.87 | `gemma-4-31b` | 63.3 | 5/5 | -| $1.47 | `qwen3.8-27b` | 80.9 | 5/5 | -| $2.15 | `deepseek-v3.2` | 63.9 | 5/5 | -| $2.47 | `kimi-k2.7-code` | 71.7 | 5/5 | -| $3.51 | `minimax-m3` | 60.6 | 5/5 | -| $4.18 | `claude-opus-4-5` | 66.2 | 5/5 | -| $4.33 | `kimi-k3` | 84.2 | 5/5 | -| $4.67 | `claude-sonnet-5` | 80.5 | 5/5 | -| $4.95 | `claude-opus-4-6-v1` | 75.4 | 5/5 | -| $5.32 | `claude-opus-4-8` | 81.1 | 5/5 | -| $7.04 | `glm-5.3` | 82.6 | 5/5 | -| $7.35 | `claude-opus-4-7` | 77.0 | 5/5 | -| $11.95 | `claude-opus-5` | 86.2 | 5/5 | - -**`medium`** — scan down until the score clears your floor. - -| $/task | Model | Score | Finished | -|---:|---|---:|:-:| -| $0.26 | `qwen3.6-35b` | 59.9 | 6/6 | -| $0.47 | `qwen3-coder-30b` | 40.8 | 6/6 | -| $0.48 | `minimax-m2.5` | 57.2 | 6/6 | -| $0.76 | `claude-haiku-4-5` | 57.8 | 6/6 | -| $0.79 | `devstral-2-123b` | 51.0 | **4/6** | -| $0.87 | `gemma-4-31b` | 59.3 | 6/6 | -| $1.47 | `qwen3.8-27b` | 78.7 | 6/6 | -| $2.15 | `deepseek-v3.2` | 62.1 | 6/6 | -| $2.47 | `kimi-k2.7-code` | 69.6 | 6/6 | -| $3.51 | `minimax-m3` | 56.8 | 6/6 | -| $4.18 | `claude-opus-4-5` | 67.9 | 6/6 | -| $4.33 | `kimi-k3` | 76.8 | 6/6 | -| $4.67 | `claude-sonnet-5` | 77.5 | 6/6 | -| $4.95 | `claude-opus-4-6-v1` | 68.3 | 6/6 | -| $5.32 | `claude-opus-4-8` | 73.3 | 6/6 | -| $7.04 | `glm-5.3` | 78.5 | 6/6 | -| $7.35 | `claude-opus-4-7` | 73.6 | 6/6 | -| $11.95 | `claude-opus-5` | 81.8 | 6/6 | - -**`high`** — scan down until the score clears your floor. - -| $/task | Model | Score | Finished | -|---:|---|---:|:-:| -| $0.26 | `qwen3.6-35b` | 56.0 | 5/5 | -| $0.47 | `qwen3-coder-30b` | 35.8 | 5/5 | -| $0.48 | `minimax-m2.5` | 51.4 | 5/5 | -| $0.76 | `claude-haiku-4-5` | 53.4 | 5/5 | -| $0.79 | `devstral-2-123b` | 40.8 | 5/5 | -| $0.87 | `gemma-4-31b` | 50.0 | 5/5 | -| $1.47 | `qwen3.8-27b` | 71.5 | **4/5** | -| $2.15 | `deepseek-v3.2` | 56.0 | 5/5 | -| $2.47 | `kimi-k2.7-code` | 58.5 | **3/5** | -| $3.51 | `minimax-m3` | 59.7 | 5/5 | -| $4.18 | `claude-opus-4-5` | 62.0 | 5/5 | -| $4.33 | `kimi-k3` | 74.3 | 5/5 | -| $4.67 | `claude-sonnet-5` | 73.7 | 5/5 | -| $4.95 | `claude-opus-4-6-v1` | 65.1 | 5/5 | -| $5.32 | `claude-opus-4-8` | 70.8 | 5/5 | -| $7.04 | `glm-5.3` | 74.7 | 5/5 | -| $7.35 | `claude-opus-4-7` | 74.4 | 5/5 | -| $11.95 | `claude-opus-5` | 79.8 | 5/5 | +A column mean leaves out the tasks a model failed, so a model that finished 8 of 9 very hard tasks looks as good as one that finished 9 of 9. The policy's `min_completion_in_bucket` (default 1.0) skips any model that failed a benchmark task in the task's bucket. `route.py` names what it skipped. -### Where the numbers stop +### The columns -The hardest tasks measured are bounded changes inside one repository — a rate-limiting subsystem, server-side OAuth token storage. Nothing here is a rewrite, a language port, or a change spanning several services. +Every measured model, cheapest first. A fraction marks a bucket where the model failed some tasks; with the default policy it is skipped there. The benchmark has no task in the easy bucket, so an easy task reads the moderate column, which can only understate a model. -Asked about work beyond that range, the skill recommends the strongest model available and tells you the measurements do not reach that far. Extrapolating a cheap model onto a job three orders of magnitude larger than anything tested is not a saving. +| Model | $/task | moderate (2 tasks) | hard (10) | very hard (9) | +|---|---:|---:|---:|---:| +| `claude-haiku-5-5` | 0.14 | 83.2 | 76.5 | 65.3 (8/9) | +| `qwen3.6-35b` | 0.26 | 62.6 | 62.3 | 55.1 | +| `minicpm5-2b` | 0.40 | 42.6 | 47.2 (9/10) | 37.4 (8/9) | +| `qwen3-coder-30b` | 0.47 | 52.4 | 45.4 | 37.2 | +| `minimax-m2.5` | 0.48 | 58.9 | 58.2 | 52.7 | +| `claude-haiku-4-5` | 0.76 | 62.7 | 57.0 | 53.8 | +| `devstral-2-123b` | 0.79 | 65.7 | 55.2 (6/10) | 44.5 (8/9) | +| `gemma-4-31b` | 0.87 | 68.6 | 63.0 | 54.2 | +| `qwen3.8-27b` | 1.47 | 83.6 | 81.3 | 73.6 (8/9) | +| `deepseek-v3.2` | 2.15 | 66.7 | 64.9 | 56.9 | +| `kimi-k2.7-code` | 2.47 | 79.9 | 71.9 | 62.1 (7/9) | +| `claude-sonnet-5-5` | 2.62 | 86.5 | 80.1 | 77.2 | +| `minimax-m3` | 3.51 | 74.3 | 60.9 | 57.0 | +| `kimi-k3-bedrock` | 3.87 | 83.6 | 73.1 | 70.2 | +| `claude-opus-4-5` | 4.18 | 68.0 | 70.9 | 60.9 | +| `kimi-k3` | 4.33 | 84.8 | 81.0 | 75.9 | +| `claude-opus-5-5` | 4.44 | 89.7 | 80.4 | 80.2 | +| `claude-sonnet-5` | 4.67 | 81.8 | 78.1 | 74.6 | +| `claude-opus-4-6-v1` | 4.95 | 78.8 | 73.8 | 65.3 | +| `claude-opus-4-8` | 5.32 | 81.1 | 74.9 | 73.0 | +| `glm-5.3` | 7.04 | 86.8 | 81.0 | 77.7 | +| `claude-opus-4-7` | 7.35 | 81.0 | 76.5 | 73.4 | +| `claude-opus-5` | 11.95 | 87.4 | 84.3 | 80.2 | + +### Where the numbers stop + +The hardest tasks measured are bounded changes inside one repository: a rate-limiting subsystem, server-side OAuth token storage. Nothing here is a rewrite, a language port, or a change spanning several services. Asked about work beyond that range, the skill recommends the strongest permitted model and says the measurements do not reach that far. ## Running the selection directly `route.py` is the whole decision procedure, and it works without the skill: ```bash -python3 route.py --tier high --floor 70 +python3 route.py --confidence high --rubric '{"hidden_contract": 1, "silent_invariant": 0, "coordinated_surfaces": 2, "ambiguity": 1, "verification": 1, "blast_radius": 1}' ``` -It prints JSON with the pick, the runners-up cheapest-first, how far above the floor the pick sits and whether that margin beats the noise, whether the model finished every task at that tier, and what policy excluded. `--allowed-file` points at another allow-list and `--no-allow-list` ignores it; `--available` narrows the set further, for the case where a developer says one permitted model is not wired up for them. - -### What it delivers, measured - -Running the router over the 21 benchmark tasks, using each task's real difficulty and then checking the recommended model's **actual score on that specific task**: +It prints JSON with the bucket, the required quality, the pick, the runners-up cheapest first, how far above the bar the pick sits and whether that beats the noise, and what was excluded or skipped. `--confidence low` returns `status: unsure` and keeps the developer on their model. `--policy` points at another policy file, `--allowed-file` at another allow-list, and `--no-allow-list` ignores it. -| Floor | $/task | Cleared the floor in reality | -|---:|---:|---| -| 55 | $0.26 | 16/21 (76%) | -| 60 | $1.04 | 17/21 (81%) | -| 65 | $1.33 | 17/21 (81%) | -| 70 | $1.47 | 18/21 (86%) | -| 75 | $3.05 | 18/21 (86%) | +The old interface, `python3 route.py --tier high --floor 70`, still works for one release. It reads the dataset's hand-set tiers (`score_by_complexity`) and does not skip models for failures. -`claude-opus-5` on the same 21 tasks clears a floor of 70 on **21 of 21**, at **$11.95** per task. +### What it delivers, measured -So the honest summary is not "the same quality for less". It is **86% of the bar for an eighth of the price**. Whether that trade is right depends on what a miss costs you, which is the question the floor was supposed to encode. +Replaying the router over the 21 benchmark tasks, each routed from columns that leave that task out, then looking up the chosen model's **actual score and cost on that task**, against running `claude-opus-5` on everything ($251.04 for the 21 tasks, mean score 82.83): -The gap has a cause worth knowing: a tier score is a *mean*, so a model sitting near the floor will fall below it on the harder half of that tier by construction. Asking for headroom fixes most of it: +| Router | Cost saved | Quality change | Tasks below the required quality | +|---|---:|---:|---:| +| Previous skill (floor and tier judged in prose) | 70.5% | −5.06 | 9 | +| Rubric | 75.3% | −3.75 | 4 | +| **Rubric with the failure rule (this version)** | **74.7%** | **−3.22** | **4** | -| Floor | Headroom | Hit rate | $/task | -|---:|---:|---|---:| -| 70 | +0 | 18/21 (86%) | $1.47 | -| 70 | +5 | 20/21 (95%) | $3.05 | -| 65 | +3 | 19/21 (90%) | $1.47 | -| 65 | +8 | 20/21 (95%) | $2.24 | +On a second repository, cli/cli in Go, columns built from this benchmark alone routed the tasks with no adjustment and erred towards the stronger model. The failure rule cannot foresee a failure the benchmark never recorded: `claude-haiku-5-5` finished all 10 hard tasks here and then failed a hard cli/cli task by writing its output to the wrong folder. -Going from 86% to 95% at a floor of 70 costs $1.58 per task and still runs at a quarter of always-opus. If a miss is expensive for you, raise the floor rather than distrusting the router. +These come from replaying the router against its own evidence with [`scripts/eval_swe_router.py`](../../benchmarks/scripts/eval_swe_router.py) in the benchmark repository: for whatever the skill picks, every model has already run every task, so the score and cost it would have got are a lookup rather than a new run. ## Regenerating the data @@ -327,8 +213,15 @@ uv run scripts/build_vended_models.py # rewrite models.json uv run scripts/build_vended_models.py --check # CI: fail if out of date ``` -`models.json` is generated from `docs/metrics/pareto-frontier-omp-swe3.json` and never edited by hand. +`models.json` is generated and never edited by hand. Scores and costs come from `docs/metrics/pareto-frontier-omp-swe3.json` and the run summaries behind it. Each benchmark task's bucket comes from the dataset's rubric pass, `docs/metrics/swe-router-rubric--claude-opus-5-5.json`, recomputed from its stored answers with `routing-policy.json`. A new model needs only a rebuild; a new dataset, or a change to `rubric.md`, needs a new rubric pass first: + +```bash +uv run scripts/run-swe-router-headless.py --mode rubric --agent omp --provider bedrock \ + --model us.anthropic.claude-opus-5-5 --dataset dataset/.yaml --repeats 3 +``` ## Source -Measurements, harness and methodology: [aws-samples/sample-agentic-coding-harness-benchmarks](https://github.com/aws-samples/sample-agentic-coding-harness-benchmarks). Design notes and open questions: issue #161. +Measurements, harness and methodology: [aws-samples/sample-agentic-coding-harness-benchmarks](https://github.com/aws-samples/sample-agentic-coding-harness-benchmarks). + +Whether these measurements carry over to a different repository is still open (issue #221): every task behind them comes from one Python/FastAPI and React service. Rankings travel further than absolute scores, so re-measure on your own code before trusting the numbers rather than the shape. diff --git a/vend/swe-router/SKILL.md b/vend/swe-router/SKILL.md index e5f33111..a1e7318f 100644 --- a/vend/swe-router/SKILL.md +++ b/vend/swe-router/SKILL.md @@ -1,27 +1,27 @@ --- name: swe-router -description: "Checks whether the current model is the right one for the coding task about to be done, and names a cheaper or stronger one when it is not. Uses measured scores and cost per task from a real 21-task benchmark rather than vendor claims, and usually recommends switching DOWN, because most developers run the top model for everything. Advisory only: it recommends a model in a few lines and stops -- it does not run the task, change any setting, or write code. The recommendation ENDS the turn: print it and hand back, so the developer can switch model before any work starts. Run it BEFORE starting a substantial coding task (a feature, a bug fix, a refactor, a migration), and whenever someone asks which model to use, whether a cheaper one would do, or whether they are overpaying. Do NOT run it for trivial requests, for questions, or once work on a task has already started." +description: "Checks whether the current model is the right one for the coding task about to be done, and names a cheaper or stronger one when it is not. Scores the task on a six-question rubric, then picks the cheapest permitted model whose measured benchmark score on tasks of the same difficulty clears the quality the task needs. Usually recommends switching DOWN, because most developers run the top model for everything. Advisory only: it recommends a model in a few lines and stops -- it does not run the task, change any setting, or write code. The recommendation ENDS the turn: print it and hand back, so the developer can switch model before any work starts. Run it BEFORE starting a substantial coding task (a feature, a bug fix, a refactor, a migration), and whenever someone asks which model to use, whether a cheaper one would do, or whether they are overpaying. Do NOT run it for trivial requests, for questions, or once work on a task has already started." license: Apache-2.0 metadata: author: Amit Arora - version: "0.1.0" + version: "0.2.0" --- # SWE Router -Tell the developer which model to switch to. Recommend, then stop — the recommendation is the end of your turn. +Tell the developer which model to switch to. Recommend, then stop: the recommendation is the end of your turn. -You do not run their task. You do not change a setting. You do not write code. The output is a recommendation, the reasoning behind it, and then control back to the developer. +You do not run their task. You do not change a setting. You do not write code. The output is a recommendation, the numbers behind it, and then control back to the developer. ## Why this runs here and not in a gateway -A gateway routes a request that already exists. By then it can see the prompt text, a token count, maybe a model hint. What it cannot see is the thing that decides this choice: what breaks if the change is wrong. +A gateway routes a request that already exists. By then it can see the prompt text, a token count, maybe a model hint. It cannot see what decides this choice: what breaks if the change is wrong, and what the repository already does to catch it. -That is knowable only while the work is still an idea. An auth path, a payments flow, a docs page -- the developer knows which one they are about to touch, and nothing on the wire recovers it. So the decision belongs where the task is framed, before a request exists. That is here. +You can see both, because you are in the repository while the work is still an idea. So the decision belongs here, before a request exists. ## Do not run when -Stop immediately, say nothing, and get on with what was asked, if any of these hold: +Stop at once, say nothing, and get on with what was asked, if any of these hold: - **The request is small.** A typo, a rename, one obvious line. The advice would cost more attention than the task. - **Work has already started.** Switching model mid-task loses the context built so far, which is worth more than the price difference. @@ -29,277 +29,176 @@ Stop immediately, say nothing, and get on with what was asked, if any of these h - **The developer already chose.** If they named a model, they have decided. Do not second-guess it unasked. - **You ran already for this task.** Once per task. A second opinion on the same work is noise. -Everything below assumes a substantial coding task about to begin: a feature, a bug fix, a refactor, a migration. When one is, be quick — the whole output is a few lines, and a developer who wanted a lecture would have asked for one. +Everything below assumes a substantial coding task about to begin: a feature, a bug fix, a refactor, a migration. Be quick. The whole output is a few lines. ## What you have to work with -These sit beside this one: +These sit beside this file: -- **`models.json`** — 18 models measured on a 21-task benchmark: mean score out of 100, cost per task in dollars, hosting, and how many tasks each finished. Read the `provenance` block: it names the date, harness, dataset and judge. -- **`model-aliases.json`** — the same models under the names different assistants use, plus the rules for matching them. -- **`route.py`** — runs the selection. Standard library only, so `python3 route.py` works wherever the skill is installed. -- **`allowed-models.txt`** — the organisation's approved model list, and a hard constraint when present. One model per line, `#` starts a comment. `route.py` finds and reads it; you never open it. It ships allowing the five models on the measured frontier, which a platform team is expected to edit. +- **`rubric.md`**: six questions about the task, each answered 0, 1 or 2, with what each answer means. You answer them. +- **`route.py`**: turns your six answers into a model. Standard library only, so `python3 route.py` works wherever the skill is installed. +- **`routing-policy.json`**: how `route.py` turns answers into numbers. A platform team may have edited it; you never need to open it. +- **`models.json`**: every measured model's mean benchmark score in each difficulty bucket, its cost per task, and how many benchmark tasks in each bucket it finished. Its `provenance` block names the date, dataset, harness and judge. +- **`model-aliases.json`**: the same models under the names different assistants use. +- **`allowed-models.txt`**: the models your organisation permits. `route.py` reads it; you never open it. -Read both before you answer. Never quote a score or a price from memory; if a model is not in `models.json` it has not been measured, and you say so rather than guessing. +Never quote a score or a price from memory. If a model is not in `models.json`, it has not been measured, and you say so rather than guess. ## Three steps -### 1. Understand the repo, then the task, then set a quality floor +### 1. Score the task with the rubric -**Start with the repo's own map.** Read `AGENTS.md` at the repository root. If there is none, read `CLAUDE.md`. If neither exists, fall back to `README.md` and `CONTRIBUTING.md`. +Read `rubric.md` and do what it says, in order: read the repository's map (`AGENTS.md`, else `CLAUDE.md`, else `README.md`, plus `SECURITY.md` if present), read the task, look at the code and tests it touches, then answer the six questions. Each answer is 0, 1 or 2, with one line of evidence: a file, a test, a document, or a sentence of the task. -That file is written for an agent working in this codebase, so it is the fastest route to what matters: the stack, the directory layout, where the tests live, what the project treats as risky. Use it to navigate rather than crawling the tree — a repo map read in one file beats twenty guesses at where things are. +The six questions do two jobs, and you do not need to work out either: -Two things to watch for. `AGENTS.md` sometimes points at other documents rather than holding the detail itself, so follow the links it names. And it describes the project's intent, not necessarily its current state — if it contradicts what you see in the code, trust the code. - -**Then look at the task.** What is being changed, which files, how much of the system it touches, and whether the repo map flags that area as sensitive. - -Then ask the question that decides the answer: **what happens if this change is wrong?** - -| The change is… | Floor | Because | -|---|---|---| -| a throwaway prototype, a spike, a scratch script | **55** | nobody merges it; wrong is cheap | -| an internal tool, a test fixture, a docs page | **65** | a human reads it before it matters | -| a production service, anything users touch | **70** | it ships; a defect reaches someone | -| auth, payments, data deletion, a security path | **75** | wrong is expensive and slow to find | - -The repo map often answers this for you. A project that calls a directory security-critical, or names a path as user-facing, has already told you the consequence of getting it wrong there. Use its language. - -Ask the developer only when the repo does not settle it. One question, not four. - -**The floor comes from consequence, not size.** A one-line change to an auth path needs a good model. A three-file mechanical edit does not. Do not raise the floor because the task looks big, or lower it because it looks small. - -**One adjustment worth making.** If the task hinges on getting a single specific thing right — an API contract, a portability trap, a security invariant, an exact version comparison — raise the floor by 5. That is where the measured gap between models opens up. - -### 1b. Also classify how hard the task is - -The floor is one half. The other half is **which table to read it against**. - -`score_by_complexity` is four tables, one per difficulty tier. Classifying the task picks the table; the floor then decides which row in it you take. A model has a different score in each, so reading the wrong table gives the wrong answer. - -Place the task in one of four tiers: - -| Tier | Looks like | +| Questions | What `route.py` makes of them | |---|---| -| `trivial` | a docs page, one render condition, two config passthroughs | -| `low` | a default changed across a couple of files, one small feature | -| `medium` | a feature touching several files, a config surface, a template rewrite | -| `high` | a subsystem: rate limiting, server-side token storage, a new authenticated endpoint | - -Take that tier's table from `score_by_complexity`, and read `completion_by_complexity` for the same tier beside it. Models do not degrade at the same rate between tables, and some stop finishing tasks at all in the harder ones: - -| Model | overall | on `low` | on `high` | finished on `high` | -|---|---:|---:|---:|:-:| -| `claude-opus-5` | 82.83 | 86.2 | 79.8 | 5/5 | -| `claude-sonnet-5` | 76.97 | 80.5 | 73.7 | 5/5 | -| `qwen3.8-27b` | 78.48 | 80.9 | 71.5 | **4/5** | -| `kimi-k2.7-code` | 69.13 | 71.7 | 58.5 | **3/5** | -| `gemma-4-31b` | 59.74 | 63.3 | 50.0 | 5/5 | - -Two things the overall mean hides. `qwen3.8-27b` outscores `claude-sonnet-5` overall (78.48 against 76.97) and trails it by 2.2 points on hard work — the ranking between them flips with the tier. And it did not finish one hard task at all, which the mean cannot show because a failure is excluded from it rather than averaged in. - -**A completion rate below the full count is a warning, not a rounding detail.** A model that finishes 4 of 5 hard tasks fails one in five outright. Say so when you recommend it, and prefer a model that finishes when the floor is close either way. - -Where `score_by_complexity` has no entry for a tier, fall back to the overall `score` and say you did. - -**When the task is beyond the measured range, say so.** The hardest tasks here are bounded single-repo changes. A language port, a framework migration, a rewrite, or anything spanning many services is outside what these numbers cover. Do not extrapolate: recommend the strongest model available and tell the developer the measurements do not reach that far. A weak model on an out-of-range task is not a saving. - -### 2. Build the candidate set - -Two things have to be true of a model before it can be recommended: the organisation permits it, and this benchmark measured it. **`route.py` applies both** — it reads `allowed-models.txt` and checks each name against `models.json`. Do not parse those files yourself; a second reading of them is a second answer waiting to disagree with the first. - -**Take `allowed-models.txt` as the list of models this assistant can select.** That is the contract with whoever maintains it: a platform team putting a model on that list is saying developers can reach it from the agent they run. Every entry is a candidate, whatever it is and wherever it runs. +| hidden_contract, silent_invariant, coordinated_surfaces, ambiguity | added up, they give the task's **bucket**: its difficulty level, one of easy, moderate, hard or very hard | +| verification, blast_radius | the **required quality**: the benchmark score a model must reach for this task | -So do not narrow the list a second time. Do not ask the assistant what its picker offers, do not go looking at the provider config, and **do not pass `--available`**. Guessing at what is reachable is how a permitted model gets dropped for no reason — and the guess is usually wrong, because a self-hosted model reached through a proxy or a second agent does not show up anywhere you would think to look. +Do not try to steer either number. Answer each question about the task as it would land in this repository, and let the policy price it. Size is not a signal, as `rubric.md` says: a one-line change to a password check scores higher on blast radius than a twenty-file change to an internal dashboard. -The one exception is the developer telling you otherwise. If they say a listed model is not wired up for them, pass the ones that are through `--available` and say you did. Their word overrides the file; your inference does not. +Then rate your confidence in the six answers as a whole: **high** when the repository and the task settle every one, **medium** when one or two rest on inference, **low** when you could not find what you needed. Low is a real answer. It keeps the developer on their current model, which is the right call when you cannot tell. -**When nothing survives, `route.py` returns `status: "no_candidates"` with a `reason`.** Read it out. Either nothing permitted was measured, or nothing measured is permitted — both belong to whoever maintains the list, and neither is something you route around. Stop there. Do not fall back to a model that failed one of the tests. +Ask the developer only when the repository cannot settle blast radius, the one answer that rests on how the change will be used. One question, not four. -**Getting a model is not your problem.** If a self-hosted model is on the list, someone stood up the server. If it is not, it is not a candidate. Never tell anyone to provision hardware. +**When the task is beyond the measured range, say so.** The hardest benchmark tasks are bounded changes inside one repository. A language port, a framework migration, a rewrite, or work spanning several services is outside what these numbers cover. Recommend the strongest permitted model and tell the developer the measurements do not reach that far. Do not run `route.py` for it. -**Do not treat a model differently because of where it runs.** What it costs to run is already in `cost_per_task_usd`, which is what you rank on. A self-hosted figure comes from the server's hourly price divided by throughput measured under concurrent load — a platform team serving a group of developers, which is how self-hosting is actually done — so it is a cost per task on the same footing as a metered one. Rank them together and say nothing about hosting unless the developer asks. +### 2. Run the selection -### 3. Run the selection - -You have the two inputs. `route.py` does the rest, so the arithmetic is arithmetic rather than something you work out in prose: +Pass your six answers and your confidence: ```bash -python3 route.py --tier high --floor 70 +python3 route.py --confidence high --rubric '{"hidden_contract": 1, "silent_invariant": 0, "coordinated_surfaces": 2, "ambiguity": 1, "verification": 1, "blast_radius": 1}' ``` -Two arguments, and that is the whole invocation. It finds `allowed-models.txt` on its own and ranks every permitted model that the benchmark measured. `--allowed-file` overrides the path; `--available` narrows the set further, for the one case in step 2 where the developer says a listed model is not reachable; `--no-allow-list` ignores policy entirely, which you should not do unasked. +**If `python3` is missing or the command fails to start, stop.** Tell the developer the skill needs Python 3.8 or later on PATH, with no packages, and recommend nothing. Do not work out the bucket, the required quality or the pick yourself: the arithmetic lives in `route.py` so that it is never done in prose. + +That is the whole invocation. `route.py` finds `allowed-models.txt` and `routing-policy.json` on its own. It works out the bucket and the required quality, keeps the permitted models the benchmark measured, skips any that failed a benchmark task in that bucket, and takes the cheapest whose score in that bucket clears the required quality. It prints JSON. The fields that matter: | Field | Use | |---|---| -| `status` | `ok`, `nothing_clears_floor`, or `no_candidates` | -| `recommended` | the model, its score at that tier, and cost per task | -| `margin_over_floor` / `margin_is_meaningful` | how far above the floor, and whether that is more than noise | -| `finished_every_task` / `completion` | whether it finished every task at this tier | -| `cleared_floor` | the runners-up, cheapest first — the next step up is the second entry | +| `status` | `ok`, `nothing_clears_floor`, `no_candidates`, or `unsure` | +| `bucket`, `difficulty_sum` | the task's difficulty level, and the sum of the four answers behind it | +| `required_quality` | the score a model must reach, from blast radius and verification | +| `recommended` | the model, its score in that bucket, its cost per task, and how many of the bucket's tasks it finished | +| `margin_over_floor` / `margin_is_meaningful` | how far above the required quality, and whether that is more than noise | +| `cleared_floor` | every model that cleared, cheapest first; the next step up is the second entry | +| `skipped_for_failures` | models that would have cleared but failed a benchmark task in this bucket | +| `scored_from_harder_bucket` | models read from a harder bucket because the benchmark had no task in this one | | `reason` | why nothing was recommended, when nothing was | -| `excluded` | what policy or availability removed, so you can name it | - -**Report what it returns; do not re-derive it.** If the numbers in the JSON disagree with your reading of `models.json`, the JSON is right. +| `excluded` | what the allow-list removed, so you can name it | -Run it once. If you find yourself running it twice with different floors to get a nicer answer, stop — the floor came from the consequence of being wrong, and that has not changed. +**Report what it returns; do not re-derive it.** If the JSON disagrees with your reading of `models.json`, the JSON is right. Run it once. If you catch yourself running it again with different answers to get a nicer result, stop: the task has not changed. -**Treat a gap under 3 points at a tier as no gap at all.** Each tier holds 5 or 6 tasks, run once each, so a small difference between two models is sampling noise rather than a finding. Dropping a single task from a tier reverses the order of two models 82% of the time when they sit within 1 point, 53% within 2, and 47% within 3. Past 5 points it reverses 5% of the time, and past 8 it never does. +### 3. Read the result and say it -So when the candidates are within 3 points of each other at the task's tier, they are indistinguishable: **take the cheaper one and say they were tied**. Do not present a 1.4-point difference as a reason to prefer a model. +**The candidate set is the allow-list.** A platform team that puts a model on `allowed-models.txt` is saying developers can reach it from this assistant. Do not narrow the list a second time: do not inspect the model picker, do not read provider config, and do not pass `--available`. Guessing at what is reachable drops permitted models for no reason. The one exception is the developer telling you a listed model is not wired up for them; then pass the ones that are through `--available` and say you did. -The same applies to the floor itself. A model scoring 71 against a floor of 70 has not reliably cleared it. Recommend it if nothing better is available, and say it sits inside the margin. +**When `status` is `no_candidates`,** read out the `reason`. Either nothing permitted was measured, or nothing measured is permitted. Both belong to whoever maintains the list. Stop there. -That is the whole selection rule. There is no separate step that drops dominated models, because taking the cheapest model above the floor already yields a non-dominated answer — anything that beat it on both axes would have been cheaper, and would have been picked instead. +**When `status` is `unsure`,** your confidence was low. Recommend staying on the current model and say which answer you could not settle. -**Read the tier's table, never the frontier flags.** `models.json` marks each model with `on_combined_frontier` and `on_hosting_frontier` — nothing else beats it on both score and cost across the whole dataset. That is worth mentioning when you recommend a model, and it is not how you choose one. Those flags come from overall means, so they can disagree with the tier that matters: `qwen3.8-27b` is on the combined frontier and still trails `claude-sonnet-5` on high-complexity work. `on_hosting_frontier` compares within one hosting basis, `on_combined_frontier` across both. +**Treat a gap under 3 points as no gap.** Each bucket holds a handful of benchmark tasks, run once each, so a small difference between two models is noise. When candidates sit within 3 points, take the cheaper one and say they tied. The same goes for the required quality: a model 1 point above it has not reliably cleared it. `margin_is_meaningful` tells you which case you are in; recommend the model anyway if nothing better is permitted, and say it sits inside the margin. -The ranking is worked out here in any case, over the models this developer can actually select, at the tier this task actually sits in. A published frontier answers neither question. +**Select on the bucket, never on the frontier flags.** `on_combined_frontier` and `on_hosting_frontier` in `models.json` come from overall means. Mention them as context; do not choose by them. -Then say it plainly, in the block under *The format of the answer* — that layout is required, not a suggestion. **The first line names the recommended model**, every time — that is the output, and burying it under the reasoning wastes the developer's attention. Say the model `route.py` returned in `recommended`, not a hedge about what you would have picked under other conditions: +**Do not treat a model differently because of where it runs.** `cost_per_task_usd` already holds what it costs. A self-hosted figure is the server's hourly price divided by throughput measured under load, so it sits on the same footing as a metered price. Rank them together and say nothing about hosting unless asked. Never tell anyone to provision hardware. -- **A candidate clears the floor** → `Recommendation: switch to `. Give its score at the task's tier and its cost per task, say what they are on now and what changes. Mention if it is on the frontier, as context. If a better model exists above it, say what the next step up would cost per extra point, so they can overrule you. -- **They are already on it** → `Recommendation: stay on `. Still name it, and still show the runners-up you ranked it against, so they can see the call was made rather than skipped. "Stay where you are" is a real answer and it is often the right one. -- **Nothing clears the floor** → say that on the first line. Name the closest, how far short it falls, and let them decide. Do not promote a model past its measured score to produce a recommendation. +Then say it, in the layout under *The format of the answer*: -The recommendation is the one `route.py` returned over the permitted models. Do not qualify it with a model you left out, and never present a model as unavailable on your own inference — if you narrowed the set at all, it was because the developer told you to, and you say which of their words you acted on. +- **A candidate clears** → `Recommendation: switch to `. Give its score in the bucket and its cost per task, what they are on now, and what changes. If a stronger model sits above it, say what the next step up costs per extra point, so they can overrule you. +- **They are already on it** → `Recommendation: stay on `. Still show what it was ranked against. +- **Nothing clears** → say so on the first line. Name the closest model and how far short it falls, and name any model skipped for failing tasks in the bucket. Let them decide. -Never invent a switch. A router that always recommends a change is a router that recommends noise. +Never invent a switch. A router that always recommends a change recommends noise. ## Then stop and hand back The recommendation ends the turn. Print it and stop: no tool call after it, no "meanwhile, I have started on...", no first file written while the developer is still reading which model to use. -A recommendation they cannot act on is not a recommendation. Switching model costs them one command, but only before the work begins — carry straight on and the context is built under the old model, so the advice is dead the moment it is printed. It also reads as though you asked the question and then overruled the answer. - -So the basis line is the last thing in the turn. The developer switches or does not, and tells you to go. Then you work. - -This holds when the answer is to stay, too. `stay on ` still ends the turn: they may disagree with the floor, the tier, or the model, and they get the chance to say so before anything is written. +Switching model costs the developer one command, but only before the work begins. Carry on under the old model and the advice is dead the moment it is printed. This holds when the answer is to stay, too: they may disagree with an answer, and they get the chance to say so before anything is written. ## Expect to recommend downward -Most developers run the top model for everything. That is what this skill is for. +Most developers run the top model for everything. That is what this skill is for. On most tasks a cheaper model's score in the bucket clears the required quality, so recommending down is the common case. -On the benchmark, if someone is on `claude-opus-5`, then `claude-sonnet-5` lands within 5 points on **10 of 21 tasks** at **61% lower cost**, and matches or beats it outright on 5. Recommending down is the common case, not the exception. +Two cautions go with it: -Three cautions that go with it: - -- **A downgrade inside the noise is a free saving, not a risk.** If the cheaper model is within 3 points at the tier, the measurements do not distinguish them, so the cheaper one is the right call and the saving is real. What needs care is the opposite: do not describe a 2-point difference as a quality loss the developer is accepting, because it is not measurable. -- **Cheaper is not a free win at low floors.** The weakest models are far behind on every kind of task, easy ones included. On the benchmark the cheapest model scores 42.58 against the best at 82.83 — that is a different outcome, not the same result for less. -- **Downgrades get riskier as the task gets harder.** How much a cheaper model costs you depends on the pair *and* the tier. `claude-sonnet-5` trails `claude-opus-5` by 5.7 points on easy work and 6.1 on hard, so that swap holds up everywhere. `qwen3.8-27b` beats `claude-sonnet-5` by 0.4 on easy work and trails it by 2.2 on hard, while also failing one hard task in five. Check the tier column and the completion count before recommending down on difficult work. +- **A downgrade inside the noise is a free saving.** If the cheaper model is within 3 points, the measurements do not tell them apart. Do not describe that gap as a quality loss. +- **Downgrades get riskier as the bucket gets harder.** Models lose points at different rates as tasks get harder, and some start failing tasks outright. `route.py` already reads the bucket and skips models with failures there; report what it skipped. ## Say what the recommendation rests on -Every recommendation carries its basis. Not a footnote, a line the developer reads: - -> Measured on one repository — a Python/FastAPI and React service with nginx, Terraform and bash around it — over 21 design-and-implement tasks, scored by an LLM judge. Rankings travel better than absolute scores. If your codebase is very different, treat this as a starting point. - -State the measurement date from `provenance.measured_on` too. These numbers move: a fix to token accounting once changed `claude-opus-5` from $7.63 to $11.95 per task while every score stayed the same. A stale price with a visible date is honest; a stale price presented as current is not. - -**Say that the numbers are not about this task.** Every score and every dollar figure in the block is a mean over the benchmark's 21 tasks — past work, on somebody else's repository, under a different harness. None of it is a forecast for the change in front of the developer. The cost especially: what this task costs depends on how long it runs, how much of the repo it reads and how many turns it takes, and that is not knowable before it starts. So put it in the basis, plainly: - -> Scores and costs are that benchmark's per-task averages, not an estimate for this task. What this one costs depends on how long it runs. +Every recommendation carries its basis, as a line the developer reads. Build it from `models.json`: the dataset in `provenance.dataset`, the number of benchmark tasks, the harness in `provenance.harness`, the date in `provenance.measured_on`, and the repository description in `measurement_basis.single_repository_warning`. -A developer who reads `$11.95 per task` and expects a bill of $11.95 for their change has been misled, and it is the number in the block most likely to be taken literally. The comparison between two models is the point. The absolute figure is scaffolding under the comparison. +Say that the numbers are not about this task. Every score and every dollar figure is a mean over benchmark tasks on somebody else's repository. None of it forecasts the change in front of the developer. The cost especially: what this task costs depends on how long it runs. A developer who reads `$4.44 per task` and expects that bill for their change has been misled. ## The format of the answer -Use this layout. Not something in its spirit — this one, with these labels, in this order: +Use this layout, with these labels, in this order: ``` Recommendation: Task - Complexity - Consequence - Floor + Difficulty (rubric sum of 8) + Required (blast radius <0-2>, verification <0-2>) - You are on on tasks / $ per task - Recommended on tasks / $ per task () - Saving % per task, points of headroom above the floor + You are on on tasks / $ per task + Recommended on tasks / $ per task () + Saving % per task, points above the required quality Next step up for $ (+ points over the pick) -Basis: 21 design-and-implement tasks on one Python/React service repo, -harness, judged by an LLM. Measured . Scores and costs are that benchmark's -per-task averages, not an estimate for this task. Rankings travel better than -absolute scores. +Basis: benchmark tasks on , harness, +judged by an LLM. Measured . Scores and costs are that benchmark's +per-task averages for tasks, not an estimate for this task. ``` -Six rules hold it together: +The rules that hold it together: - **The first line names the model**, prefixed `Recommendation:`. Nothing above it. - **Two indented spaces, labels in one column, model names in the next, numbers after.** It is read by eye, in a terminal, in about four seconds. -- **Every score carries its tier** — `86.2 on low tasks`, never a bare `86.2`. The reader has to know which table it came from. -- **Show the completion count** beside a recommended model, and always when it is short of the full count: `(4/5 finished)` is a one-in-five outright failure and belongs in front of them. -- **Drop a row with nothing to say.** No `Saving` when the answer is to stay, no `Next step up` when they are already on the strongest permitted model. Pluralise to `Next steps up` when you list more than one. Do not add rows of prose. -- **The basis is the last thing in the turn.** Nothing after it — see *Then stop and hand back*. +- **Every score carries its bucket**: `80.1 on hard tasks`, never a bare `80.1`. +- **Show the completion count** beside the recommended model. +- **Drop a row with nothing to say.** No `Saving` when the answer is to stay, no `Next step up` when they are on the strongest permitted model. Add a `Skipped` row when `skipped_for_failures` names a model that would otherwise have been picked. +- **The basis is the last thing in the turn.** -A worked one, where the answer is to switch: +A worked example, with made-up model names and numbers: ``` -Recommendation: switch to claude-sonnet-5 - - Task add an env passthrough to two compose files - Complexity trivial - Consequence internal tooling, a human reviews it before it matters - Floor 65 - - You are on claude-opus-5 83.8 on trivial tasks / $11.95 per task - Recommended claude-sonnet-5 76.1 on trivial tasks / $4.67 per task (5/5 finished) - Saving 61% per task, 11.1 points of headroom above the floor - - Next step up claude-opus-5 83.8 for $11.95 (+7.7 points over the pick, - outside the +/-3 noise band, so it is a real gap) - -Basis: 21 design-and-implement tasks on one Python/React service repo, omp -harness, judged by an LLM. Measured 2026-09-08. Scores and costs are that -benchmark's per-task averages, not an estimate for this task. Rankings travel -better than absolute scores. -``` - -And when the answer is to stay. Same layout, same labels — it still names the -model on the first line and still shows what it was ranked against: - -``` -Recommendation: stay on qwen3.8-27b +Recommendation: switch to mid-model Task add per-caller rate limiting to the proxy - Complexity high - Consequence a production service, a defect reaches someone - Floor 70 + Difficulty very_hard (rubric sum 6 of 8) + Required 70 (blast radius 1, verification 1) - You are on qwen3.8-27b 71.5 on high tasks / $1.47 per task (4/5 finished) - cheapest permitted model above the floor, and it sits - 1.5 above it -- inside the noise band, not clear of it; - it also failed one of the five hard tasks + You are on large-model 80.2 on very_hard tasks / $4.44 per task + Recommended mid-model 77.2 on very_hard tasks / $2.62 per task (9/9 finished) + Saving 41% per task, 7.2 points above the required quality + Skipped small-model failed 1 of 9 very_hard tasks - Next step up glm-5.3 74.7 on high tasks / $7.04 per task - 3.2 points for $5.57 more, just outside +/-3 + Next step up large-model 80.2 for $4.44 (+3.0 points over the pick, + inside the +/-3 noise band, so no measurable gain) - Below the floor qwen3.6-35b 56.0 - gemma-4-31b 50.0 - -Basis: 21 design-and-implement tasks on one Python/React service repo, omp -harness, judged by an LLM. Measured 2026-09-08. Scores and costs are that -benchmark's per-task averages, not an estimate for this task. Rankings travel -better than absolute scores. +Basis: 21 benchmark tasks on one Python/FastAPI and React service, omp +harness, judged by an LLM. Measured 2026-10-11. Scores and costs are that +benchmark's per-task averages for very_hard tasks, not an estimate for this task. ``` -Short. The numbers they need to disagree with you, and nothing else. - ## Stay inside the lines -- Recommend a model. Do not switch it, do not run the task, do not write the code. -- End the turn on the recommendation. Do not start the task in the same turn, whether the answer was switch or stay. -- Do not quote a number that is not in `models.json`. -- Do not present a benchmark average as this task's cost or score. It is what the model averaged on 21 other tasks, not a forecast for this one. -- Do not compare a floor against an overall mean when `score_by_complexity` has the tier. -- Do not filter or rank on `on_combined_frontier` or `on_hosting_frontier`. Report them; select on the tier score. -- Do not present a gap under 3 points at a tier as a difference. It is noise. +- Recommend a model. Do not switch it, run the task, or write the code. +- End the turn on the recommendation, whether the answer was switch or stay. +- Do not compute the bucket or the required quality yourself; `route.py` does. +- Do not quote a number that is not in `models.json` or the `route.py` output. +- Do not present a benchmark average as this task's cost or score. +- Do not present a gap under 3 points as a difference. - Do not recommend a cheaper model for a task beyond the measured range. -- Do not score a model that was not measured. +- Do not filter or rank on `on_combined_frontier` or `on_hosting_frontier`. - Do not tell anyone to buy or provision hardware. -- If `models.json` has a `schema_version` whose major differs from `1`, stop and say the data is newer than these instructions. +- If `models.json` has a `schema_version` whose major is not `2`, stop and say the data and these instructions do not match. diff --git a/vend/swe-router/allowed-models.txt b/vend/swe-router/allowed-models.txt index 54297b06..534648fc 100644 --- a/vend/swe-router/allowed-models.txt +++ b/vend/swe-router/allowed-models.txt @@ -27,35 +27,41 @@ # --------------------------------------------------------------------------- # What ships here, and why you should change it # --------------------------------------------------------------------------- -# These six are the models on the measured cost/quality frontier: nothing +# These five are the models on the measured cost/quality frontier: nothing # else scores at least as well for at most the cost, across the whole # 21-task benchmark. That is a defensible starting point and a poor # finished one, for two reasons. # -# Five of the six are self-hosted, and the shipped file cannot know whether -# you serve them. If you do not, delete those four lines -- left in, they +# One of the five, qwen3.8-27b, is self-hosted, and the shipped file cannot +# know whether you serve it. If you do not, delete that line -- left in, it # will be recommended, and a developer on a hosted API will be told to -# switch to a model they cannot select. With only claude-opus-5 left they -# would be sent there for everything, including a docs page, so add the -# hosted models they do have. +# switch to a model they cannot select. The other four run on Amazon +# Bedrock. # # Frontier membership also comes from whole-dataset means, which is not the -# ranking that decides a task. claude-sonnet-5 is missing because -# qwen3.8-27b beats it on average -- 78.48 to 76.97, at a third of the -# cost. On high-complexity work that reverses: sonnet scores 73.68 having -# finished all five hard tasks, qwen3.8-27b scores 71.45 having failed one. +# ranking that decides a task. kimi-k3 is missing because claude-sonnet-5-5 +# beats it on average -- 79.49 to 79.17, at three fifths of the cost -- yet +# kimi-k3 scores higher on trivial work (81.80 to 77.36). And the two +# cheapest entries fall away on hard work: qwen3.8-27b drops to 71.45 and +# claude-haiku-5-5 to 60.00 on high-complexity tasks, and each failed one +# of the five. # -# So add the models your developers can actually reach. Uncomment these two -# if they are on a hosted API: +# So add the models your developers can actually reach. If you self-host, +# these were on the frontier before the Claude 5.5 runs and remain the best +# self-hosted value: # -# claude-sonnet-5 # 76.97, $4.67/task. Beats qwen3.8-27b on hard work. -# claude-haiku-4-5 # 56.18, $0.76/task. Cheapest hosted option. +# kimi-k3 # 79.17, $4.33/task, self-hosted. +# gemma-4-31b # 59.74, $0.87/task, self-hosted. +# qwen3.6-35b # 59.24, $0.26/task, self-hosted. Cheapest self-hosted. claude-opus-5 # 82.83, $11.95/task, Bedrock. Highest scoring measured. -glm-5.3 # 80.11, $7.04/task, self-hosted. Within 2.7 of the top. -kimi-k3 # 79.17, $4.33/task, self-hosted. New frontier point. -qwen3.8-27b # 78.48, $1.47/task, self-hosted. Best value here, but - # drops to 71.45 on high-complexity work and did not +claude-opus-5-5 # 81.16, $4.44/task, Bedrock. Within 1.7 of the top at + # under two fifths of the price. +claude-sonnet-5-5 # 79.49, $2.62/task, Bedrock. Highest on medium work + # (82.90) and finished all five hard tasks. +qwen3.8-27b # 78.48, $1.47/task, self-hosted. Drops to 71.45 on + # high-complexity work and did not finish one of the + # five hard tasks. +claude-haiku-5-5 # 72.69, $0.14/task, Bedrock. Cheapest measured by far, + # but 60.00 on high-complexity work, where it did not # finish one of the five hard tasks. -gemma-4-31b # 59.74, $0.87/task, self-hosted. -qwen3.6-35b # 59.24, $0.26/task, self-hosted. Cheapest measured. diff --git a/vend/swe-router/install.sh b/vend/swe-router/install.sh index 78a72ab2..a227cc91 100755 --- a/vend/swe-router/install.sh +++ b/vend/swe-router/install.sh @@ -24,11 +24,12 @@ SKILLS_DIR="${SWE_ROUTER_DIR:-.claude/skills}" FORCE="${SWE_ROUTER_FORCE:-0}" SKILL_NAME="swe-router" -SKILL_FILES="SKILL.md route.py models.json model-aliases.json allowed-models.txt" -JSON_FILES="models.json model-aliases.json" -# The one file a user is expected to edit. An upgrade preserves it, because -# overwriting it would silently replace an organisation's model policy. -POLICY_FILE="allowed-models.txt" +SKILL_FILES="SKILL.md rubric.md route.py models.json model-aliases.json routing-policy.json allowed-models.txt" +JSON_FILES="models.json model-aliases.json routing-policy.json" +# The files a user is expected to edit. An upgrade preserves them, because +# overwriting either would silently replace an organisation's policy: which +# models it allows, and what quality each kind of task must reach. +POLICY_FILES="allowed-models.txt routing-policy.json" usage() { @@ -42,9 +43,9 @@ Options: to hold a version steady. --repo OWNER/NAME Source repository (default: aws-samples/sample-agentic-coding-harness-benchmarks). - --force Overwrite an existing allowed-models.txt. Without this, an - existing model policy is kept and the rest of the skill is - still upgraded. + --force Overwrite an existing allowed-models.txt and routing-policy.json. + Without this, existing policy files are kept and the rest of + the skill is still upgraded. -h, --help Show this help. Environment variables: SWE_ROUTER_DIR, SWE_ROUTER_REF, SWE_ROUTER_REPO, @@ -82,6 +83,21 @@ if ! command -v curl >/dev/null 2>&1; then exit 1 fi +# The skill runs route.py, which needs only the Python standard library but +# does need python3, 3.8 or later, on PATH. Without it the skill cannot +# recommend anything, so refuse to install rather than leave a skill that +# fails the first time it fires. Nothing has been downloaded yet. +if ! command -v python3 >/dev/null 2>&1 \ + || ! python3 -c 'import sys; sys.exit(0 if sys.version_info >= (3, 8) else 1)' 2>/dev/null; then + echo "install.sh: swe-router needs python3 (3.8 or later) on PATH, and none was found." >&2 + echo "It needs no Python packages, only the interpreter. Install one, then re-run:" >&2 + echo " macOS: xcode-select --install" >&2 + echo " Debian/Ubuntu: sudo apt install python3" >&2 + echo " Windows: use WSL, or install from python.org" >&2 + echo "Nothing was written." >&2 + exit 1 +fi + BASE="https://raw.githubusercontent.com/${REPO}/${REF}/vend/${SKILL_NAME}" TARGET="${SKILLS_DIR}/${SKILL_NAME}" @@ -121,25 +137,35 @@ fi mkdir -p "$TARGET" -kept_policy=0 +kept="" for f in $SKILL_FILES; do - if [ "$f" = "$POLICY_FILE" ] && [ -f "${TARGET}/${f}" ] && [ "$FORCE" != "1" ]; then - kept_policy=1 - continue - fi + case " $POLICY_FILES " in + *" $f "*) + if [ -f "${TARGET}/${f}" ] && [ "$FORCE" != "1" ]; then + kept="${kept} ${f}" + continue + fi + ;; + esac mv "${STAGE}/${f}" "${TARGET}/${f}" done chmod +x "${TARGET}/route.py" + echo echo "Installed ${SKILL_NAME} to ${TARGET}" -if [ "$kept_policy" = "1" ]; then - echo "Kept your existing ${POLICY_FILE}. Re-run with --force to replace it." -else - echo - echo "Next step, and do not skip it: edit ${TARGET}/${POLICY_FILE}" - echo "It ships allowing five models, four of them self-hosted. Left as-is in a" - echo "setup that cannot reach those, the skill will recommend a model nobody can" - echo "select. Cut the list down to what your setup actually serves." +if [ -n "$kept" ]; then + echo "Kept your existing${kept}. Re-run with --force to replace them." fi +case " $kept " in + *" allowed-models.txt "*) ;; + *) + echo + echo "Next step, and do not skip it: edit ${TARGET}/allowed-models.txt" + echo "It ships allowing the five models on the measured frontier, one of them" + echo "self-hosted. Left as-is in a setup that cannot reach that one, the skill" + echo "will recommend a model nobody can select. Cut the list down to what your" + echo "setup actually serves." + ;; +esac diff --git a/vend/swe-router/model-aliases.json b/vend/swe-router/model-aliases.json index f43942e3..e981250d 100644 --- a/vend/swe-router/model-aliases.json +++ b/vend/swe-router/model-aliases.json @@ -11,6 +11,15 @@ ], "unmatched_model_policy": "A model the user has that is absent from models.json has NOT been measured. Say so and leave it out of the ranking. Never estimate a score for it.", "aliases": { + "claude-opus-5-5": [ + "claude-opus-5-5", + "us.anthropic.claude-opus-5-5", + "global.anthropic.claude-opus-5-5", + "anthropic/claude-opus-5-5", + "Claude Opus 5.5", + "opus-5.5", + "opus-5-5" + ], "claude-opus-5": [ "claude-opus-5", "us.anthropic.claude-opus-5", @@ -52,6 +61,24 @@ "opus-4.5", "opus-4-5" ], + "claude-sonnet-5-5": [ + "claude-sonnet-5-5", + "us.anthropic.claude-sonnet-5-5", + "global.anthropic.claude-sonnet-5-5", + "anthropic/claude-sonnet-5-5", + "Claude Sonnet 5.5", + "sonnet-5.5", + "sonnet-5-5" + ], + "claude-haiku-5-5": [ + "claude-haiku-5-5", + "us.anthropic.claude-haiku-5-5", + "global.anthropic.claude-haiku-5-5", + "anthropic/claude-haiku-5-5", + "Claude Haiku 5.5", + "haiku-5.5", + "haiku-5-5" + ], "claude-sonnet-5": [ "claude-sonnet-5", "us.anthropic.claude-sonnet-5", @@ -90,6 +117,11 @@ "Kimi K2.7 Code", "kimi-k2.7" ], + "kimi-k3-bedrock": [ + "kimi-k3-bedrock", + "moonshotai.kimi-k3", + "Kimi K3 on Amazon Bedrock" + ], "kimi-k3": [ "kimi-k3", "moonshotai/Kimi-K3", diff --git a/vend/swe-router/models.json b/vend/swe-router/models.json index f7b2ed0b..f4e84b1a 100644 --- a/vend/swe-router/models.json +++ b/vend/swe-router/models.json @@ -1,10 +1,10 @@ { - "schema_version": "1.0", + "schema_version": "2.0", "generated_by": "benchmarks/scripts/build_vended_models.py", "provenance": { - "measured_on": "2026-09-10", + "measured_on": "2026-10-11", "source_file": "docs/metrics/pareto-frontier-omp-swe3.json", - "source_commit": "5f32859", + "source_sha256": "72b1ed9ee222", "harness": "omp", "skill": "swe3", "dataset": "mcp-gateway-registry-v2", @@ -13,14 +13,22 @@ "reasoning_effort": "high", "repo_grounded": true }, + "rubric_scores": { + "file": "docs/metrics/swe-router-rubric-mcp-gateway-registry-v2-claude-opus-5-5.json", + "sha256": "72ef0449b746", + "judge": "us.anthropic.claude-opus-5-5", + "repeats": 3, + "buckets_recomputed_with": "vend/swe-router/routing-policy.json" + }, "repository": "https://github.com/aws-samples/sample-agentic-coding-harness-benchmarks" }, "measurement_basis": { "what_the_score_is": "Mean 0-100 score over the dataset's tasks. Each task is judged on six artifacts: a GitHub issue spec, a low-level design, an expert review, a testing plan, a patch, and an implementation summary.", "single_repository_warning": "Every task comes from one repository -- a Python/FastAPI and React service with nginx, Terraform, Helm and bash around it. Rankings are more portable than absolute scores, but applying these to a very different codebase is extrapolation.", "runs_per_task": 1, - "frontier_flags": "on_combined_frontier and on_hosting_frontier mark models that nothing else beats on both score and cost across the WHOLE dataset. They are context for a reader, not a selection key: they come from overall means, so they can disagree with the per-tier ranking. Select on score_by_complexity at the task's tier. on_hosting_frontier is computed within one hosting basis (Bedrock or self-hosted), which is the apples-to-apples comparison; on_combined_frontier mixes the two and is directional.", - "complexity_tiers": "trivial / low / medium / high, assigned by the scope of the change. The hardest tasks measured are bounded single-repo changes -- a rate-limiting subsystem, server-side OAuth token storage. Nothing here approaches a rewrite or a language port, so a task far beyond that range is outside what these numbers cover.", + "frontier_flags": "on_combined_frontier and on_hosting_frontier mark models that nothing else beats on both score and cost across the WHOLE dataset. They are context for a reader, not a selection key: they come from overall means, so they can disagree with the per-bucket ranking. Select on score_by_difficulty at the task's bucket. on_hosting_frontier is computed within one hosting basis (Bedrock or self-hosted), which is the apples-to-apples comparison; on_combined_frontier mixes the two and is directional.", + "difficulty_buckets": "easy / moderate / hard / very_hard: each benchmark task's difficulty level, from four rubric scores (hidden contract, silent invariant, coordinated surfaces, ambiguity) summed and bucketed by routing-policy.json. score_by_difficulty is a model's mean over the tasks in a bucket, failures excluded; completion_by_difficulty is where failures show. A bucket with no benchmark task has no entry.", + "complexity_tiers": "trivial / low / medium / high, set by hand in the dataset from the scope of the change; score_by_complexity serves the legacy --tier path only. The hardest tasks measured are bounded single-repo changes -- a rate-limiting subsystem, server-side OAuth token storage. Nothing here approaches a rewrite or a language port, so a task far beyond that range is outside what these numbers cover.", "cost_basis": { "Bedrock": "Metered Amazon Bedrock token pricing -- what the invoice says.", "self-hosted": "The server's hourly price divided by throughput measured at a stated concurrency -- a shared server under load, which is how a platform team runs one for a group of developers. A cost per task on the same footing as a metered bill." @@ -37,6 +45,16 @@ "excluded_tasks": [], "on_combined_frontier": true, "on_hosting_frontier": true, + "score_by_difficulty": { + "moderate": 87.4, + "hard": 84.26, + "very_hard": 80.22 + }, + "completion_by_difficulty": { + "moderate": "2/2", + "hard": "10/10", + "very_hard": "9/9" + }, "score_by_complexity": { "trivial": 83.8, "low": 86.2, @@ -50,6 +68,39 @@ "high": "5/5" } }, + { + "model": "claude-opus-5-5", + "score": 81.16, + "cost_per_task_usd": 4.44, + "hosting": "Bedrock", + "tasks_completed": 21, + "tasks_total": 21, + "excluded_tasks": [], + "on_combined_frontier": true, + "on_hosting_frontier": true, + "score_by_difficulty": { + "moderate": 89.7, + "hard": 80.36, + "very_hard": 80.16 + }, + "completion_by_difficulty": { + "moderate": "2/2", + "hard": "10/10", + "very_hard": "9/9" + }, + "score_by_complexity": { + "trivial": 81.52, + "low": 83.16, + "medium": 79.97, + "high": 80.24 + }, + "completion_by_complexity": { + "trivial": "5/5", + "low": "5/5", + "medium": "6/6", + "high": "5/5" + } + }, { "model": "glm-5.3", "score": 80.11, @@ -58,8 +109,18 @@ "tasks_completed": 21, "tasks_total": 21, "excluded_tasks": [], - "on_combined_frontier": true, + "on_combined_frontier": false, "on_hosting_frontier": true, + "score_by_difficulty": { + "moderate": 86.8, + "hard": 80.98, + "very_hard": 77.67 + }, + "completion_by_difficulty": { + "moderate": "2/2", + "hard": "10/10", + "very_hard": "9/9" + }, "score_by_complexity": { "trivial": 84.96, "low": 82.64, @@ -73,6 +134,39 @@ "high": "5/5" } }, + { + "model": "claude-sonnet-5-5", + "score": 79.49, + "cost_per_task_usd": 2.62, + "hosting": "Bedrock", + "tasks_completed": 21, + "tasks_total": 21, + "excluded_tasks": [], + "on_combined_frontier": true, + "on_hosting_frontier": true, + "score_by_difficulty": { + "moderate": 86.5, + "hard": 80.12, + "very_hard": 77.22 + }, + "completion_by_difficulty": { + "moderate": "2/2", + "hard": "10/10", + "very_hard": "9/9" + }, + "score_by_complexity": { + "trivial": 77.36, + "low": 83.76, + "medium": 82.9, + "high": 73.24 + }, + "completion_by_complexity": { + "trivial": "5/5", + "low": "5/5", + "medium": "6/6", + "high": "5/5" + } + }, { "model": "kimi-k3", "score": 79.17, @@ -81,8 +175,18 @@ "tasks_completed": 21, "tasks_total": 21, "excluded_tasks": [], - "on_combined_frontier": true, + "on_combined_frontier": false, "on_hosting_frontier": true, + "score_by_difficulty": { + "moderate": 84.8, + "hard": 81.02, + "very_hard": 75.87 + }, + "completion_by_difficulty": { + "moderate": "2/2", + "hard": "10/10", + "very_hard": "9/9" + }, "score_by_complexity": { "trivial": 81.8, "low": 84.24, @@ -108,6 +212,16 @@ ], "on_combined_frontier": true, "on_hosting_frontier": true, + "score_by_difficulty": { + "moderate": 83.6, + "hard": 81.34, + "very_hard": 73.62 + }, + "completion_by_difficulty": { + "moderate": "2/2", + "hard": "10/10", + "very_hard": "8/9" + }, "score_by_complexity": { "trivial": 81.4, "low": 80.92, @@ -130,7 +244,17 @@ "tasks_total": 21, "excluded_tasks": [], "on_combined_frontier": false, - "on_hosting_frontier": true, + "on_hosting_frontier": false, + "score_by_difficulty": { + "moderate": 81.8, + "hard": 78.1, + "very_hard": 74.64 + }, + "completion_by_difficulty": { + "moderate": "2/2", + "hard": "10/10", + "very_hard": "9/9" + }, "score_by_complexity": { "trivial": 76.12, "low": 80.48, @@ -154,6 +278,16 @@ "excluded_tasks": [], "on_combined_frontier": false, "on_hosting_frontier": false, + "score_by_difficulty": { + "moderate": 81.0, + "hard": 76.5, + "very_hard": 73.4 + }, + "completion_by_difficulty": { + "moderate": "2/2", + "hard": "10/10", + "very_hard": "9/9" + }, "score_by_complexity": { "trivial": 77.76, "low": 77.0, @@ -177,6 +311,16 @@ "excluded_tasks": [], "on_combined_frontier": false, "on_hosting_frontier": false, + "score_by_difficulty": { + "moderate": 81.1, + "hard": 74.9, + "very_hard": 73.02 + }, + "completion_by_difficulty": { + "moderate": "2/2", + "hard": "10/10", + "very_hard": "9/9" + }, "score_by_complexity": { "trivial": 73.84, "low": 81.08, @@ -190,6 +334,74 @@ "high": "5/5" } }, + { + "model": "kimi-k3-bedrock", + "score": 72.87, + "cost_per_task_usd": 3.87, + "hosting": "Bedrock", + "tasks_completed": 21, + "tasks_total": 21, + "excluded_tasks": [], + "on_combined_frontier": false, + "on_hosting_frontier": false, + "score_by_difficulty": { + "moderate": 83.6, + "hard": 73.08, + "very_hard": 70.24 + }, + "completion_by_difficulty": { + "moderate": "2/2", + "hard": "10/10", + "very_hard": "9/9" + }, + "score_by_complexity": { + "trivial": 73.6, + "low": 80.08, + "medium": 66.03, + "high": 73.12 + }, + "completion_by_complexity": { + "trivial": "5/5", + "low": "5/5", + "medium": "6/6", + "high": "5/5" + } + }, + { + "model": "claude-haiku-5-5", + "score": 72.69, + "cost_per_task_usd": 0.14, + "hosting": "Bedrock", + "tasks_completed": 20, + "tasks_total": 21, + "excluded_tasks": [ + "registration-admission-control-gate" + ], + "on_combined_frontier": true, + "on_hosting_frontier": true, + "score_by_difficulty": { + "moderate": 83.2, + "hard": 76.52, + "very_hard": 65.28 + }, + "completion_by_difficulty": { + "moderate": "2/2", + "hard": "10/10", + "very_hard": "8/9" + }, + "score_by_complexity": { + "trivial": 74.92, + "low": 79.16, + "medium": 73.9, + "high": 60.0 + }, + "completion_by_complexity": { + "trivial": "5/5", + "low": "5/5", + "medium": "6/6", + "high": "4/5" + } + }, { "model": "claude-opus-4-6-v1", "score": 70.64, @@ -200,6 +412,16 @@ "excluded_tasks": [], "on_combined_frontier": false, "on_hosting_frontier": false, + "score_by_difficulty": { + "moderate": 78.8, + "hard": 73.84, + "very_hard": 65.27 + }, + "completion_by_difficulty": { + "moderate": "2/2", + "hard": "10/10", + "very_hard": "9/9" + }, "score_by_complexity": { "trivial": 74.2, "low": 75.4, @@ -226,6 +448,16 @@ ], "on_combined_frontier": false, "on_hosting_frontier": false, + "score_by_difficulty": { + "moderate": 79.9, + "hard": 71.9, + "very_hard": 62.09 + }, + "completion_by_difficulty": { + "moderate": "2/2", + "hard": "10/10", + "very_hard": "7/9" + }, "score_by_complexity": { "trivial": 72.36, "low": 71.68, @@ -248,7 +480,17 @@ "tasks_total": 21, "excluded_tasks": [], "on_combined_frontier": false, - "on_hosting_frontier": true, + "on_hosting_frontier": false, + "score_by_difficulty": { + "moderate": 68.0, + "hard": 70.86, + "very_hard": 60.91 + }, + "completion_by_difficulty": { + "moderate": "2/2", + "hard": "10/10", + "very_hard": "9/9" + }, "score_by_complexity": { "trivial": 68.8, "low": 66.24, @@ -272,6 +514,16 @@ "excluded_tasks": [], "on_combined_frontier": false, "on_hosting_frontier": false, + "score_by_difficulty": { + "moderate": 66.7, + "hard": 64.94, + "very_hard": 56.89 + }, + "completion_by_difficulty": { + "moderate": "2/2", + "hard": "10/10", + "very_hard": "9/9" + }, "score_by_complexity": { "trivial": 64.44, "low": 63.92, @@ -295,6 +547,16 @@ "excluded_tasks": [], "on_combined_frontier": false, "on_hosting_frontier": false, + "score_by_difficulty": { + "moderate": 74.3, + "hard": 60.9, + "very_hard": 57.02 + }, + "completion_by_difficulty": { + "moderate": "2/2", + "hard": "10/10", + "very_hard": "9/9" + }, "score_by_complexity": { "trivial": 65.72, "low": 60.64, @@ -316,8 +578,18 @@ "tasks_completed": 21, "tasks_total": 21, "excluded_tasks": [], - "on_combined_frontier": true, + "on_combined_frontier": false, "on_hosting_frontier": true, + "score_by_difficulty": { + "moderate": 68.6, + "hard": 63.0, + "very_hard": 54.16 + }, + "completion_by_difficulty": { + "moderate": "2/2", + "hard": "10/10", + "very_hard": "9/9" + }, "score_by_complexity": { "trivial": 66.52, "low": 63.28, @@ -339,8 +611,18 @@ "tasks_completed": 21, "tasks_total": 21, "excluded_tasks": [], - "on_combined_frontier": true, + "on_combined_frontier": false, "on_hosting_frontier": true, + "score_by_difficulty": { + "moderate": 62.6, + "hard": 62.28, + "very_hard": 55.11 + }, + "completion_by_difficulty": { + "moderate": "2/2", + "hard": "10/10", + "very_hard": "9/9" + }, "score_by_complexity": { "trivial": 58.44, "low": 62.48, @@ -363,7 +645,17 @@ "tasks_total": 21, "excluded_tasks": [], "on_combined_frontier": false, - "on_hosting_frontier": true, + "on_hosting_frontier": false, + "score_by_difficulty": { + "moderate": 62.7, + "hard": 57.04, + "very_hard": 53.78 + }, + "completion_by_difficulty": { + "moderate": "2/2", + "hard": "10/10", + "very_hard": "9/9" + }, "score_by_complexity": { "trivial": 55.4, "low": 57.8, @@ -387,6 +679,16 @@ "excluded_tasks": [], "on_combined_frontier": false, "on_hosting_frontier": false, + "score_by_difficulty": { + "moderate": 58.9, + "hard": 58.24, + "very_hard": 52.73 + }, + "completion_by_difficulty": { + "moderate": "2/2", + "hard": "10/10", + "very_hard": "9/9" + }, "score_by_complexity": { "trivial": 54.72, "low": 60.2, @@ -416,6 +718,16 @@ ], "on_combined_frontier": false, "on_hosting_frontier": false, + "score_by_difficulty": { + "moderate": 65.7, + "hard": 55.2, + "very_hard": 44.48 + }, + "completion_by_difficulty": { + "moderate": "2/2", + "hard": "6/10", + "very_hard": "8/9" + }, "score_by_complexity": { "trivial": 56.5, "low": 61.33, @@ -442,6 +754,16 @@ ], "on_combined_frontier": false, "on_hosting_frontier": false, + "score_by_difficulty": { + "moderate": 42.6, + "hard": 47.2, + "very_hard": 37.4 + }, + "completion_by_difficulty": { + "moderate": "2/2", + "hard": "9/10", + "very_hard": "8/9" + }, "score_by_complexity": { "trivial": 43.76, "low": 50.96, @@ -465,6 +787,16 @@ "excluded_tasks": [], "on_combined_frontier": false, "on_hosting_frontier": false, + "score_by_difficulty": { + "moderate": 52.4, + "hard": 45.44, + "very_hard": 37.22 + }, + "completion_by_difficulty": { + "moderate": "2/2", + "hard": "10/10", + "very_hard": "9/9" + }, "score_by_complexity": { "trivial": 45.6, "low": 48.4, diff --git a/vend/swe-router/route.py b/vend/swe-router/route.py index 7c14faa0..4aea2666 100755 --- a/vend/swe-router/route.py +++ b/vend/swe-router/route.py @@ -1,31 +1,42 @@ #!/usr/bin/env python3 -"""Pick a model, given a quality floor and a task difficulty tier. +"""Pick a model for a coding task from six rubric scores. -The skill decides two things by reading the repo: how good the output has to be -(the floor) and how hard the task is (the tier). This does the rest, so the -selection is arithmetic rather than something a language model works out in -prose and might get wrong. +The skill reads the repository and the task and scores the six questions in +``rubric.md``, 0 to 2 each. This does the rest, so the selection is arithmetic +rather than something a language model works out in prose and might get wrong: - python3 route.py --floor 70 --tier high + python3 route.py --rubric '{"hidden_contract": 1, "silent_invariant": 0, + "coordinated_surfaces": 2, "ambiguity": 1, "verification": 1, + "blast_radius": 1}' --confidence high It prints JSON: the recommendation, the runners-up, and why anything was excluded. Standard library only, so it runs wherever the skill is installed. The rule, in full: -1. Start from every model in ``models.json``. -2. Drop anything the organisation does not allow, if ``allowed-models.txt`` is - present beside this file, at the repository root, or in ``.claude/``. That - file is also the availability list: whoever maintains it is saying every - entry is a model the developer's agent can select. -3. Drop anything ``--available`` leaves out, if it was given. It rarely is -- - it exists for the case where a developer says one listed model is not - wired up for them. -4. Read each survivor's score at the task's tier -- not its overall mean, which - is a ranking that holds at no tier. -5. Keep the ones at or above the floor and take the cheapest. Models within - ``--tie-band`` points of each other are treated as equal, because with five - or six tasks per tier a smaller gap does not survive dropping one task. +1. Turn the scores into two numbers with ``routing-policy.json``. Four + difficulty scores add up to a **bucket** (the task's difficulty level: + easy, moderate, hard, very hard). ``blast_radius`` and ``verification`` set + the **required quality**, the score a model must reach. +2. If the scorer reported low confidence, stop and keep the developer on the + model they have. +3. Start from every model in ``models.json``. Drop anything the organisation + does not allow, if ``allowed-models.txt`` is present beside this file, at + the repository root, or in ``.claude/``. That file is also the availability + list: whoever maintains it is saying every entry is a model the developer's + agent can select. ``--available`` narrows it further, rarely. +4. Read each survivor's score in the task's bucket -- not its overall mean. + A model with no benchmark task in that bucket is read from the next harder + bucket it has, which can only understate it. +5. Skip a model that failed more of that bucket's benchmark tasks than the + policy's ``min_completion_in_bucket`` allows. A column mean leaves failures + out, so this is the only place they count. +6. Keep the ones at or above the required quality and take the cheapest. + Models within the tie band of each other are treated as equal, because with + a handful of tasks per bucket a smaller gap does not survive dropping one. + +``--tier`` and ``--floor`` still work, reading the old per-tier columns, for +one release. """ from __future__ import annotations @@ -37,6 +48,12 @@ from pathlib import Path from typing import Any +# Fail with a message the skill can read out, not a traceback, on a Python too +# old for this file. The installer checks the same floor. +if sys.version_info < (3, 8): # pragma: no cover - depends on the interpreter + print('{"status": "error", "error": "route.py needs Python 3.8 or later"}') + sys.exit(2) + _HERE = Path(__file__).resolve().parent # Five or six tasks per tier, one run each. Dropping a single task reverses the @@ -47,6 +64,21 @@ TIERS = ("trivial", "low", "medium", "high") +# The rubric. Kept in step with rubric.md, which the skill's model reads. +DIMENSIONS = ( + "hidden_contract", + "silent_invariant", + "coordinated_surfaces", + "ambiguity", + "verification", + "blast_radius", +) +SCORES = (0, 1, 2) +BUCKETS = ("easy", "moderate", "hard", "very_hard") +CONFIDENCE_LEVELS = ("high", "medium", "low") +DEFAULT_POLICY = _HERE / "routing-policy.json" +SCHEMA_MAJORS = ("1", "2") + # Where an organisation's allow-list might sit, nearest the developer first: a # team overriding the shipped default is the point of having one. ALLOW_LIST_NAMES = ("allowed-models.txt",) @@ -98,14 +130,125 @@ def load_models(path: Path) -> dict[str, Any]: except json.JSONDecodeError as exc: raise RouteError(f"{path} is not valid JSON: {exc}") from exc major = str(data.get("schema_version", "")).split(".", 1)[0] - if major != "1": + if major not in SCHEMA_MAJORS: raise RouteError( f"models.json is schema {data.get('schema_version')}; this router " - "understands major version 1. Update the router." + f"understands major versions {', '.join(SCHEMA_MAJORS)}. Update the router." ) return data +def load_policy(path: Path) -> dict[str, Any]: + """Load and check the routing policy. + + Args: + path: Path to routing-policy.json. + + Returns: + The parsed policy. + + Raises: + RouteError: If the file is missing, unparseable, or does not define a + quality for every blast radius and a range for every bucket. + """ + if not path.is_file(): + raise RouteError(f"no routing policy at {path}") + try: + policy = json.loads(path.read_text(encoding="utf-8")) + except json.JSONDecodeError as exc: + raise RouteError(f"{path} is not valid JSON: {exc}") from exc + try: + by_radius = policy["required_quality"]["by_blast_radius"] + dims = policy["difficulty"]["dimensions"] + buckets = policy["difficulty"]["buckets"] + except (KeyError, TypeError) as exc: + raise RouteError(f"{path} is missing {exc}") from exc + if sorted(by_radius) != [str(v) for v in SCORES]: + raise RouteError(f"{path}: by_blast_radius must define 0, 1 and 2") + if not set(dims) <= set(DIMENSIONS): + raise RouteError(f"{path}: unknown difficulty dimension in {dims}") + if list(buckets) != list(BUCKETS): + raise RouteError(f"{path}: buckets must be {', '.join(BUCKETS)}, in order") + # A platform team edits this file by hand, so check the shapes the + # arithmetic relies on and say which entry is wrong. + for name, edges in buckets.items(): + if not ( + isinstance(edges, list) + and len(edges) == 2 + and all(isinstance(e, int) and not isinstance(e, bool) for e in edges) + ): + raise RouteError(f"{path}: bucket {name} must be [low, high] integers") + numbers = [ + *by_radius.values(), + policy["required_quality"].get("add_when_verification_is_2", 0), + ] + if not all( + isinstance(n, (int, float)) and not isinstance(n, bool) for n in numbers + ): + raise RouteError(f"{path}: required quality values must be numbers") + return policy + + +def parse_rubric(text: str) -> dict[str, int]: + """Read the six rubric scores from JSON. + + Accepts ``{"hidden_contract": 1, ...}``, the same with each value as + ``{"score": 1, "evidence": "..."}``, or either wrapped in ``{"scores": ...}``. + + Args: + text: The JSON the skill passed. + + Returns: + Dimension -> score. + + Raises: + RouteError: Unless all six dimensions are present, each 0, 1 or 2. + """ + try: + data = json.loads(text) + except json.JSONDecodeError as exc: + raise RouteError(f"--rubric is not valid JSON: {exc}") from exc + if isinstance(data, dict) and isinstance(data.get("scores"), dict): + data = data["scores"] + if not isinstance(data, dict) or set(data) != set(DIMENSIONS): + raise RouteError(f"--rubric must score exactly: {', '.join(DIMENSIONS)}") + scores = {} + for dim in DIMENSIONS: + value = data[dim] + if isinstance(value, dict): + value = value.get("score") + if isinstance(value, bool) or value not in SCORES: + raise RouteError(f"--rubric {dim} must be 0, 1 or 2, got {value!r}") + scores[dim] = int(value) + return scores + + +def required_quality(scores: dict[str, int], policy: dict[str, Any]) -> float: + """The score a model must reach: blast radius, plus a margin when untested.""" + rq = policy["required_quality"] + base = float(rq["by_blast_radius"][str(scores["blast_radius"])]) + extra = float(rq.get("add_when_verification_is_2", 0)) + return base + (extra if scores["verification"] == 2 else 0.0) + + +def difficulty_sum(scores: dict[str, int], policy: dict[str, Any]) -> int: + """Sum of the policy's difficulty dimensions.""" + return sum(scores[d] for d in policy["difficulty"]["dimensions"]) + + +def bucket_for(scores: dict[str, int], policy: dict[str, Any]) -> str: + """The task's difficulty level, from the policy's bucket ranges. + + Raises: + RouteError: If the policy's ranges leave this sum uncovered. + """ + total = difficulty_sum(scores, policy) + for name, (lo, hi) in policy["difficulty"]["buckets"].items(): + if lo <= total <= hi: + return name + raise RouteError(f"difficulty sum {total} falls in no bucket of the policy") + + def build_alias_index(models: list[dict], aliases_path: Path) -> dict[str, str]: """Return a lookup from any spelling of a model to its canonical slug. @@ -187,39 +330,81 @@ def parse_allow_list(path: Path) -> list[str]: return names +def _completion_rate(completion: str | None) -> float | None: + """``"9/10"`` -> 0.9; None when the count is missing or malformed.""" + try: + done, total = (int(x) for x in (completion or "").split("/")) + except ValueError: + return None + return done / total if total else None + + +def _score_at( + m: dict, level: str, columns: str, fallback: tuple[str, ...] +) -> tuple[float | None, str | None]: + """A model's score at ``level``, else at the first ``fallback`` level it has. + + Returns: + The score and the level it came from, or (None, None). + """ + table = m.get(columns) or {} + for candidate in (level, *fallback): + if table.get(candidate) is not None: + return table[candidate], candidate + return None, None + + def select( models: list[dict], *, tier: str, floor: float, tie_band: float = DEFAULT_TIE_BAND, + columns: str = "score_by_complexity", + completions: str = "completion_by_complexity", + fallback: tuple[str, ...] = (), + min_completion: float = 0.0, ) -> dict[str, Any]: """Choose the cheapest candidate whose score at ``tier`` clears ``floor``. Args: models: Candidate model entries, already filtered. - tier: One of TIERS. - floor: Minimum acceptable score at that tier. + tier: The column to read: a difficulty bucket, or a legacy tier. + floor: Minimum acceptable score there (the required quality). tie_band: Score difference treated as no difference. + columns: The models.json field holding per-level scores. + completions: The models.json field holding per-level completion. + fallback: Levels to read, in order, when a model has no score at + ``tier``. The rubric path passes the harder buckets. + min_completion: Skip a model that finished less than this share of the + benchmark tasks at the level its score came from. Returns: - A dict with the pick, the models that cleared the floor, and those that - did not. + A dict with the pick, the models that cleared the floor, those that + did not, and those skipped for failing tasks. """ - scored, unscored = [], [] + scored, unscored, borrowed, incomplete = [], [], {}, [] for m in models: - score = (m.get("score_by_complexity") or {}).get(tier) + score, level = _score_at(m, tier, columns, fallback) if score is None: - # No measurement at this tier; fall back to the overall mean and + # No measurement at this level; fall back to the overall mean and # record that, rather than dropping the model without saying so. - score = m.get("score") + score, level = m.get("score"), None if score is None: continue unscored.append(m["model"]) - scored.append((score, m)) - - clears = [(s, m) for s, m in scored if s >= floor] - misses = sorted(((s, m) for s, m in scored if s < floor), key=lambda x: -x[0]) + elif level != tier: + borrowed[m["model"]] = level + rate = _completion_rate((m.get(completions) or {}).get(level or tier)) + if rate is not None and rate < min_completion: + incomplete.append((score, m, level)) + continue + scored.append((score, m, level)) + + clears = [(s, m, lv) for s, m, lv in scored if s >= floor] + misses = sorted( + ((s, m, lv) for s, m, lv in scored if s < floor), key=lambda x: -x[0] + ) pick = None tied = [] @@ -229,26 +414,24 @@ def select( # prefer the higher score among those, since the cost is the same or # better and the score difference is not measurable either way. band = [ - (s, m) - for s, m in clears - if m["cost_per_task_usd"] <= cheapest[1]["cost_per_task_usd"] - and abs(s - cheapest[0]) <= tie_band + e + for e in clears + if e[1]["cost_per_task_usd"] <= cheapest[1]["cost_per_task_usd"] + and abs(e[0] - cheapest[0]) <= tie_band ] pick = max(band, key=lambda x: x[0]) if band else cheapest - tied = [m["model"] for s, m in band if m["model"] != pick[1]["model"]] + tied = [e[1]["model"] for e in band if e[1]["model"] != pick[1]["model"]] - def row(entry: tuple[float, dict]) -> dict[str, Any]: - score, m = entry - completion = (m.get("completion_by_complexity") or {}).get(tier) - done, total = ( - (completion.split("/") + [None])[:2] if completion else (None, None) - ) + def row(entry: tuple[float, dict, str | None]) -> dict[str, Any]: + score, m, level = entry + completion = (m.get(completions) or {}).get(level or tier) + rate = _completion_rate(completion) return { "model": m["model"], "score": round(score, 2), "cost_per_task_usd": m["cost_per_task_usd"], "completion": completion, - "finished_every_task": completion is None or done == total, + "finished_every_task": rate is None or rate == 1.0, # How far above the floor, and whether that margin is bigger than # the noise. A model at 71 against a floor of 70 has not reliably # cleared it, and saying "clears the floor" flat would oversell it. @@ -263,30 +446,108 @@ def row(entry: tuple[float, dict]) -> dict[str, Any]: row(e) for e in sorted(clears, key=lambda x: x[1]["cost_per_task_usd"]) ], "below_floor": [row(e) for e in misses], + "skipped_for_failures": [row(e) for e in incomplete], + "scored_from_harder_bucket": borrowed, "scored_from_overall_mean": unscored, } +def _rubric_inputs( + rubric: dict[str, int], policy_path: Path +) -> tuple[dict[str, Any], dict[str, Any]]: + """Turn rubric scores into the bucket, required quality and policy knobs.""" + policy = load_policy(policy_path) + return policy, { + "rubric": rubric, + "difficulty_sum": difficulty_sum(rubric, policy), + "bucket": bucket_for(rubric, policy), + "required_quality": required_quality(rubric, policy), + } + + def route( *, - tier: str, - floor: float, + tier: str | None = None, + floor: float | None = None, available: list[str] | None, models_path: Path, aliases_path: Path, allowed_file: Path | None, no_allow_list: bool, - tie_band: float, + tie_band: float | None = None, + rubric: dict[str, int] | None = None, + confidence: str | None = None, + policy_path: Path = DEFAULT_POLICY, + min_completion: float | None = None, ) -> dict[str, Any]: """Run the whole selection and return a result the caller can print. + Give either ``rubric`` (the six scores; the policy sets the bucket and the + required quality) or the legacy ``tier`` and ``floor``. + + Args: + tier: Legacy: one of TIERS. + floor: Legacy: the quality floor. + available: Models to narrow the candidate set to, or None. + models_path: models.json. + aliases_path: model-aliases.json. + allowed_file: Explicit allow-list path, or None to search. + no_allow_list: Ignore any allow-list. + tie_band: Points below which two models tie; None takes the policy's + value on the rubric path and DEFAULT_TIE_BAND otherwise. + rubric: The six rubric scores. + confidence: The scorer's confidence; "low" stops the routing when the + policy says to stay on the current model when unsure. + policy_path: routing-policy.json. + min_completion: Override the policy's ``min_completion_in_bucket``. + Raises: RouteError: On unusable inputs. """ - if tier not in TIERS: - raise RouteError(f"--tier must be one of {', '.join(TIERS)}, got {tier!r}") + head: dict[str, Any] = {} + select_args: dict[str, Any] = {} + if rubric is not None: + if confidence is not None and confidence not in CONFIDENCE_LEVELS: + raise RouteError( + f"--confidence must be one of {', '.join(CONFIDENCE_LEVELS)}" + ) + policy, head = _rubric_inputs(rubric, policy_path) + tier, floor = head["bucket"], head["required_quality"] + if tie_band is None: + tie_band = float(policy.get("tie_band_points", DEFAULT_TIE_BAND)) + if min_completion is None: + min_completion = float(policy.get("min_completion_in_bucket", 1.0)) + select_args = { + "columns": "score_by_difficulty", + "completions": "completion_by_difficulty", + "fallback": BUCKETS[BUCKETS.index(tier) + 1 :], + } + head["confidence"] = confidence + if confidence == "low" and policy.get("when_unsure") == "stay_on_current_model": + return { + **head, + "status": "unsure", + "recommended": None, + "reason": "the rubric scores were marked low confidence, and the " + "policy keeps the developer on their current model when unsure", + } + else: + if tier not in TIERS: + raise RouteError(f"--tier must be one of {', '.join(TIERS)}, got {tier!r}") + if floor is None: + raise RouteError("--floor is required with --tier") + head = {"tier": tier} + tie_band = DEFAULT_TIE_BAND if tie_band is None else tie_band + select_args["min_completion"] = min_completion or 0.0 data = load_models(models_path) + if rubric is not None and not any( + m.get("score_by_difficulty") for m in data["models"] + ): + raise RouteError( + f"{models_path} has no score_by_difficulty columns; the rubric needs " + "schema 2 or later. Update the skill." + ) all_models = data["models"] index = build_alias_index(all_models, aliases_path) @@ -320,7 +581,7 @@ def resolve(names: list[str]) -> tuple[list[str], list[str]]: candidates = [m for m in candidates if m["model"] in have_set] result: dict[str, Any] = { - "tier": tier, + **head, "floor": floor, "tie_band": tie_band, "allow_list": str(allow_path) if allow_path else None, @@ -335,16 +596,23 @@ def resolve(names: list[str]) -> tuple[list[str], list[str]]: result["reason"] = _empty_reason(excluded, allow_path is not None, available) return result - result.update(select(candidates, tier=tier, floor=floor, tie_band=tie_band)) + result.update( + select(candidates, tier=tier, floor=floor, tie_band=tie_band, **select_args) + ) result["status"] = "ok" if result["recommended"] else "nothing_clears_floor" if result["status"] == "nothing_clears_floor": best = result["below_floor"][0] if result["below_floor"] else None + skipped = [r["model"] for r in result["skipped_for_failures"]] result["reason"] = ( f"nothing available reaches {floor} on {tier} tasks; closest is " f"{best['model']} at {best['score']}, short by " f"{round(floor - best['score'], 2)}" if best else f"nothing available has a score for {tier} tasks" + ) + ( + f"; skipped for failing {tier} tasks: {', '.join(skipped)}" + if skipped + else "" ) # Sanity: the pick must not be beaten on both axes by another candidate. if result.get("recommended"): @@ -395,10 +663,30 @@ def _empty_reason( def _parse_args(argv: list[str] | None = None) -> argparse.Namespace: """Parse command-line arguments.""" p = argparse.ArgumentParser( - description="Pick a model given a quality floor and a difficulty tier.", + description="Pick a model for a coding task from six rubric scores.", + epilog=( + "Example:\n" + " python3 route.py --confidence high --rubric " + '\'{"hidden_contract": 1, "silent_invariant": 0, ' + '"coordinated_surfaces": 2, "ambiguity": 1, ' + '"verification": 1, "blast_radius": 1}\'\n' + "Legacy, for one release:\n" + " python3 route.py --tier high --floor 70" + ), + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + p.add_argument( + "--rubric", + help="The six rubric scores as JSON, each 0, 1 or 2. See rubric.md.", ) - p.add_argument("--tier", required=True, choices=TIERS) - p.add_argument("--floor", required=True, type=float) + p.add_argument( + "--confidence", + choices=CONFIDENCE_LEVELS, + help="How sure the scorer is. 'low' keeps the developer on their model.", + ) + p.add_argument("--policy", type=Path, default=DEFAULT_POLICY) + p.add_argument("--tier", choices=TIERS, help="Legacy: a difficulty tier.") + p.add_argument("--floor", type=float, help="Legacy: a quality floor.") p.add_argument( "--available", help="Comma-separated models to narrow the allow-list further. Omit " @@ -411,10 +699,21 @@ def _parse_args(argv: list[str] | None = None) -> argparse.Namespace: action="store_true", help="Ignore any allow-list on disk.", ) - p.add_argument("--tie-band", type=float, default=DEFAULT_TIE_BAND) + p.add_argument( + "--tie-band", + type=float, + default=None, + help="Points below which two models tie. Default: the policy's, or " + f"{DEFAULT_TIE_BAND} with --tier.", + ) p.add_argument("--models", type=Path, default=_HERE / "models.json") p.add_argument("--aliases", type=Path, default=_HERE / "model-aliases.json") - return p.parse_args(argv) + args = p.parse_args(argv) + if args.rubric is None and (args.tier is None or args.floor is None): + p.error("give --rubric, or the legacy --tier and --floor") + if args.rubric is not None and (args.tier is not None or args.floor is not None): + p.error("--rubric sets the bucket and required quality; drop --tier/--floor") + return args def main(argv: list[str] | None = None) -> int: @@ -424,6 +723,9 @@ def main(argv: list[str] | None = None) -> int: result = route( tier=args.tier, floor=args.floor, + rubric=parse_rubric(args.rubric) if args.rubric is not None else None, + confidence=args.confidence, + policy_path=args.policy, available=( [s.strip() for s in args.available.split(",") if s.strip()] if args.available diff --git a/vend/swe-router/routing-policy.json b/vend/swe-router/routing-policy.json new file mode 100644 index 00000000..af5191ab --- /dev/null +++ b/vend/swe-router/routing-policy.json @@ -0,0 +1,49 @@ +{ + "schema_version": "1.1", + "required_quality": { + "by_blast_radius": { + "0": 60, + "1": 70, + "2": 75 + }, + "add_when_verification_is_2": 5, + "meaning": { + "60": "Bottom of the judge's 'strong' band: a usable foundation; expect review to fix real gaps.", + "65": "Low 'strong': implementation-ready in parts; review will find gaps.", + "70": "Middle of 'strong': implementation-ready in most respects with limited gaps.", + "75": "Top of 'strong': few gaps and well-grounded; for security, money or data paths.", + "80": "Approaching the judge's 'excellent' band: comprehensive and precise." + } + }, + "difficulty": { + "dimensions": [ + "hidden_contract", + "silent_invariant", + "coordinated_surfaces", + "ambiguity" + ], + "bucketing": "absolute", + "buckets": { + "easy": [ + 0, + 1 + ], + "moderate": [ + 2, + 3 + ], + "hard": [ + 4, + 5 + ], + "very_hard": [ + 6, + 8 + ] + } + }, + "tie_band_points": 3, + "min_completion_in_bucket": 1.0, + "min_completion_in_bucket_meaning": "Skip a model that finished fewer than this share of the benchmark tasks in the task's bucket. A column mean leaves failed tasks out, so without this a model that failed 1 of 10 hard tasks looks as good as one that finished all 10. 1.0 skips any model with a failure in the bucket.", + "when_unsure": "stay_on_current_model" +} diff --git a/vend/swe-router/rubric.md b/vend/swe-router/rubric.md new file mode 100644 index 00000000..c7214524 --- /dev/null +++ b/vend/swe-router/rubric.md @@ -0,0 +1,91 @@ +# Task rubric + +Score the task on six dimensions, each 0, 1 or 2. The scores decide which model gets the work, so each one must describe the task as it would land in **this** repository, not in the abstract. + +## Before you score + +Read the repository's own map first: `AGENTS.md` at the root, else `CLAUDE.md`, else `README.md`. Follow the documents it links to. It tells you the stack, the layout, where the tests live and what the project treats as risky. If the repository has a `SECURITY.md`, read it too: it says what the project treats as security-sensitive, which bears on silent invariant and blast radius. Where any of these disagree with the code, trust the code. + +Then read the task, and look at the parts of the repository it touches: the files it names or implies, the tests around them, and anything those files depend on or are depended on by. You are judging the task, so do not write or change code. + +Three rules apply to every dimension: + +- **Size is not a signal.** Line count, number of files, length of the task text and how long the work might take do not raise or lower any score. A one-line change can score 2; a twenty-file change can score 0. +- **Score what the task needs, not what a model might get wrong.** You are describing the work, not predicting a failure. +- **Every score needs evidence.** Name the file, document, test, issue or sentence of the task that justifies it. A score you cannot point to is a guess; say so and lower your confidence. + +## 1. Hidden contract + +**Question:** Is the correct behaviour fully stated in the task, or does it depend on something defined elsewhere? + +**Look at:** what the task links to or mentions (an earlier issue, a design doc, an RFC, another service's API); comments and docs near the code it touches; tests that pin current behaviour other parts rely on; how the repository map describes the interfaces between components. + +- **0:** The task states the behaviour completely. You could write the acceptance test from the task text alone. +- **1:** The behaviour follows from nearby code or project conventions the task does not mention, such as a naming pattern, an existing error format, or how sibling commands already behave. Reading the surrounding code reveals it. +- **2:** The behaviour is fixed by something outside this change: an earlier issue or design decision, another service's contract, a spec, or callers that depend on it. A change that ignores it can look complete and still contradict it. + +**Common mistake:** scoring 0 because the task is short. "Implement the gate agreed in #809" is a 2. + +## 2. Silent invariant + +**Question:** Could a wrong change look like it works? + +**Look at:** whether the code touches authentication, authorisation, input validation, secrets, money, persisted data, migrations, caching, concurrency or retries; what the repository map calls security-critical; whether a mistake would show up in normal use or only under a particular input, timing or attacker. + +- **0:** A wrong change is visible: an error, a wrong value on screen, a failing command. +- **1:** A wrong change shows only in edge cases a normal check would miss: an unusual input, an alternate configuration, a second platform. +- **2:** Correctness hinges on a security, data-integrity or concurrency invariant, and breaking it is silent: the feature appears to work while it leaks, corrupts, races or lets the wrong caller through. + +**Common mistake:** scoring 0 because the change is small. Passing a configuration value through to a container is a 2 if that value is a security allow-list. + +## 3. Coordinated surfaces + +**Question:** How many separate places must change, and stay consistent with each other, for the task to be done? + +**Look at:** every surface that exposes or consumes the behaviour: API, command line, user interface, configuration schema, deployment manifests (Docker, Helm, Terraform), documentation, other services and generated clients. + +- **0:** One place. +- **1:** Two or three places inside one component that must agree. +- **2:** Several components or layers that must agree, where missing one leaves the feature half-built: for example the API, the CLI and the deployment configuration. + +**Common mistake:** counting files. Ten files in one module that change mechanically are a 0 or 1; a two-line change that must also reach a second service is a 2. + +## 4. Ambiguity + +**Question:** Is there a tempting answer that looks complete but misses what is actually needed? + +**Look at:** whether the task describes a symptom or a goal; whether it names the root cause; whether a narrower fix would make the reported symptom disappear while leaving the underlying problem; whether there are two reasonable readings of what to build. + +- **0:** One obvious reading, and the obvious fix is the right one. +- **1:** Some interpretation is needed, but the repository makes the intended answer clear once you look. +- **2:** A narrower or different solution would look complete and pass a quick review, yet miss the real problem or the intended design. + +**Common mistake:** scoring 0 when the task describes only a symptom. "Session cookies exceed 4096 bytes" invites shrinking one cookie when the real fix may be storing the data server-side. + +## 5. Verification + +**Question:** If the change were wrong, would anything in the repository catch it? + +**Look at:** the tests that cover the code the task touches, what they assert, whether they run locally and in CI, and whether the behaviour can be tested at all without external services, hardware or credentials. + +- **0:** Strong tests already cover this behaviour and would fail on a wrong change. +- **1:** Some tests exist but would miss part of what could go wrong. +- **2:** No meaningful tests cover it, or it cannot be tested without systems the repository does not have. A mistake would ship unnoticed. + +**Common mistake:** counting test files. A test suite that never exercises this code path is a 2. + +## 6. Blast radius + +**Question:** If this change shipped wrong, what would it cost, and how easily could it be undone? + +**Look at:** who uses the code (only its authors, internal staff, or customers), what the repository map says is production or user-facing, whether it touches security, money or stored data, and whether a mistake could be fixed by reverting a deployment. + +- **0:** Throwaway or internal: a prototype, a script, a test fixture, internal documentation. A mistake costs a rerun. +- **1:** Users would notice, but a revert fixes it with no lasting harm. +- **2:** Security, money or data loss, or anything hard to undo: leaked credentials, corrupted records, a bad migration, an authorisation hole. + +**Common mistake:** confusing blast radius with difficulty. A trivial change to a password check is a 2 here; a hard change to an internal dashboard is a 0. + +## Confidence + +After scoring, say how confident you are in the six scores as a whole: **high** when the repository and the task settle every score, **medium** when one or two rest on inference, **low** when you could not find what you needed to judge. From f64fa386d89fe08b172f322aac1715e168965964 Mon Sep 17 00:00:00 2001 From: shekharprateek Date: Tue, 20 Oct 2026 11:03:56 -0400 Subject: [PATCH 2/2] docs: correct three lines the 0.2.0 port left behind All three describe shipped behaviour inaccurately, and all three were introduced or missed by the two migration PRs, so they are cheaper to fix on this branch than as a follow-up against main. - README.md:121 still described the 0.1.x router -- a quality floor plus a tier table -- two paragraphs below the line 69 rewrite that describes the rubric. A reader got two incompatible accounts of the same skill. The row's link also pointed at .claude/skills/swe-router/SKILL.md, which is a symlink (mode 120000) and so does not resolve on github.com; it now points at vend/swe-router/SKILL.md, the real file. - docs/repository-structure.md:28 itemised the skill as the five-file 0.1.x layout. Added rubric.md and routing-policy.json, the two new files. - docs/strands-setup.md promised derived Claude pricing: "the table covers the Claude models from Haiku 4.5 to the Claude 5 family". It does not, and that is deliberate -- an Anthropic model reports its cost through the agent, so the table carries only the GPT models. Strands is not a pass-through agent though: it reports usage and no cost, so a Claude model under --agent strands records total_cost_usd null, which a live Bedrock run confirmed. The paragraph now says that and points at the token counts instead. Prose only; the price table is intentional and untouched. 549 tests, no new failures, no dangling links in the changed files. --- README.md | 2 +- docs/repository-structure.md | 2 +- docs/strands-setup.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index fcc71225..ee383ad6 100644 --- a/README.md +++ b/README.md @@ -118,7 +118,7 @@ Where to read more, by topic: | [docs/repository-structure.md](docs/repository-structure.md) | What lives where in this repository. | | [docs/diagrams/cost-quality-abstracted.html](docs/diagrams/cost-quality-abstracted.html) | Source for the cost/quality diagram above — a self-contained HTML page with the data inline. Edit and re-screenshot to regenerate. | | [.claude/skills/setup-machine/SKILL.md](.claude/skills/setup-machine/SKILL.md) | **Start here on a new machine.** What `/setup-machine` inspects and installs, why each component is needed, where the vLLM venv lands on a small root disk, and what it deliberately does not do. | -| [.claude/skills/swe-router/SKILL.md](.claude/skills/swe-router/SKILL.md) | The `/swe-router` skill: how it sets a quality floor from the consequence of a change being wrong, picks the tier table to read it against, and selects the cheapest model that clears it. Advisory -- it recommends and stops. | +| [vend/swe-router/SKILL.md](vend/swe-router/SKILL.md) | The `/swe-router` skill: the six rubric questions it answers about a task, how four of them set a difficulty bucket and two set the quality a model must reach, and how it then takes the cheapest permitted model that clears the bar in that bucket. Advisory -- it recommends and stops. | | [docs/vision.md](docs/vision.md) | The north star: a cost-aware harness that routes each task (and each phase) to the right model on the frontier -- frontier / workhorse / budget -- switching automatically. | | [benchmarks/README.md](benchmarks/README.md) | The benchmark harness landing page: the three hosting paths, how a run works, and how to reproduce the results above. | | [benchmarks/docs/harness-reference.md](benchmarks/docs/harness-reference.md) | Full harness reference: config, the `/swe2` flow, context-window/auto-compaction, and the LLM-as-judge scoring. | diff --git a/docs/repository-structure.md b/docs/repository-structure.md index 80a59a12..4c319fc6 100644 --- a/docs/repository-structure.md +++ b/docs/repository-structure.md @@ -25,7 +25,7 @@ sample-agentic-coding-harness-benchmarks/ │ └── vllm-setup/ /vllm-setup — stand up the EC2 vLLM server (Path 3) ├── docs/ ← Concepts, setup guides, and methodology ├── vend/ ← Vendored, installable artifacts -│ └── swe-router/ /swe-router skill: route.py, models.json, install.sh +│ └── swe-router/ /swe-router skill: route.py, rubric.md, routing-policy.json, models.json, install.sh ├── benchmarks/ ← The benchmark harness │ ├── README.md Harness landing page │ ├── docs/ Shared harness reference + one guide per hosting path diff --git a/docs/strands-setup.md b/docs/strands-setup.md index cbb9684c..f8913cb1 100644 --- a/docs/strands-setup.md +++ b/docs/strands-setup.md @@ -57,7 +57,7 @@ The runner reports the SDK's accumulated usage. The two providers define `inputT - Amazon Bedrock reports fresh input tokens, with cache reads and cache writes as separate counts. The harness uses them as they are. - The OpenAI-compatible client copies the endpoint's `prompt_tokens`, which includes cached tokens, into `inputTokens`, and reports the cached part again as `cacheReadInputTokens`. The harness subtracts the cached part, so the cached prompt is not billed twice. -Strands reports no dollar cost. The harness prices the tokens with [bedrock_pricing.py](../benchmarks/scripts/bedrock_pricing.py), as it does for codex. The table covers the Claude models from Haiku 4.5 to the Claude 5 family, read from the AWS Price List API: a `us.` or bare id pays the Regional rate, and a `global.` id pays the Global cross-region rate, which is 10% lower. A model missing from the table gets a null cost. `num_turns` is the SDK's event-loop cycle count: one model call and the tool calls it asked for. +Strands reports no dollar cost, so the harness falls back to the local price table in [bedrock_pricing.py](../benchmarks/scripts/bedrock_pricing.py), as it does for codex. That table deliberately carries only the GPT models: an Anthropic model reports its cost through the agent, so there is nothing to derive for the agents that pass it through. Strands is not one of those agents -- it reports usage but no cost -- so a Claude model driven by `--agent strands` or `--agent strands-harness` records `total_cost_usd` as **null**, and the token counts are the measurement to use. Price the run yourself from `metrics_that_matter` if you need a figure. `num_turns` is the SDK's event-loop cycle count: one model call and the tool calls it asked for. ## Known limits