diff --git a/README.md b/README.md index 016a5423..45a10002 100644 --- a/README.md +++ b/README.md @@ -14,11 +14,7 @@ Cost Methodology

-

- -

+https://github.com/user-attachments/assets/84f60f0f-cd7d-4513-b90b-37a0ffb5feed

swe-router in about 70 seconds: the right model, handed to the developer for each task, while the platform team benchmarks once and spends less.

@@ -48,9 +44,9 @@ One command per model. The `/benchmark` skill runs the pre-flight checks, the ha `agent` names the coding agent that drives the task and defaults to `claude`. The same flow runs headless from [`run-e2e-benchmark.sh`](benchmarks/scripts/run-e2e-benchmark.sh) (`--provider bedrock|litellm|vllm --model ... --dataset ... --agent claude|pi|omp|kiro|codex|strands|strands-harness --skill swe2|swe3`). Repeat across your model list, then the generators plot the cost/quality frontier for your repo and model set. -![Cost versus quality for eleven model classes on one repository: the six on the cost/quality frontier plus five that mark out the spread. Model names are abstracted — closed-source models by tier and generation, open-weight models by parameter count and dense or MoE architecture. A dashed frontier connects the models that nothing else beats on both score and cost.](docs/images/cost-quality-abstracted.png) +![Cost versus quality for twelve model classes on one repository: the six on the cost/quality frontier plus six that mark out the spread. Model names are abstracted — closed-source models by tier and generation, open-weight models by parameter count and dense or MoE architecture. A dashed frontier connects the models that nothing else beats on both score and cost.](docs/images/cost-quality-abstracted.png) -**What the shape tells you, and why you should measure your own.** The frontier is a step function, not a line: most of the quality is reached early and cheaply, and past that point each further point of score costs several times more per task. Where those steps fall depends on your code, your tasks and your hardware — the picture above is one repository and one run per task, so the useful output is the *shape*, not the numbers. Eleven of the nineteen models measured are plotted; the eight left out all sit below the frontier, so none of them changes its shape. Model identities are abstracted to their class deliberately: the class is what tells you the hardware a model needs, and that is the part that transfers. Regenerate it for your own repositories with the chart scripts in [`benchmarks/scripts/`](benchmarks/scripts/). The source for this diagram is [`docs/diagrams/cost-quality-abstracted.html`](docs/diagrams/cost-quality-abstracted.html). +**What the shape tells you, and why you should measure your own.** The frontier is a step function, not a line: most of the quality is reached early and cheaply, and past that point each further point of score costs several times more per task. Where those steps fall depends on your code, your tasks and your hardware — the picture above is one repository and one run per task, so the useful output is the *shape*, not the numbers. Twelve of the twenty-three models measured are plotted; the eleven left out all sit below the frontier, so none of them changes its shape. Model identities are abstracted to their class deliberately: the class is what tells you the hardware a model needs, and that is the part that transfers — the generations match the names in [`vend/swe-router/models.json`](vend/swe-router/models.json). The source for this diagram is [`docs/diagrams/cost-quality-abstracted.html`](docs/diagrams/cost-quality-abstracted.html); edit the data at the top of its script and re-render it at 1600×1104 with a device scale factor of 2 to get the committed 3200×2208 PNG. For charts straight off your own run metrics, use the plot scripts in [`benchmarks/scripts/`](benchmarks/scripts/). ## Step 2 — Developers install the skill diff --git a/docs/diagrams/cost-quality-abstracted.html b/docs/diagrams/cost-quality-abstracted.html index c3b5fada..2ade5fe5 100644 --- a/docs/diagrams/cost-quality-abstracted.html +++ b/docs/diagrams/cost-quality-abstracted.html @@ -79,12 +79,12 @@

Cost vs. quality on one repository — by model class

Mean judge score against mean cost per task, over 21 software-engineering tasks in a single repository. - Eleven of the nineteen models measured are shown: the six on the frontier, plus five that mark out the spread. + Twelve of the twenty-three models measured are shown: the six on the frontier, plus six that mark out the spread. Model names are abstracted to their class — closed-source models by tier and generation, open-weight models by architecture.

+ role="img" aria-label="Scatter plot of mean task score against mean cost per task for twelve model classes — the six on the cost/quality frontier plus six off it. A dashed line connects the frontier points, which step steeply from 59 to 78 points of score for under a dollar more per task, then rise only a few points across the remaining tenfold increase in cost."> @@ -96,9 +96,9 @@

Cost vs. quality on one repository — by model class

What is abstracted, and what is left out. Only the model names. Every score, cost and position is the measured value from the run. - A closed-source point is named by tier and generation — generation numbers ascend with recency and are relative to this chart, not a vendor's version numbering. + A closed-source point is named by tier and generation — generation numbers ascend with recency over every model measured here, matching vend/swe-router/models.json, and are not a vendor's version numbering. An open-weight point is named by parameter count and dense/MoE architecture, since that is what determines the hardware it needs. - Eight further measured models are omitted to keep the chart readable; all of them sit below the frontier, so none of them changes its shape. + Eleven further measured models are omitted to keep the chart readable; all of them sit below the frontier, so none of them changes its shape.
Cost basis — the two colours are not one currency. A closed-source point is a metered per-token bill. An open-weight point is hardware-derived: the GPU instance's hourly rate divided by measured throughput at a stated concurrency. @@ -108,14 +108,15 @@

Cost vs. quality on one repository — by model class