Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 3 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,11 +14,7 @@
<a href="docs/cost-per-task-methodology.md">Cost Methodology</a>
</p>

<p align="center">
<video src="https://github.com/user-attachments/assets/84f60f0f-cd7d-4513-b90b-37a0ffb5feed" controls width="820">
Your browser cannot play this video inline. <a href="https://github.com/user-attachments/assets/84f60f0f-cd7d-4513-b90b-37a0ffb5feed">Download the swe-router explainer</a>.
</video>
</p>
https://github.com/user-attachments/assets/84f60f0f-cd7d-4513-b90b-37a0ffb5feed

<p align="center"><em>swe-router in about 70 seconds: the right model, handed to the developer for each task, while the platform team benchmarks once and spends less.</em></p>

Expand Down Expand Up @@ -48,9 +44,9 @@ One command per model. The `/benchmark` skill runs the pre-flight checks, the ha

`agent` names the coding agent that drives the task and defaults to `claude`. The same flow runs headless from [`run-e2e-benchmark.sh`](benchmarks/scripts/run-e2e-benchmark.sh) (`--provider bedrock|litellm|vllm --model ... --dataset ... --agent claude|pi|omp|kiro|codex|strands|strands-harness --skill swe2|swe3`). Repeat across your model list, then the generators plot the cost/quality frontier for your repo and model set.

![Cost versus quality for eleven model classes on one repository: the six on the cost/quality frontier plus five that mark out the spread. Model names are abstracted — closed-source models by tier and generation, open-weight models by parameter count and dense or MoE architecture. A dashed frontier connects the models that nothing else beats on both score and cost.](docs/images/cost-quality-abstracted.png)
![Cost versus quality for twelve model classes on one repository: the six on the cost/quality frontier plus six that mark out the spread. Model names are abstracted — closed-source models by tier and generation, open-weight models by parameter count and dense or MoE architecture. A dashed frontier connects the models that nothing else beats on both score and cost.](docs/images/cost-quality-abstracted.png)

**What the shape tells you, and why you should measure your own.** The frontier is a step function, not a line: most of the quality is reached early and cheaply, and past that point each further point of score costs several times more per task. Where those steps fall depends on your code, your tasks and your hardware — the picture above is one repository and one run per task, so the useful output is the *shape*, not the numbers. Eleven of the nineteen models measured are plotted; the eight left out all sit below the frontier, so none of them changes its shape. Model identities are abstracted to their class deliberately: the class is what tells you the hardware a model needs, and that is the part that transfers. Regenerate it for your own repositories with the chart scripts in [`benchmarks/scripts/`](benchmarks/scripts/). The source for this diagram is [`docs/diagrams/cost-quality-abstracted.html`](docs/diagrams/cost-quality-abstracted.html).
**What the shape tells you, and why you should measure your own.** The frontier is a step function, not a line: most of the quality is reached early and cheaply, and past that point each further point of score costs several times more per task. Where those steps fall depends on your code, your tasks and your hardware — the picture above is one repository and one run per task, so the useful output is the *shape*, not the numbers. Twelve of the twenty-three models measured are plotted; the eleven left out all sit below the frontier, so none of them changes its shape. Model identities are abstracted to their class deliberately: the class is what tells you the hardware a model needs, and that is the part that transfers — the generations match the names in [`vend/swe-router/models.json`](vend/swe-router/models.json). The source for this diagram is [`docs/diagrams/cost-quality-abstracted.html`](docs/diagrams/cost-quality-abstracted.html); edit the data at the top of its script and re-render it at 1600×1104 with a device scale factor of 2 to get the committed 3200×2208 PNG. For charts straight off your own run metrics, use the plot scripts in [`benchmarks/scripts/`](benchmarks/scripts/).

## Step 2 — Developers install the skill

Expand Down
33 changes: 17 additions & 16 deletions docs/diagrams/cost-quality-abstracted.html
Original file line number Diff line number Diff line change
Expand Up @@ -79,12 +79,12 @@
<h1 class="viz-title">Cost vs. quality on one repository — by model class</h1>
<p class="viz-sub">
Mean judge score against mean cost per task, over 21 software-engineering tasks in a single repository.
Eleven of the nineteen models measured are shown: the six on the frontier, plus five that mark out the spread.
Twelve of the twenty-three models measured are shown: the six on the frontier, plus six that mark out the spread.
Model names are abstracted to their class — closed-source models by tier and generation, open-weight models by architecture.
</p>

<svg id="chart" viewBox="0 0 1536 872" width="1536" height="872"
role="img" aria-label="Scatter plot of mean task score against mean cost per task for eleven model classes — the six on the cost/quality frontier plus five off it. A dashed line connects the frontier points, which step steeply from 59 to 78 points of score for under a dollar more per task, then rise only a few points across the remaining tenfold increase in cost.">
role="img" aria-label="Scatter plot of mean task score against mean cost per task for twelve model classes — the six on the cost/quality frontier plus six off it. A dashed line connects the frontier points, which step steeply from 59 to 78 points of score for under a dollar more per task, then rise only a few points across the remaining tenfold increase in cost.">
<defs>
<linearGradient id="frontFill" x1="0" y1="0" x2="0" y2="1">
<stop offset="0%" stop-color="#1baf7a" stop-opacity="0.10"/>
Expand All @@ -96,9 +96,9 @@ <h1 class="viz-title">Cost vs. quality on one repository — by model class</h1>

<p class="viz-note">
<strong>What is abstracted, and what is left out.</strong> Only the model names. Every score, cost and position is the measured value from the run.
A closed-source point is named by tier and generation — generation numbers ascend with recency and are relative to this chart, not a vendor's version numbering.
A closed-source point is named by tier and generation — generation numbers ascend with recency over every model measured here, matching vend/swe-router/models.json, and are not a vendor's version numbering.
An open-weight point is named by parameter count and dense/MoE architecture, since that is what determines the hardware it needs.
Eight further measured models are omitted to keep the chart readable; all of them sit below the frontier, so none of them changes its shape.
Eleven further measured models are omitted to keep the chart readable; all of them sit below the frontier, so none of them changes its shape.
<br>
<strong>Cost basis — the two colours are not one currency.</strong> A closed-source point is a metered per-token bill.
An open-weight point is hardware-derived: the GPU instance's hourly rate divided by measured throughput at a stated concurrency.
Expand All @@ -108,14 +108,15 @@ <h1 class="viz-title">Cost vs. quality on one repository — by model class</h1>
</div>

<script>
// A readable subset of the 19 models measured: the six on the frontier, plus five
// A readable subset of the 23 models measured: the six on the frontier, plus six
// off-frontier points that each make a distinct point about the spread. Generation
// numbers ascend with recency and are relative to this set only.
// numbers ascend with recency over the whole measured set and match models.json.
const DATA = [
// closed-source, metered bill
{ c: 11.9543, s: 82.83, k: 'closed', label: 'Closed frontier (gen 3, current)', la: 'end', dx: -8, dy: -20 },
{ c: 4.6715, s: 76.97, k: 'closed', label: 'Closed mid-tier (gen 3)', la: 'start', dx: 18, dy: 6 },
{ c: 5.3200, s: 74.69, k: 'closed', label: 'Closed frontier (gen 2)', la: 'start', dx: 18, dy: 6 },
{ c: 11.9543, s: 82.83, k: 'closed', label: 'Closed frontier (gen 5)', la: 'end', dx: -8, dy: -20 },
{ c: 4.4400, s: 81.16, k: 'closed', label: 'Closed frontier (gen 6, current)', la: 'start', dx: 18, dy: -14 },
{ c: 4.6715, s: 76.97, k: 'closed', label: 'Closed mid-tier (gen 5)', la: 'start', dx: 18, dy: 6 },
{ c: 5.3200, s: 74.69, k: 'closed', label: 'Closed frontier (gen 4)', la: 'start', dx: 18, dy: 6 },
{ c: 4.1800, s: 66.32, k: 'closed', label: 'Closed frontier (gen 1)', la: 'end', dx: -16, dy: 6 },

// open-weight, self-hosted, hardware-derived
Expand All @@ -131,16 +132,16 @@ <h1 class="viz-title">Cost vs. quality on one repository — by model class</h1>
// The cost/quality frontier: nothing scores at least as high for at most the cost.
const FRONTIER = [
{ c: 0.2594, s: 59.24 }, { c: 0.8708, s: 59.74 }, { c: 1.4738, s: 78.48 },
{ c: 4.3310, s: 79.17 }, { c: 7.0397, s: 80.11 }, { c: 11.9543, s: 82.83 },
{ c: 4.3310, s: 79.17 }, { c: 4.4400, s: 81.16 }, { c: 11.9543, s: 82.83 },
];

const PANEL = [
['Open MoE 35B / 3B', '59.2', '$0.26', ''],
['Open dense 31B', '59.7', '$0.87', '+0.5 / +$0.61'],
['Open dense 27B', '78.5', '$1.47', '+18.8 / +$0.60'],
['Open MoE 2.8T', '79.2', '$4.33', '+0.7 / +$2.86'],
['Open MoE 743B', '80.1', '$7.04', '+0.9 / +$2.71'],
['Closed frontier', '82.8', '$11.95', '+2.7 / +$4.91'],
['Open MoE 35B / 3B', '59.2', '$0.26', ''],
['Open dense 31B', '59.7', '$0.87', '+0.5 / +$0.61'],
['Open dense 27B', '78.5', '$1.47', '+18.8 / +$0.60'],
['Open MoE 2.8T', '79.2', '$4.33', '+0.7 / +$2.86'],
['Closed frontier (gen 6)', '81.2', '$4.44', '+2.0 / +$0.11'],
['Closed frontier (gen 5)', '82.8', '$11.95', '+1.7 / +$7.51'],
];

const L = 116, R = 1516, T = 30, B = 762; // plot rect
Expand Down
Binary file modified docs/images/cost-quality-abstracted.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading