Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .claude/skills/benchmark/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ Do these in order. Do not skip the HF-token check; a missing token is the most c
**2a. HuggingFace token must be available BEFORE starting vLLM.** `vllm-serve.sh` resolves a token from `$HF_TOKEN`, else a `.hf_token` file in the repo root, the `self-hosted/vllm/` dir, or `$HOME`. Check that at least one is present:

```bash
cd ~/agentic-coding-harness-benchmarks
cd ~/sample-agentic-coding-harness-benchmarks
if [ -n "${HF_TOKEN:-}" ] || [ -s .hf_token ] || [ -s self-hosted/vllm/.hf_token ] || [ -s "$HOME/.hf_token" ]; then
echo "HF token available"
else
Expand Down Expand Up @@ -195,7 +195,7 @@ Run the end-to-end script from `benchmarks/`. It re-runs every pre-flight check

```bash
cd benchmarks
./scripts/run-e2e-benchmark.sh --provider {provider} --model {model} --dataset {dataset} [--agent claude|pi|omp|codex|kiro] [--yes] [--count N] [--skip-judge]
./scripts/run-e2e-benchmark.sh --provider {provider} --model {model} --dataset {dataset} [--agent claude|pi|omp|codex|kiro|strands|strands-harness] [--yes] [--count N] [--skip-judge]
```

Tell the user, before it runs:
Expand Down
2 changes: 1 addition & 1 deletion benchmarks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,7 @@ Two flags carry most of the method:
- `--holdout` routes each task from tier means recomputed with **that task excluded**. Without it the evaluation is in-sample: the skill builds `models.json` from these same 21 tasks, so it would score partly on data it has already seen. Leave-one-out gives the honest number. Run it in-sample and you get an upper bound.
- `--no-allow-list` ignores the organisation's approved-model list, so the result measures routing rather than local policy. Drop it to see what the shipped allow-list permits.

`--floor-sweep 55,65,70,75` replaces the judged floors with fixed ones, which shows how much the whole result depends on where the floor is set. Results and caveats: [Does routing beat picking one model?](../README.md#does-routing-beat-picking-one-model)
`--floor-sweep 55,65,70,75` replaces the judged floors with fixed ones, which shows how much the whole result depends on where the floor is set. Results and caveats: [Does routing pay for itself?](../docs/swe-router-evaluation.md)

## Quick start

Expand Down
2 changes: 1 addition & 1 deletion benchmarks/docs/path-open-weight-on-bedrock-litellm.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Path 2 - open-weight models on Amazon Bedrock via a LiteLLM proxy

Use this path to benchmark **non-Anthropic (open-weight) models hosted on Amazon Bedrock** -- Moonshot AI's Kimi, Qwen, DeepSeek, Mistral, MiniMax, GLM, GPT-OSS, and so on. `claude -p` only speaks the Anthropic Messages API, and Bedrock's native Anthropic route rejects these models (see [Path 1](path-anthropic-on-bedrock.md#anthropic-only)). The fix is a [LiteLLM proxy](https://docs.litellm.ai/docs/simple_proxy) that we run in front of Bedrock: it translates between the Anthropic Messages format Claude Code speaks and the OpenAI Chat Completions format the open-weight Bedrock models speak, so **any open-weight model on Bedrock can be wired into Claude Code** and driven through the harness with `provider: endpoint`.
Use this path to benchmark **non-Anthropic (open-weight) models hosted on Amazon Bedrock** -- Moonshot AI's Kimi, Qwen, DeepSeek, Mistral, MiniMax, GLM, GPT-OSS, and so on. `claude -p` only speaks the Anthropic Messages API, and Bedrock's native Anthropic route rejects these models (see [Path 1](path-anthropic-on-bedrock.md#supported-models)). The fix is a [LiteLLM proxy](https://docs.litellm.ai/docs/simple_proxy) that we run in front of Bedrock: it translates between the Anthropic Messages format Claude Code speaks and the OpenAI Chat Completions format the open-weight Bedrock models speak, so **any open-weight model on Bedrock can be wired into Claude Code** and driven through the harness with `provider: endpoint`.

The [scripts/bedrock-mantle-proxy.sh](../scripts/bedrock-mantle-proxy.sh) helper starts the proxy for you. For everything common to all paths -- dataset format, runner config, metrics file, and the judge -- see the [harness reference](harness-reference.md).

Expand Down
3 changes: 2 additions & 1 deletion benchmarks/scripts/run-benchmark-batch.sh
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,8 @@
# --dataset dataset YAML, relative to benchmarks/ (required)
# --models comma-separated model ids, run in order (required)
# --tasks comma-separated task ids; default is every task in the dataset
# --agent coding agent: pi (default), claude, omp, kiro
# --agent coding agent: pi (default), claude, omp, kiro, codex, strands,
# strands-harness
# --skill swe3 (default) or swe2
# --aws-region region for the codex judge (default us-east-1). Needed even on
# the vllm path: the model is local but the judge is on Bedrock.
Expand Down
6 changes: 3 additions & 3 deletions docs/vision.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# The vision: a cost-aware coding harness that routes across the frontier

This repository measures a [cost/quality Pareto frontier](../README.md#results-a-worked-example) -- the set of models where nothing else is both better and cheaper. Building that frontier is not the end goal. It is the **lookup table** for the thing we actually want to build: a coding harness (or agent) that, given a task, **intelligently picks the right model from the frontier for that task** -- and switches models mid-task when the situation changes -- so a developer gets frontier-level results at workhorse-level cost without ever having to think about model selection.
This repository measures a [cost/quality Pareto frontier](../README.md#step-1--benchmark-and-get-a-frontier) -- the set of models where nothing else is both better and cheaper. Building that frontier is not the end goal. It is the **lookup table** for the thing we actually want to build: a coding harness (or agent) that, given a task, **intelligently picks the right model from the frontier for that task** -- and switches models mid-task when the situation changes -- so a developer gets frontier-level results at workhorse-level cost without ever having to think about model selection.

## The problem this solves

Expand All @@ -23,7 +23,7 @@ The result: the developer states a task and a budget posture ("cheap", "balanced

## The first concrete step: `/swe-auto`

The first slice of this vision is **`/swe-auto`** -- a router skill that runs on **either Claude Code or pi**. The developer does not choose the model. Given a repo + ref + problem, a configurable **router model** triages the task read-only, classifies it as **frontier / workhorse / budget**, consults the measured [cost/quality Pareto frontier](../README.md#results-a-worked-example) to pick the cheapest non-dominated model that clears that tier's quality band, then shells out to the existing headless runner to run the **`/swe3`** skill with the selected model and harness -- producing the six artifacts (and, optionally, an `eval.json`). If the first pick fails to complete or scores below its band, it escalates one tier and re-runs, bounded by `max_escalations`.
The first slice of this vision is **`/swe-auto`** -- a router skill that runs on **either Claude Code or pi**. The developer does not choose the model. Given a repo + ref + problem, a configurable **router model** triages the task read-only, classifies it as **frontier / workhorse / budget**, consults the measured [cost/quality Pareto frontier](../README.md#step-1--benchmark-and-get-a-frontier) to pick the cheapest non-dominated model that clears that tier's quality band, then shells out to the existing headless runner to run the **`/swe3`** skill with the selected model and harness -- producing the six artifacts (and, optionally, an `eval.json`). If the first pick fails to complete or scores below its band, it escalates one tier and re-runs, bounded by `max_escalations`.

The key design decision: the skill does the triage and frontier lookup **inline** (cheap), then **executes via `run-swe-headless.py`** rather than spawning a subagent -- so the same executor path works whether the router runs under Claude Code or pi (pi cannot fan out). Note the two independent harness choices: which agent runs the *router skill*, and which agent the *executor* drives `/swe3` under (`--agent`); they can match or differ.

Expand Down Expand Up @@ -71,7 +71,7 @@ sequenceDiagram

You cannot route intelligently without knowing, per model:

1. **How good it is** -- the quality benchmark (judge scores, and the [per-dimension breakdown](../README.md#quality-by-dimension-where-models-are-strong-or-weak) that tells you *which kinds* of work each model is reliable at).
1. **How good it is** -- the quality benchmark (judge scores, and the [per-dimension breakdown](cost-per-task-methodology.md) that tells you *which kinds* of work each model is reliable at).
2. **What it costs** -- the throughput benchmark, turned into a hardware-derived [cost per task](cost-per-task-methodology.md).
3. **Where it sits relative to every other model** -- the Pareto frontier that combines the two.

Expand Down
2 changes: 1 addition & 1 deletion self-hosted/vllm/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@ It runs the steps below, reports each one, and stops for you at the "inference w
### Option B — run the scripts by hand

```bash
git clone https://github.com/aws-samples/sample-claude-code-multi-model ~/repo
git clone https://github.com/aws-samples/sample-agentic-coding-harness-benchmarks ~/repo
cd ~/repo/self-hosted/vllm/scripts

./vllm-install.sh # driver check → apt deps → uv → venv → vLLM → monitoring → GPU check
Expand Down
Loading