From ac837c087096bb13a73b2f6c858ffd5c9d4fbe34 Mon Sep 17 00:00:00 2001 From: shekharprateek Date: Tue, 20 Oct 2026 11:47:47 -0400 Subject: [PATCH] docs: correct stale agent lists, five broken anchors, and two wrong repo URLs Loose ends from PRs #31 and #32, plus link rot two earlier link-fix PRs could not catch because they resolved file paths only, never anchors. Stale agent lists (both are user-facing help, not comments): - run-benchmark-batch.sh:45 listed "pi (default), claude, omp, kiro". The flag is parsed generically and forwarded, and `-h` prints the script's own header block, so that line IS the help. Added codex, strands and strands-harness. - .claude/skills/benchmark/SKILL.md: the example command line offered --agent claude|pi|omp|codex|kiro, missing both agents PR #32 wired up. Broken anchors, all five verified absent on both sides before retargeting: - path-open-weight-on-bedrock-litellm.md pointed at path-anthropic-on-bedrock.md#anthropic-only. Self-inflicted: d92bf64 renamed that heading to "Supported models" without touching the inbound link. - docs/vision.md used ../README.md#results-a-worked-example TWICE (lines 3 and 26) and #quality-by-dimension-where-models-are-strong-or-weak once. No such headings exist here or upstream. Repointed at the Step 1 heading and at cost-per-task-methodology.md. - benchmarks/README.md:103 pointed at ../README.md#does-routing-beat-picking-one-model, also nonexistent. Repointed at docs/swe-router-evaluation.md. Wrong repo identities: - self-hosted/vllm/README.md:109 cloned sample-claude-code-multi-model, the predecessor repo, then cd'd into a script that lives in THIS one. The ~/repo target is kept so the following cd still works. - .claude/skills/benchmark/SKILL.md:69 was `cd ~/agentic-coding-harness- benchmarks`, the fork's clone directory name. Checked with an anchor-aware sweep of all 77 markdown files. Worth recording how GitHub builds a slug, because getting it wrong manufactures false positives: it lowercases, drops anything that is not a word character, space or hyphen (underscores are word characters and survive), then replaces EACH space with a hyphen without collapsing runs. So removed punctuation leaves a DOUBLE hyphen -- "API / CLI Design" is #api--cli-design, not #api-cli-design. Links like #pagedattention--kv-cache and #client-test-programs-pyprojecttoml--uv are correct as written and are deliberately untouched. After this the sweep reports one remaining item, SUPPORT.md -> ../../issues, which is the GitHub relative-URL idiom for the repo's issues page and resolves on github.com. Left alone, as PR #29 left it. 530 tests pass with no new failures; shell syntax checked. --- .claude/skills/benchmark/SKILL.md | 4 ++-- benchmarks/README.md | 2 +- benchmarks/docs/path-open-weight-on-bedrock-litellm.md | 2 +- benchmarks/scripts/run-benchmark-batch.sh | 3 ++- docs/vision.md | 6 +++--- self-hosted/vllm/README.md | 2 +- 6 files changed, 10 insertions(+), 9 deletions(-) diff --git a/.claude/skills/benchmark/SKILL.md b/.claude/skills/benchmark/SKILL.md index 01da9b27..a540975d 100644 --- a/.claude/skills/benchmark/SKILL.md +++ b/.claude/skills/benchmark/SKILL.md @@ -66,7 +66,7 @@ Do these in order. Do not skip the HF-token check; a missing token is the most c **2a. HuggingFace token must be available BEFORE starting vLLM.** `vllm-serve.sh` resolves a token from `$HF_TOKEN`, else a `.hf_token` file in the repo root, the `self-hosted/vllm/` dir, or `$HOME`. Check that at least one is present: ```bash -cd ~/agentic-coding-harness-benchmarks +cd ~/sample-agentic-coding-harness-benchmarks if [ -n "${HF_TOKEN:-}" ] || [ -s .hf_token ] || [ -s self-hosted/vllm/.hf_token ] || [ -s "$HOME/.hf_token" ]; then echo "HF token available" else @@ -195,7 +195,7 @@ Run the end-to-end script from `benchmarks/`. It re-runs every pre-flight check ```bash cd benchmarks -./scripts/run-e2e-benchmark.sh --provider {provider} --model {model} --dataset {dataset} [--agent claude|pi|omp|codex|kiro] [--yes] [--count N] [--skip-judge] +./scripts/run-e2e-benchmark.sh --provider {provider} --model {model} --dataset {dataset} [--agent claude|pi|omp|codex|kiro|strands|strands-harness] [--yes] [--count N] [--skip-judge] ``` Tell the user, before it runs: diff --git a/benchmarks/README.md b/benchmarks/README.md index bc215bf9..28e63805 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -100,7 +100,7 @@ Two flags carry most of the method: - `--holdout` routes each task from tier means recomputed with **that task excluded**. Without it the evaluation is in-sample: the skill builds `models.json` from these same 21 tasks, so it would score partly on data it has already seen. Leave-one-out gives the honest number. Run it in-sample and you get an upper bound. - `--no-allow-list` ignores the organisation's approved-model list, so the result measures routing rather than local policy. Drop it to see what the shipped allow-list permits. -`--floor-sweep 55,65,70,75` replaces the judged floors with fixed ones, which shows how much the whole result depends on where the floor is set. Results and caveats: [Does routing beat picking one model?](../README.md#does-routing-beat-picking-one-model) +`--floor-sweep 55,65,70,75` replaces the judged floors with fixed ones, which shows how much the whole result depends on where the floor is set. Results and caveats: [Does routing pay for itself?](../docs/swe-router-evaluation.md) ## Quick start diff --git a/benchmarks/docs/path-open-weight-on-bedrock-litellm.md b/benchmarks/docs/path-open-weight-on-bedrock-litellm.md index c6e4eb64..61948491 100644 --- a/benchmarks/docs/path-open-weight-on-bedrock-litellm.md +++ b/benchmarks/docs/path-open-weight-on-bedrock-litellm.md @@ -1,6 +1,6 @@ # Path 2 - open-weight models on Amazon Bedrock via a LiteLLM proxy -Use this path to benchmark **non-Anthropic (open-weight) models hosted on Amazon Bedrock** -- Moonshot AI's Kimi, Qwen, DeepSeek, Mistral, MiniMax, GLM, GPT-OSS, and so on. `claude -p` only speaks the Anthropic Messages API, and Bedrock's native Anthropic route rejects these models (see [Path 1](path-anthropic-on-bedrock.md#anthropic-only)). The fix is a [LiteLLM proxy](https://docs.litellm.ai/docs/simple_proxy) that we run in front of Bedrock: it translates between the Anthropic Messages format Claude Code speaks and the OpenAI Chat Completions format the open-weight Bedrock models speak, so **any open-weight model on Bedrock can be wired into Claude Code** and driven through the harness with `provider: endpoint`. +Use this path to benchmark **non-Anthropic (open-weight) models hosted on Amazon Bedrock** -- Moonshot AI's Kimi, Qwen, DeepSeek, Mistral, MiniMax, GLM, GPT-OSS, and so on. `claude -p` only speaks the Anthropic Messages API, and Bedrock's native Anthropic route rejects these models (see [Path 1](path-anthropic-on-bedrock.md#supported-models)). The fix is a [LiteLLM proxy](https://docs.litellm.ai/docs/simple_proxy) that we run in front of Bedrock: it translates between the Anthropic Messages format Claude Code speaks and the OpenAI Chat Completions format the open-weight Bedrock models speak, so **any open-weight model on Bedrock can be wired into Claude Code** and driven through the harness with `provider: endpoint`. The [scripts/bedrock-mantle-proxy.sh](../scripts/bedrock-mantle-proxy.sh) helper starts the proxy for you. For everything common to all paths -- dataset format, runner config, metrics file, and the judge -- see the [harness reference](harness-reference.md). diff --git a/benchmarks/scripts/run-benchmark-batch.sh b/benchmarks/scripts/run-benchmark-batch.sh index 6abbad31..ca98d203 100755 --- a/benchmarks/scripts/run-benchmark-batch.sh +++ b/benchmarks/scripts/run-benchmark-batch.sh @@ -42,7 +42,8 @@ # --dataset dataset YAML, relative to benchmarks/ (required) # --models comma-separated model ids, run in order (required) # --tasks comma-separated task ids; default is every task in the dataset -# --agent coding agent: pi (default), claude, omp, kiro +# --agent coding agent: pi (default), claude, omp, kiro, codex, strands, +# strands-harness # --skill swe3 (default) or swe2 # --aws-region region for the codex judge (default us-east-1). Needed even on # the vllm path: the model is local but the judge is on Bedrock. diff --git a/docs/vision.md b/docs/vision.md index 53032db1..1b2ac0a2 100644 --- a/docs/vision.md +++ b/docs/vision.md @@ -1,6 +1,6 @@ # The vision: a cost-aware coding harness that routes across the frontier -This repository measures a [cost/quality Pareto frontier](../README.md#results-a-worked-example) -- the set of models where nothing else is both better and cheaper. Building that frontier is not the end goal. It is the **lookup table** for the thing we actually want to build: a coding harness (or agent) that, given a task, **intelligently picks the right model from the frontier for that task** -- and switches models mid-task when the situation changes -- so a developer gets frontier-level results at workhorse-level cost without ever having to think about model selection. +This repository measures a [cost/quality Pareto frontier](../README.md#step-1--benchmark-and-get-a-frontier) -- the set of models where nothing else is both better and cheaper. Building that frontier is not the end goal. It is the **lookup table** for the thing we actually want to build: a coding harness (or agent) that, given a task, **intelligently picks the right model from the frontier for that task** -- and switches models mid-task when the situation changes -- so a developer gets frontier-level results at workhorse-level cost without ever having to think about model selection. ## The problem this solves @@ -23,7 +23,7 @@ The result: the developer states a task and a budget posture ("cheap", "balanced ## The first concrete step: `/swe-auto` -The first slice of this vision is **`/swe-auto`** -- a router skill that runs on **either Claude Code or pi**. The developer does not choose the model. Given a repo + ref + problem, a configurable **router model** triages the task read-only, classifies it as **frontier / workhorse / budget**, consults the measured [cost/quality Pareto frontier](../README.md#results-a-worked-example) to pick the cheapest non-dominated model that clears that tier's quality band, then shells out to the existing headless runner to run the **`/swe3`** skill with the selected model and harness -- producing the six artifacts (and, optionally, an `eval.json`). If the first pick fails to complete or scores below its band, it escalates one tier and re-runs, bounded by `max_escalations`. +The first slice of this vision is **`/swe-auto`** -- a router skill that runs on **either Claude Code or pi**. The developer does not choose the model. Given a repo + ref + problem, a configurable **router model** triages the task read-only, classifies it as **frontier / workhorse / budget**, consults the measured [cost/quality Pareto frontier](../README.md#step-1--benchmark-and-get-a-frontier) to pick the cheapest non-dominated model that clears that tier's quality band, then shells out to the existing headless runner to run the **`/swe3`** skill with the selected model and harness -- producing the six artifacts (and, optionally, an `eval.json`). If the first pick fails to complete or scores below its band, it escalates one tier and re-runs, bounded by `max_escalations`. The key design decision: the skill does the triage and frontier lookup **inline** (cheap), then **executes via `run-swe-headless.py`** rather than spawning a subagent -- so the same executor path works whether the router runs under Claude Code or pi (pi cannot fan out). Note the two independent harness choices: which agent runs the *router skill*, and which agent the *executor* drives `/swe3` under (`--agent`); they can match or differ. @@ -71,7 +71,7 @@ sequenceDiagram You cannot route intelligently without knowing, per model: -1. **How good it is** -- the quality benchmark (judge scores, and the [per-dimension breakdown](../README.md#quality-by-dimension-where-models-are-strong-or-weak) that tells you *which kinds* of work each model is reliable at). +1. **How good it is** -- the quality benchmark (judge scores, and the [per-dimension breakdown](cost-per-task-methodology.md) that tells you *which kinds* of work each model is reliable at). 2. **What it costs** -- the throughput benchmark, turned into a hardware-derived [cost per task](cost-per-task-methodology.md). 3. **Where it sits relative to every other model** -- the Pareto frontier that combines the two. diff --git a/self-hosted/vllm/README.md b/self-hosted/vllm/README.md index 96e93b73..06ff66ad 100644 --- a/self-hosted/vllm/README.md +++ b/self-hosted/vllm/README.md @@ -106,7 +106,7 @@ It runs the steps below, reports each one, and stops for you at the "inference w ### Option B — run the scripts by hand ```bash -git clone https://github.com/aws-samples/sample-claude-code-multi-model ~/repo +git clone https://github.com/aws-samples/sample-agentic-coding-harness-benchmarks ~/repo cd ~/repo/self-hosted/vllm/scripts ./vllm-install.sh # driver check → apt deps → uv → venv → vLLM → monitoring → GPU check