diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index 8f6fa5d..a461af1 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -12,7 +12,7 @@ "displayName": "Agent Stack", "source": "./plugins/agent-stack", "description": "Four skills: agent-orchestrator — tool-calling loops, pipelines with checkpoints, provider routing with fallback, memory architecture, plus the wallet side of reselling LLM access; agent-evals — run/trace/thread evals, LLM judges, and fixtures grown from production; agent-interop — MCP servers and clients, A2A agent cards, the MCP Registry, and gateways; agent-harness — system prompts, tool shaping, workflow-vs-agent, and auditing an agent system.", - "version": "0.17.1", + "version": "0.18.0", "author": { "name": "ssheleg", "url": "https://x.com/sshlg93" diff --git a/CHANGELOG.md b/CHANGELOG.md index 6a32431..d9a2ad8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,51 @@ +## v0.18.0 — how many runs before a difference is real, and the trajectory rule between its two measured edges + +`agent-evals` shipped 315 lines of doctrine about **what** to assert with **no +`references/` directory at all**, and said nothing anywhere about how many times to run +anything. A skill whose whole job is answering *did it get better* had no way to say +whether a number was a result or noise. + +**`references/statistics.md`** is that layer, and every figure in it was recomputed rather +than quoted: + +- `SE(p) = √(p(1−p)/n)` — at n=100, p=0.70 the 95% band is **±8.98 pp**, so a 73%-vs-70% + comparison on a hundred cases is a number inside its own noise. Error falls as `1/√n`, + which makes the remedy *more tasks*, not more argument. +- **`pass@k` and `pass^k` differ by 91 points on the same agent** — at p=0.6, k=5 they are + 99.0% and 7.8%. The first is a capability ceiling a human picks from; the second is what + a payment or a permission change needs. An operation with side effects may not "retry + until it works", so `pass@k` is not available to it as a metric at all. +- **Trials are not independent, and the published data proves it.** τ-bench's airline + Pass^k for claude-3-5-sonnet runs 0.460 → 0.326 → 0.263 → 0.225, where independence from + Pass^1 would predict 0.460 → 0.212 → 0.097 → 0.045. Successes cluster by task, not by + trial. So `pass^k` cannot be computed from `pass^1`, and Anthropic's `0.75³ ≈ 42%` is the + right shape for an argument and the wrong number for a gate. **This one is ours** — it + came out of recomputing the table rather than restating it. +- Pairing on the same tasks and the same 3–5 seeds with McNemar or a paired bootstrap, + because the task-difficulty variance the previous point measures is exactly what pairing + removes. +- **The harness is a variable**: 6 pp between the most- and least-resourced setups on + Terminal-Bench 2.0 (p<0.01), moving within noise from 1× to 3× (p=0.40) and lifting ~4 pp + from 3× to uncapped — because generous headroom lets the agent attempt strategies a tight + cap forbids. Two caps measure two agents. The remedy is a floor *and* a ceiling. +- A ladder for what a given piece of evidence authorises next, ending on the rule people + skip: **4/4 on a slice is not 100% system-wide.** + +**§5's trajectory rule moved between two measurements rather than being deleted.** It read +*"Judge the trajectory, not just the answer. Right tools, right order, right arguments."* +Anthropic calls exact tool-order assertions *"too rigid … agents regularly find valid +approaches that eval designers didn't anticipate"*, with a worked case of an agent that +solved a τ²-bench booking task through a policy loophole and failed the eval while serving +the user better. But the opposite edge is measured too: a grader blind to the trajectory +misses **44% of safety violations and 13% of robustness failures**. So the rule now reads +*read the trajectory; do not match it* — assert what was produced and what changed, and use +the trajectory as a **set and a forbidden list** for the claims an outcome cannot carry. +§2's axis table lost its `→` sequence example for the same reason. + +The single-step example keeps its *"must call `find_meeting_times` first"*, with a sentence +saying why: at that granularity the fixture **is** one decision, so ordering is the subject +rather than a proxy for it. Across a trajectory it stops being one. + # Changelog ## v0.17.1 — the evals run for the first time, and the tails go to zero diff --git a/docs/evidence/backlog.md b/docs/evidence/backlog.md index 5581e65..0c8b69f 100644 --- a/docs/evidence/backlog.md +++ b/docs/evidence/backlog.md @@ -38,3 +38,4 @@ row per line, appended, so two runs closing two rows do not land on the same lin | AG-06b | **The 2592 existing trees are reported and left alone.** `find "$TMPDIR" -maxdepth 4 -type f -path '*/copy/sub/b.sh' \\| wc -l` → **2592** on 2026-08-20, of which **16 were added by this run's own verification** — one run of the leaking fixture before the port, and one of the v0.12.0 archive's copy of it. They are plain `tmpXXXXXXXX` and indistinguishable from every other program's, so they are named rather than swept. The identical fixture still leaks in **`seo-aeo-audit`**, which another agent holds; referred, not edited. | Open by decision: state this run cannot prove is its own is reported, never deleted. | | B-118 (umbrella) | **The committed social card clips its eyebrow at the canvas.** The 91-char eyebrow line needs ~1270px at its set size against the 1200px canvas — roughly the last 11 characters are lost, "LLM RESALE" among them (measured by the umbrella's v1.11.0 run, 2026-08-31; the finding's home is the umbrella board row B-118). `test/social_preview.py` cannot see this by design: it checks the PNG signature, the under-1MB limit and the 1200x630 dimensions — bytes, never rendered text — so the gate stayed green over a clipped card and will stay green over the fix. | Open by instruction: the repair is a role-cell shortening that must be coordinated across the family's cards, so it was deliberately NOT folded into v0.17.1 (coordinator decision, 2026-08-31). Fix lands when the umbrella coordinates the wording. | | AG-04a | **This board's own priority is a scalar.** The header computes `P = blast × (1 + age_runs) / effort` while `audit.md` now argues that publishing axes beats multiplying them. The two are not the same claim — the audit is doctrine this pack ships to users about somebody else's system, the board is one repository's own triage — but the tension is real and it is not resolved by saying so. The method's home is the umbrella board (`sshlg-skills/docs/evidence/backlog.md`), which another agent holds; the check added here deliberately scopes to documents under `plugins/`, so it neither polices nor blesses this file. Referred to the umbrella. | Open: the board's formula and `audit.md` disagree, and neither has been chosen. | +| B-124 | **`agent-evals` had no `references/` directory and no statistics layer: 315 lines saying what to assert, nothing saying how many times.** A skill whose job is answering *did it get better* could not distinguish a result from noise — `grep -niE 'pass@|pass\^|variance|significance|seeds?'` over SKILL.md returned one hit, about escalation. Separately, §5's *"Right tools, right order"* is the exact practice Anthropic calls *"too rigid"*. | 2026-08-31 harness-engineering harvest; the statistics gap was reported independently by **four** of the twenty agents | 3 | 0 | 2 | **1.5** | **closed 2026-08-31, shipped in v0.18.0** — `references/statistics.md`: the noise band with its table, `pass@k` vs `pass^k` and the 91-point gap, the τ-bench data showing trials are correlated so `pass^k` cannot be exponentiated, paired comparison with 3–5 seeds, the harness as a 6 pp variable with a floor AND a ceiling, and the evidence ladder ending on *4/4 on a slice is not 100% system-wide*. Every figure recomputed before it was written; the correlation finding is ours, from recomputing rather than restating. §5 moved **between** its two measured edges — Anthropic's *too rigid* and Claw-Eval's 44% of safety violations missed by trajectory-blind grading — rather than flipping to the other extreme. | diff --git a/docs/evidence/verification.md b/docs/evidence/verification.md index 2193d82..a421134 100644 --- a/docs/evidence/verification.md +++ b/docs/evidence/verification.md @@ -10,6 +10,23 @@ This file exists because its absence read as zero exposure. `sshlg-skills` board --- +## Shipped state — v0.18.0 (2026-08-31) + +Measured on the release-candidate tree before the tag exists. B-124 from the +2026-08-31 harness-engineering harvest: the skill had no `references/` directory and no +statistics layer at all. + +| REQ | What ships | How it was confirmed | Confirmed | +|---|---|---|---| +| ST-1 | `agent-evals` gains its first reference, `references/statistics.md`, and the gate's two-way check binds it | `ls plugins/agent-stack/skills/agent-evals/references/` → `statistics.md`; `npm test` → rc=0 with `validate.py`'s references↔SKILL.md agreement passing in both directions (a link to a missing file and an unlinked orphan both fail it) | **observed** | +| ST-2 | Every figure in the reference is computed, not restated | recomputed before writing: `SE(0.70,100)=0.0458` → **±8.98 pp**, and ±4.49 / ±2.84 pp at n=400 / n=1000; `pass@5 = 1−0.4⁵ = 0.98976`; `pass^5 = 0.6⁵ = 0.07776`, a **91.2-point** gap; `0.75³ = 0.421875` | **observed** | +| ST-3 | The independence assumption behind `pass^k` is stated as false, with the published data that falsifies it | τ-bench airline Pass^k for claude-3-5-sonnet is 0.460 / 0.326 / 0.263 / 0.225; `0.460^k` gives 0.460 / 0.212 / 0.097 / 0.045 — the observed curve falls far slower, so trials are positively correlated and `pass^k` cannot be exponentiated from `pass^1`. **Not in the source that prompted this**; it came out of recomputing the table | **observed** | +| ST-4 | §5's trajectory rule sits between both measured edges rather than being deleted | `grep -c "Right tools, right order" SKILL.md` → **0**; the section now carries Anthropic's *"too rigid"* quote AND Claw-Eval's 44%/13% for trajectory-blind grading, resolving to *read the trajectory; do not match it* — a set and a forbidden list, never an order | **observed** | +| ST-5 | §2's axis table no longer prescribes an order matcher, and the one place order IS the subject says so | the Trajectory row reads *"what the run must not do, and what it must have touched — never the order"*; the single-step example keeps *"must call `find_meeting_times` first"* with a sentence explaining that at that granularity the fixture is one decision | **observed** | +| ST-6 | The checklist gains the line that makes the reference reachable from the gate a reader runs | `grep -c "statistics.md" SKILL.md` → 2 (checklist line + References table); body ~4343 tokens, inside the 4750 working limit | **observed** | + +--- + ## Shipped state — v0.17.1 (2026-08-31) Measured on the release-candidate tree before the tag exists. Wave-3 of the 2026-08-29 diff --git a/package.json b/package.json index 53a5e4c..5f52bec 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@ssheleg/agent-stack", - "version": "0.17.1", + "version": "0.18.0", "scripts": { "test": "python3 test/validate.py && python3 test/plant_guard_test.py && node test/installer_test.js" }, diff --git a/plugins/agent-stack/.claude-plugin/plugin.json b/plugins/agent-stack/.claude-plugin/plugin.json index 32f751c..0961660 100644 --- a/plugins/agent-stack/.claude-plugin/plugin.json +++ b/plugins/agent-stack/.claude-plugin/plugin.json @@ -3,7 +3,7 @@ "name": "agent-stack", "displayName": "Agent Stack", "description": "Four skills: agent-orchestrator — tool-calling loops, pipelines with checkpoints, provider routing with fallback, memory architecture, plus the wallet side of reselling LLM access; agent-evals — run/trace/thread evals, LLM judges, and fixtures grown from production; agent-interop — MCP servers and clients, A2A agent cards, the MCP Registry, and gateways; agent-harness — system prompts, tool shaping, workflow-vs-agent, and auditing an agent system.", - "version": "0.17.1", + "version": "0.18.0", "author": { "name": "ssheleg", "url": "https://x.com/sshlg93" diff --git a/plugins/agent-stack/skills/agent-evals/SKILL.md b/plugins/agent-stack/skills/agent-evals/SKILL.md index b274da2..8c201a6 100644 --- a/plugins/agent-stack/skills/agent-evals/SKILL.md +++ b/plugins/agent-stack/skills/agent-evals/SKILL.md @@ -63,6 +63,10 @@ point — tool name, argument shape. > "Schedule a meeting with Harrison tomorrow morning", with `find_meeting_times`, > `schedule_meeting` and `send_email` available, must call `find_meeting_times` first. +This is the one granularity where *first* is a legitimate assertion: the fixture is a +single decision, so the ordering claim is the subject rather than a proxy for it. Across a +whole trajectory it stops being one — see §5. + Cheap, deterministic, CI-blocking. **Precondition: a stable agent architecture.** These break on a graph refactor, and a suite that fails on every refactor gets deleted. @@ -72,7 +76,7 @@ Assert on three axes at once, with three different mechanisms: | Axis | Assert | With | |---|---|---| -| Trajectory | tool-call sequence — `read_file` → `edit_file` → `run_tests` | set/subset/order matchers | +| Trajectory | what the run **must not** do, and what it must have touched — never the order | set/subset matchers, forbidden-call lists | | Final response | quality, tone, policy compliance | rubric or judge | | **State change** | the memory row exists, the file was written, the artifact is there | direct inspection of the side effect | @@ -147,8 +151,20 @@ That is what makes a judge reproducible and a disagreement resolvable. assertions, tool-call correctness — all deterministic, all faster and cheaper than a model call. Send to a judge only what cannot be decided by code. -**Judge the trajectory, not just the answer.** Right tools, right order, right arguments. -An agent that reaches a correct answer through three wrong tool calls is a latent outage. +**Read the trajectory; do not match it.** An agent that reaches a correct answer through +three wrong tool calls is a latent outage — and asserting the *sequence* to catch that is +measurably the wrong instrument. Anthropic names the instinct and rejects its strict form: +exact tool-order assertions are *"too rigid and results in overly brittle tests, as agents +regularly find valid approaches that eval designers didn't anticipate"*, and the worked +case is an agent that solved a τ²-bench booking task through a policy loophole, failing +the eval as written while serving the user better. Grade **what was produced and what +changed**, and let the path vary. + +The opposite edge is measured too, so this is not "grade the final answer": a grader blind +to the trajectory misses **44% of safety violations and 13% of robustness failures**, +because a policy breach on the way to a correct result leaves no trace in the outcome. Use +the trajectory for the claims the outcome cannot carry — a forbidden call, a missing +confirmation, a secret read — as a **set and a forbidden list**, never as an order. **Calibrate the judge before trusting it.** Collect human labels on the same traces, measure agreement, iterate the judge prompt until agreement is high — *then* let it score @@ -304,6 +320,16 @@ None of the above runs without these, and they are the part people skip: - [ ] Every production failure minimised into a permanent fixture - [ ] Annotation queue with filters, and the two reviewer roles kept separate - [ ] Simulated users trained on real transcripts, with adversarial personas +- [ ] Every comparison that changes a decision states `n`, `k`, the reducer and the noise + band — `references/statistics.md`; a three-point gap on a hundred cases is noise + +--- + +## References + +| Load | When | +|---|---| +| [`references/statistics.md`](references/statistics.md) | a number is about to change a decision — how many runs before a difference is real, `pass@k` vs `pass^k` and why trials are not independent, paired comparison, the harness as a variable, and what a given piece of evidence authorises next | --- diff --git a/plugins/agent-stack/skills/agent-evals/references/statistics.md b/plugins/agent-stack/skills/agent-evals/references/statistics.md new file mode 100644 index 0000000..bc7c9f2 --- /dev/null +++ b/plugins/agent-stack/skills/agent-evals/references/statistics.md @@ -0,0 +1,201 @@ +# Statistics for agent evals — how many runs before a difference is real + +**Load this when a number is about to change a decision:** picking between two models or +two harness configurations, setting a release threshold, deciding whether a regression is +real, or writing the sentence "X is better than Y" in a document somebody will act on. + +Everything here is arithmetic over binary outcomes. It needs no library and no +statistician, and it is the layer most eval advice skips — including this skill's own +first eight sections, which say what to assert and never say how many times. + +## Contents + +- [The band around a pass rate](#the-band-around-a-pass-rate) +- [pass@k and pass^k are different questions](#passk-and-passk-are-different-questions) +- [Trials are not independent, and the published data says so](#trials-are-not-independent-and-the-published-data-says-so) +- [Pairing: same tasks, same seeds, per-task deltas](#pairing-same-tasks-same-seeds-per-task-deltas) +- [The harness is a variable, so pin it](#the-harness-is-a-variable-so-pin-it) +- [What a number authorises](#what-a-number-authorises) +- [The report contract](#the-report-contract) + +--- + +## The band around a pass rate + +A pass rate over `n` cases is an estimate, and its standard error is + +``` +SE(p) = sqrt( p * (1 - p) / n ) +``` + +The 95% band is roughly `±1.96 · SE`. Computed, not quoted: + +| n | p | 95% band | +|---|---|---| +| 100 | 0.70 | **±8.98 pp** | +| 400 | 0.70 | ±4.49 pp | +| 1000 | 0.70 | ±2.84 pp | + +```python +import math +def band(p, n): return 1.96 * math.sqrt(p * (1 - p) / n) * 100 # percentage points +``` + +**So "the new one gets 73% where the old one got 70%, on a hundred cases" is not a +result.** It is a number inside its own noise. The error shrinks as `1/√n`, which is the +whole practical consequence: **the fix for a 2–3 pp expected gain is more tasks, not more +argument.** Quadrupling the set halves the band. + +A corollary worth stating because leaderboards invite the opposite: **differences below +about 3 pp deserve scepticism until both configurations are documented and matched.** + +> The formula assumes independent cases. A benchmark whose tasks share a fixture, an +> environment or a generator violates that, and the true band is wider than this. Wider, +> never narrower — so the table is a floor on your uncertainty, not a ceiling. + +## pass@k and pass^k are different questions + +Two metrics, one letter apart, measuring opposite things. + +| Metric | Formula | Asks | Who it is for | +|---|---|---|---| +| `pass@k` | `1 − (1 − p)^k` | did **at least one** of k attempts succeed | a **capability ceiling** — a human picks the best of k | +| `pass^k` | `p^k` (if independent) | did **every** one of k succeed | a **reliability floor** — nobody is picking | + +At `p = 0.6`, `k = 5`: + +- `pass@5 = 1 − 0.4⁵ = 0.98976` → **99.0%** +- `pass^5 = 0.6⁵ = 0.07776` → **7.8%** + +**A 91-point gap between two numbers describing the same agent.** The first makes a demo; +the second is what a payment, a refund or a permission change actually needs. + +Anthropic states the same arithmetic for the everyday case: at a 75% per-trial rate, three +trials all passing is `0.75³ ≈ 42%` — verified, `0.421875`. + +**The rule that follows: an operation with side effects may not "retry until it works."** +If a failed attempt leaves a charge, a message or a mutated row behind, `pass@k` is not +available to you as a metric — you cannot pick the best of five refunds. Sample in a +sandbox or a rollback-capable environment, and count **every** failure. + +**A report that gives k without saying which k it means is unreadable.** *k independent +samples of one task* and *k consecutive tasks on one live pipeline* are different claims. + +## Trials are not independent, and the published data says so + +`pass^k = p^k` assumes each trial is a fresh coin flip. Real benchmarks do not behave that +way, and the τ-bench leaderboard is the cleanest demonstration — claude-3-5-sonnet on the +airline domain, published Pass^k beside what independence would predict from Pass^1: + +| k | published Pass^k | `0.460^k` if independent | +|---|---|---| +| 1 | 0.460 | 0.460 | +| 2 | **0.326** | 0.212 | +| 3 | **0.263** | 0.097 | +| 4 | **0.225** | 0.045 | + +The observed curve falls far slower than independence predicts. The reason is not +mysterious: **some tasks are reliably easy and some reliably hard**, so successes cluster +by task rather than scattering by trial. Positive correlation across trials of the same +task. + +Two consequences, and they cut in opposite directions: + +- **You cannot compute `pass^k` from `pass^1`.** Exponentiating a headline rate gives a + number far below the truth. Measure `pass^k` directly, at the k you care about. +- **Anthropic's `0.75³ ≈ 42%` is a worst case, not a forecast.** It is the right shape for + an argument — *consistency is a much harder bar* — and the wrong number to put in a + release gate. + +The other half of independence is the harness, not the task: Anthropic requires each trial +start from a clean environment, because *"unnecessary shared state between runs (leftover +files, cached data, resource exhaustion) can cause correlated failures."* Correlated +failures break the arithmetic above — so isolation is not hygiene, it is what makes the +metric mean anything. + +## Pairing: same tasks, same seeds, per-task deltas + +**Never subtract two independent averages.** Run both configurations over the *same* task +list with the *same* fixed seeds, record a per-task win/loss/tie, and test the deltas. + +``` +for task in tasks: # identical list + for seed in seeds: # identical seeds, 3-5 of them + a = run(config_A, task, seed) + b = run(config_B, task, seed) + delta[task, seed] = a - b +``` + +- **3–5 seeds per configuration**, reporting mean and spread. A single run screens a + direction; it does not establish one. +- Test the paired deltas with **McNemar** (binary outcomes) or a **paired bootstrap**. + Pairing removes the task-difficulty variance that dominates the unpaired comparison — + which is exactly the correlation the previous section measured. +- Testing several hypotheses at once needs a **multiple-comparisons correction**, or an + independent re-run of whichever ones came out positive. Five hypotheses at p<0.05 gives + you roughly a one-in-four chance of a false positive somewhere. + +**Ship on three conditions, not one:** the difference exceeds the noise band, it survives +the paired analysis, and it reproduces on a rerun. + +## The harness is a variable, so pin it + +The container spec is part of the measurement. On Terminal-Bench 2.0 the gap between the +most- and least-resourced setups was **6 percentage points (p < 0.01)** — larger than most +model differences anyone argues about. + +The shape of the effect matters more than the number: + +- Between **1× and 3×** the task's specified resources, scores move **within noise + (p = 0.40)**. +- From 3× to uncapped, infrastructure errors drop a further **1.6 pp** and success jumps + nearly **4 pp** — because the extra headroom lets the agent attempt strategies that only + work with it: pulling large dependencies, spawning expensive subprocesses, running + memory-hungry suites. +- Infrastructure errors alone: **5.8% of tasks at 1×**, cut to **2.1% at a 3× ceiling + (p < 0.001)**. + +So a tight cap and a generous cap **measure different agent strategies**, not the same +agent more or less precisely. The remedy is a **floor and a ceiling**, calibrated so that +scores at both fall within noise of each other — not a single pinned value. + +The effect is task-distribution dependent and does not transfer: SWE-bench moves only +**1.54 pp from 1× to 5×**. Measure it for your own suite rather than importing a multiplier. + +## What a number authorises + +Evidence licenses the next action its scope supports, and nothing further. A worked +three-round loop on a deliberately narrow slice — four tasks, one run each, model, seed, +step limit and environment fixed, arm order alternated, **one variable changed per round**: + +| Round | The only change | Success | Tokens vs control | What it authorised | +|---|---|---|---|---| +| H1 | added navigation and final-check instructions | 25% → 25% | 0.47× | the prompt is not the bottleneck — stop tuning it | +| H5 | accessibility feed → UIAutomator tree | 25% → **100%** | **2.498×** | right mechanism, too expensive — try to cheapen it | +| H5C | prune invisible/textless/non-actionable nodes | 100% → 100% | **0.506×** | qualifies for a full rerun | + +Two rules fall out, and the second is the one people skip: + +- **Observation before prompt.** More detailed instructions cannot restore information the + agent never received. When a score will not move, ask what the agent could see before + asking how it was asked. +- **4/4 on a slice is not 100% system-wide, and must not be reported as one.** With four + tasks per arm these numbers can decide whether a larger rerun is worth paying for. They + cannot estimate success across the benchmark. The ladder's output is *the next + experiment*, not a result. + +## The report contract + +A comparison that will change a decision states all of these, or it is a claim rather than +a measurement: + +- [ ] `n` — how many tasks, and `k` — how many runs each +- [ ] which reducer: `pass@k`, `pass^k`, `mean`, and **which k means what** +- [ ] the noise band for that `n`, computed +- [ ] paired or unpaired; if paired, the seed set +- [ ] the harness configuration: resource floor and ceiling, isolation between trials +- [ ] whether the difference reproduced on a rerun +- [ ] what the scope of the evidence authorises next — not what it suggests + +**A green suite with none of these is a number, not a verdict.** That distinction is the +reason this file exists.