v0.18.0 — how many runs before a difference is real

.claude-plugin/marketplace.json

@@ -12,7 +12,7 @@
       "displayName": "Agent Stack",
       "source": "./plugins/agent-stack",
       "description": "Four skills: agent-orchestrator — tool-calling loops, pipelines with checkpoints, provider routing with fallback, memory architecture, plus the wallet side of reselling LLM access; agent-evals — run/trace/thread evals, LLM judges, and fixtures grown from production; agent-interop — MCP servers and clients, A2A agent cards, the MCP Registry, and gateways; agent-harness — system prompts, tool shaping, workflow-vs-agent, and auditing an agent system.",
-      "version": "0.17.1",
+      "version": "0.18.0",
       "author": {
         "name": "ssheleg",
         "url": "https://x.com/sshlg93"

CHANGELOG.md

@@ -1,3 +1,51 @@
+## v0.18.0 — how many runs before a difference is real, and the trajectory rule between its two measured edges
+
+`agent-evals` shipped 315 lines of doctrine about **what** to assert with **no
+`references/` directory at all**, and said nothing anywhere about how many times to run
+anything. A skill whose whole job is answering *did it get better* had no way to say
+whether a number was a result or noise.
+
+**`references/statistics.md`** is that layer, and every figure in it was recomputed rather
+than quoted:
+
+- `SE(p) = √(p(1−p)/n)` — at n=100, p=0.70 the 95% band is **±8.98 pp**, so a 73%-vs-70%
+  comparison on a hundred cases is a number inside its own noise. Error falls as `1/√n`,
+  which makes the remedy *more tasks*, not more argument.
+- **`pass@k` and `pass^k` differ by 91 points on the same agent** — at p=0.6, k=5 they are
+  99.0% and 7.8%. The first is a capability ceiling a human picks from; the second is what
+  a payment or a permission change needs. An operation with side effects may not "retry
+  until it works", so `pass@k` is not available to it as a metric at all.
+- **Trials are not independent, and the published data proves it.** τ-bench's airline
+  Pass^k for claude-3-5-sonnet runs 0.460 → 0.326 → 0.263 → 0.225, where independence from
+  Pass^1 would predict 0.460 → 0.212 → 0.097 → 0.045. Successes cluster by task, not by
+  trial. So `pass^k` cannot be computed from `pass^1`, and Anthropic's `0.75³ ≈ 42%` is the
+  right shape for an argument and the wrong number for a gate. **This one is ours** — it
+  came out of recomputing the table rather than restating it.
+- Pairing on the same tasks and the same 3–5 seeds with McNemar or a paired bootstrap,
+  because the task-difficulty variance the previous point measures is exactly what pairing
+  removes.
+- **The harness is a variable**: 6 pp between the most- and least-resourced setups on
+  Terminal-Bench 2.0 (p<0.01), moving within noise from 1× to 3× (p=0.40) and lifting ~4 pp
+  from 3× to uncapped — because generous headroom lets the agent attempt strategies a tight
+  cap forbids. Two caps measure two agents. The remedy is a floor *and* a ceiling.
+- A ladder for what a given piece of evidence authorises next, ending on the rule people
+  skip: **4/4 on a slice is not 100% system-wide.**
+
+**§5's trajectory rule moved between two measurements rather than being deleted.** It read
+*"Judge the trajectory, not just the answer. Right tools, right order, right arguments."*
+Anthropic calls exact tool-order assertions *"too rigid … agents regularly find valid
+approaches that eval designers didn't anticipate"*, with a worked case of an agent that
+solved a τ²-bench booking task through a policy loophole and failed the eval while serving
+the user better. But the opposite edge is measured too: a grader blind to the trajectory
+misses **44% of safety violations and 13% of robustness failures**. So the rule now reads
+*read the trajectory; do not match it* — assert what was produced and what changed, and use
+the trajectory as a **set and a forbidden list** for the claims an outcome cannot carry.
+§2's axis table lost its `→` sequence example for the same reason.
+
+The single-step example keeps its *"must call `find_meeting_times` first"*, with a sentence
+saying why: at that granularity the fixture **is** one decision, so ordering is the subject
+rather than a proxy for it. Across a trajectory it stops being one.
+
 # Changelog
 
 ## v0.17.1 — the evals run for the first time, and the tails go to zero

docs/evidence/backlog.md

@@ -38,3 +38,4 @@ row per line, appended, so two runs closing two rows do not land on the same lin
 | AG-06b | **The 2592 existing trees are reported and left alone.** `find "$TMPDIR" -maxdepth 4 -type f -path '*/copy/sub/b.sh' \\| wc -l` → **2592** on 2026-08-20, of which **16 were added by this run's own verification** — one run of the leaking fixture before the port, and one of the v0.12.0 archive's copy of it. They are plain `tmpXXXXXXXX` and indistinguishable from every other program's, so they are named rather than swept. The identical fixture still leaks in **`seo-aeo-audit`**, which another agent holds; referred, not edited. | Open by decision: state this run cannot prove is its own is reported, never deleted. |
 | B-118 (umbrella) | **The committed social card clips its eyebrow at the canvas.** The 91-char eyebrow line needs ~1270px at its set size against the 1200px canvas — roughly the last 11 characters are lost, "LLM RESALE" among them (measured by the umbrella's v1.11.0 run, 2026-08-31; the finding's home is the umbrella board row B-118). `test/social_preview.py` cannot see this by design: it checks the PNG signature, the under-1MB limit and the 1200x630 dimensions — bytes, never rendered text — so the gate stayed green over a clipped card and will stay green over the fix. | Open by instruction: the repair is a role-cell shortening that must be coordinated across the family's cards, so it was deliberately NOT folded into v0.17.1 (coordinator decision, 2026-08-31). Fix lands when the umbrella coordinates the wording. |
 | AG-04a | **This board's own priority is a scalar.** The header computes `P = blast × (1 + age_runs) / effort` while `audit.md` now argues that publishing axes beats multiplying them. The two are not the same claim — the audit is doctrine this pack ships to users about somebody else's system, the board is one repository's own triage — but the tension is real and it is not resolved by saying so. The method's home is the umbrella board (`sshlg-skills/docs/evidence/backlog.md`), which another agent holds; the check added here deliberately scopes to documents under `plugins/`, so it neither polices nor blesses this file. Referred to the umbrella. | Open: the board's formula and `audit.md` disagree, and neither has been chosen. |
+| B-124 | **`agent-evals` had no `references/` directory and no statistics layer: 315 lines saying what to assert, nothing saying how many times.** A skill whose job is answering *did it get better* could not distinguish a result from noise — `grep -niE 'pass@|pass\^|variance|significance|seeds?'` over SKILL.md returned one hit, about escalation. Separately, §5's *"Right tools, right order"* is the exact practice Anthropic calls *"too rigid"*. | 2026-08-31 harness-engineering harvest; the statistics gap was reported independently by **four** of the twenty agents | 3 | 0 | 2 | **1.5** | **closed 2026-08-31, shipped in v0.18.0** — `references/statistics.md`: the noise band with its table, `pass@k` vs `pass^k` and the 91-point gap, the τ-bench data showing trials are correlated so `pass^k` cannot be exponentiated, paired comparison with 3–5 seeds, the harness as a 6 pp variable with a floor AND a ceiling, and the evidence ladder ending on *4/4 on a slice is not 100% system-wide*. Every figure recomputed before it was written; the correlation finding is ours, from recomputing rather than restating. §5 moved **between** its two measured edges — Anthropic's *too rigid* and Claw-Eval's 44% of safety violations missed by trajectory-blind grading — rather than flipping to the other extreme. |

docs/evidence/verification.md

@@ -10,6 +10,23 @@ This file exists because its absence read as zero exposure. `sshlg-skills` board
 
 ---
 
+## Shipped state — v0.18.0 (2026-08-31)
+
+Measured on the release-candidate tree before the tag exists. B-124 from the
+2026-08-31 harness-engineering harvest: the skill had no `references/` directory and no
+statistics layer at all.
+
+| REQ | What ships | How it was confirmed | Confirmed |
+|---|---|---|---|
+| ST-1 | `agent-evals` gains its first reference, `references/statistics.md`, and the gate's two-way check binds it | `ls plugins/agent-stack/skills/agent-evals/references/` → `statistics.md`; `npm test` → rc=0 with `validate.py`'s references↔SKILL.md agreement passing in both directions (a link to a missing file and an unlinked orphan both fail it) | **observed** |
+| ST-2 | Every figure in the reference is computed, not restated | recomputed before writing: `SE(0.70,100)=0.0458` → **±8.98 pp**, and ±4.49 / ±2.84 pp at n=400 / n=1000; `pass@5 = 1−0.4⁵ = 0.98976`; `pass^5 = 0.6⁵ = 0.07776`, a **91.2-point** gap; `0.75³ = 0.421875` | **observed** |
+| ST-3 | The independence assumption behind `pass^k` is stated as false, with the published data that falsifies it | τ-bench airline Pass^k for claude-3-5-sonnet is 0.460 / 0.326 / 0.263 / 0.225; `0.460^k` gives 0.460 / 0.212 / 0.097 / 0.045 — the observed curve falls far slower, so trials are positively correlated and `pass^k` cannot be exponentiated from `pass^1`. **Not in the source that prompted this**; it came out of recomputing the table | **observed** |
+| ST-4 | §5's trajectory rule sits between both measured edges rather than being deleted | `grep -c "Right tools, right order" SKILL.md` → **0**; the section now carries Anthropic's *"too rigid"* quote AND Claw-Eval's 44%/13% for trajectory-blind grading, resolving to *read the trajectory; do not match it* — a set and a forbidden list, never an order | **observed** |
+| ST-5 | §2's axis table no longer prescribes an order matcher, and the one place order IS the subject says so | the Trajectory row reads *"what the run must not do, and what it must have touched — never the order"*; the single-step example keeps *"must call `find_meeting_times` first"* with a sentence explaining that at that granularity the fixture is one decision | **observed** |
+| ST-6 | The checklist gains the line that makes the reference reachable from the gate a reader runs | `grep -c "statistics.md" SKILL.md` → 2 (checklist line + References table); body ~4343 tokens, inside the 4750 working limit | **observed** |
+
+---
+
 ## Shipped state — v0.17.1 (2026-08-31)
 
 Measured on the release-candidate tree before the tag exists. Wave-3 of the 2026-08-29

package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ssheleg/agent-stack",
-  "version": "0.17.1",
+  "version": "0.18.0",
   "scripts": {
     "test": "python3 test/validate.py && python3 test/plant_guard_test.py && node test/installer_test.js"
   },

plugins/agent-stack/.claude-plugin/plugin.json

@@ -3,7 +3,7 @@
   "name": "agent-stack",
   "displayName": "Agent Stack",
   "description": "Four skills: agent-orchestrator — tool-calling loops, pipelines with checkpoints, provider routing with fallback, memory architecture, plus the wallet side of reselling LLM access; agent-evals — run/trace/thread evals, LLM judges, and fixtures grown from production; agent-interop — MCP servers and clients, A2A agent cards, the MCP Registry, and gateways; agent-harness — system prompts, tool shaping, workflow-vs-agent, and auditing an agent system.",
-  "version": "0.17.1",
+  "version": "0.18.0",
   "author": {
     "name": "ssheleg",
     "url": "https://x.com/sshlg93"

plugins/agent-stack/skills/agent-evals/SKILL.md

@@ -63,6 +63,10 @@ point — tool name, argument shape.
 > "Schedule a meeting with Harrison tomorrow morning", with `find_meeting_times`,
 > `schedule_meeting` and `send_email` available, must call `find_meeting_times` first.
 
+This is the one granularity where *first* is a legitimate assertion: the fixture is a
+single decision, so the ordering claim is the subject rather than a proxy for it. Across a
+whole trajectory it stops being one — see §5.
+
 Cheap, deterministic, CI-blocking. **Precondition: a stable agent architecture.** These
 break on a graph refactor, and a suite that fails on every refactor gets deleted.
 
@@ -72,7 +76,7 @@ Assert on three axes at once, with three different mechanisms:
 
 | Axis | Assert | With |
 |---|---|---|
-| Trajectory | tool-call sequence — `read_file` → `edit_file` → `run_tests` | set/subset/order matchers |
+| Trajectory | what the run **must not** do, and what it must have touched — never the order | set/subset matchers, forbidden-call lists |
 | Final response | quality, tone, policy compliance | rubric or judge |
 | **State change** | the memory row exists, the file was written, the artifact is there | direct inspection of the side effect |
 
@@ -147,8 +151,20 @@ That is what makes a judge reproducible and a disagreement resolvable.
 assertions, tool-call correctness — all deterministic, all faster and cheaper than a model
 call. Send to a judge only what cannot be decided by code.
 
-**Judge the trajectory, not just the answer.** Right tools, right order, right arguments.
-An agent that reaches a correct answer through three wrong tool calls is a latent outage.
+**Read the trajectory; do not match it.** An agent that reaches a correct answer through
+three wrong tool calls is a latent outage — and asserting the *sequence* to catch that is
+measurably the wrong instrument. Anthropic names the instinct and rejects its strict form:
+exact tool-order assertions are *"too rigid and results in overly brittle tests, as agents
+regularly find valid approaches that eval designers didn't anticipate"*, and the worked
+case is an agent that solved a τ²-bench booking task through a policy loophole, failing
+the eval as written while serving the user better. Grade **what was produced and what
+changed**, and let the path vary.
+
+The opposite edge is measured too, so this is not "grade the final answer": a grader blind
+to the trajectory misses **44% of safety violations and 13% of robustness failures**,
+because a policy breach on the way to a correct result leaves no trace in the outcome. Use
+the trajectory for the claims the outcome cannot carry — a forbidden call, a missing
+confirmation, a secret read — as a **set and a forbidden list**, never as an order.
 
 **Calibrate the judge before trusting it.** Collect human labels on the same traces,
 measure agreement, iterate the judge prompt until agreement is high — *then* let it score
@@ -304,6 +320,16 @@ None of the above runs without these, and they are the part people skip:
 - [ ] Every production failure minimised into a permanent fixture
 - [ ] Annotation queue with filters, and the two reviewer roles kept separate
 - [ ] Simulated users trained on real transcripts, with adversarial personas
+- [ ] Every comparison that changes a decision states `n`, `k`, the reducer and the noise
+      band — `references/statistics.md`; a three-point gap on a hundred cases is noise
+
+---
+
+## References
+
+| Load | When |
+|---|---|
+| [`references/statistics.md`](references/statistics.md) | a number is about to change a decision — how many runs before a difference is real, `pass@k` vs `pass^k` and why trials are not independent, paired comparison, the harness as a variable, and what a given piece of evidence authorises next |
 
 ---