v0.18.0 — how many runs before a difference is real

[sshlg]

agent-evals shipped 315 lines of doctrine with no references/ directory at all, and said nothing anywhere about how many times to run anything. A skill whose entire job is answering did it get better had no way to tell a result from noise.

grep -niE 'pass@|pass\^|variance|significance|seeds?' over its SKILL.md returned one hit — about escalation, unrelated.

references/statistics.md

Every figure recomputed before it was written, never quoted:

Claim	Recomputed
SE(p)=√(p(1−p)/n), n=100, p=0.70	95% band ±8.98 pp — so 73% vs 70% on a hundred cases is inside its own noise
pass@5 vs pass^5 at p=0.6	0.98976 vs 0.07776 — a 91-point gap describing the same agent
Anthropic's 0.75³	0.421875

And one finding that is ours, from recomputing rather than restating. pass^k = p^k assumes independent trials. τ-bench's published airline curve says otherwise:

k	published Pass^k	0.460^k if independent
1	0.460	0.460
2	0.326	0.212
3	0.263	0.097
4	0.225	0.045

The observed curve falls far slower — successes cluster by task, not by trial. So pass^k cannot be computed from pass^1, and 0.75³ ≈ 42% is the right shape for an argument and the wrong number for a release gate.

Also in the file: pairing on the same tasks and 3–5 seeds (McNemar / paired bootstrap), the harness as a 6 pp variable on Terminal-Bench 2.0 needing a floor and a ceiling because two caps measure two different agents, and an evidence ladder ending on the rule people skip — 4/4 on a slice is not 100% system-wide.

§5 moved between its two measured edges

It read "Judge the trajectory, not just the answer. Right tools, right order, right arguments."

• Anthropic calls exact tool-order assertions "too rigid … agents regularly find valid approaches that eval designers didn't anticipate" — with a worked case of an agent that solved a τ²-bench booking task through a policy loophole, failing the eval while serving the user better.
• But a trajectory-blind grader misses 44% of safety violations and 13% of robustness failures.

So the rule did not flip to "grade the final answer". It now reads read the trajectory; do not match it — assert what was produced and what changed, and use the trajectory as a set and a forbidden list for the claims an outcome cannot carry. §2's axis table lost its → sequence example for the same reason.

The single-step example keeps "must call find_meeting_times first", now with a sentence saying why: at that granularity the fixture is one decision, so ordering is the subject rather than a proxy for it.

Evidence

npm test → rc=0. The gate's references↔SKILL.md two-way check binds the new file — a link to a missing reference and an unlinked orphan both fail it. Body ~4343 tokens, inside the 4750 working limit.

Board B-124. Ledger ST-1 … ST-6.

🤖 Generated with Claude Code