From 02d31b5795adc729a510491e1eb0cac5d443e6e2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=D0=A0=D0=BE=D0=B1=D0=BE=D1=82?= Date: Tue, 1 Sep 2026 11:01:50 +0200 Subject: [PATCH] feat(evals): the wire format for section 7, and the three things it will not carry MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Section 7 said what to instrument and named no format; a grep for opentelemetry over the pack returned one table cell. references/otel-genai.md is the format, and it earns its own file on the refusals rather than the field list. - The conventions moved repositories, their Schema URL reads TODO, and every gen_ai.* attribute is Development. Adopt and pin; treat a branch on any of those names as code with an expiry date. - gen_ai.evaluation.result has no attribute for who produced the score — the field section 7 requires by name. OpenInference has it as annotation. annotator_kind with our three values; carry it as a documented extension rather than drop the requirement. - There is no cost attribute, and usage is eleven numbers. A cost from input + output is wrong in both directions: it bills cache reads at full price and misses reasoning tokens and cache writes. - The content-upload hook runs independently of the opt-in flags AND of the sampling decision, so it is the last scrubbing point and fires on spans nobody reads; a masking hook cannot change the span name. Checked against a shipping implementation: OpenLLMetry captures content when its env var is unset, the inverse of the spec's default. - "OpenTelemetry-based" is not one vocabulary, and "replay" is three guarantees. B-134, OT-1..8. Gate: npm test rc=0. Body 4631/4750. Co-Authored-By: Claude Opus 5 (1M context) --- .claude-plugin/marketplace.json | 2 +- CHANGELOG.md | 49 +++++ docs/evidence/backlog.md | 1 + docs/evidence/verification.md | 18 ++ package.json | 2 +- .../agent-stack/.claude-plugin/plugin.json | 2 +- .../agent-stack/skills/agent-evals/SKILL.md | 5 + .../agent-evals/references/otel-genai.md | 177 ++++++++++++++++++ 8 files changed, 253 insertions(+), 3 deletions(-) create mode 100644 plugins/agent-stack/skills/agent-evals/references/otel-genai.md diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index 6f7e567..e72fcae 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -12,7 +12,7 @@ "displayName": "Agent Stack", "source": "./plugins/agent-stack", "description": "Four skills: agent-orchestrator — tool-calling loops, pipelines with checkpoints, provider routing with fallback, memory architecture, plus the wallet side of reselling LLM access; agent-evals — run/trace/thread evals, LLM judges, and fixtures grown from production; agent-interop — MCP servers and clients, A2A agent cards, the MCP Registry, and gateways; agent-harness — system prompts, tool shaping, workflow-vs-agent, and auditing an agent system.", - "version": "0.22.0", + "version": "0.23.0", "author": { "name": "ssheleg", "url": "https://x.com/sshlg93" diff --git a/CHANGELOG.md b/CHANGELOG.md index 210e780..cad1975 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,52 @@ +## v0.23.0 — the wire format for §7, and the three things it will not carry + +`agent-evals` §7 said *what* to instrument in four bullets and named no format. +`references/otel-genai.md` is the format — and the reason it is worth its own file is not +the field list but the three places the standard refuses to carry what this skill requires. + +**Read the status before encoding any of it.** The conventions have moved out of the main +`semantic-conventions` repository into their own, whose README lists **Schema URL: `TODO`**, +and every document is `Status: Development`. On the inference span the only `Stable` +attributes are the ones borrowed from core semconv; **every `gen_ai.*` attribute is +Development**. Adopt it anyway — a moving standard beats a private vocabulary nobody else's +tooling reads — but pin the version and treat any branch on a `gen_ai.*` name as code with +an expiry date. + +**The evaluation event has no field for who produced the score.** `gen_ai.evaluation.result` +carries a name, a value, a low-cardinality label and an explanation, and nothing about the +scorer — while §7 requires exactly that (`human` | `llm_judge` | `code_check`), because a +score with no source cannot be calibrated or audited. OpenInference has the field under +another name, `annotation.annotator_kind`, with our three values. The reference says to carry +it as a documented extension rather than drop the requirement: a judge score and a human +label that are indistinguishable on the wire get averaged by somebody downstream, which is +what §5's calibration exists to prevent. + +**There is no cost attribute at all**, and usage is eleven numbers rather than one — +`input_tokens`, `output_tokens`, `reasoning.output_tokens`, `cache_read.input_tokens`, +`cache_write.input_tokens`, plus per-modality splits. **A cost from `input + output` alone is +wrong in both directions**: it bills cache reads at full price and misses reasoning tokens +and cache writes entirely. `gen_ai.client.token.usage` carries a hard **MUST NOT report** +when counts are unobtainable — absence over a fabricated zero, the rule `audit.md` already +states for attribution. + +**And two seams that decide a design rather than a field.** The content-upload hook *"SHOULD +operate independently of the opt-in flags"* and *"SHOULD be invoked regardless of the span +sampling decision"* — so it is the last scrubbing point and it fires on spans nobody will +read; a masking hook meanwhile **cannot change the span name**, which is built from tool and +MCP target names, so redaction and naming are one decision. Verified against a shipping +implementation rather than assumed: OpenLLMetry's `is_content_tracing_enabled()` returns +**true when its environment variable is unset**, the opposite of the spec's default, so +prompts are captured unless you turn them off (`traceloop-sdk/traceloop/sdk/config/__init__.py`, +read 2026-08-31). + +**Finally, two words that mean three things each.** *"OpenTelemetry-based"* answers a +transport question and no semantic one: Phoenix emits OpenInference, OpenLLMetry emits +`gen_ai.*` plus `traceloop.*`, neither is a subset of the other, and a trace assembled from +both is two disconnected halves under one trace id. And *"replay"* is durable execution +(recorded results, nothing re-runs), a trace playground (the call re-runs against the live +provider), or this skill's fixture replay (an assertion over a stored run) — `audit.md` asks +whether a run can be replayed without saying which, and the reference names the three. + ## v0.22.0 — the failures named from outside, and the control that is absence **`graph-engineering.md` was argued entirely from what breaks, with no citation behind it.** diff --git a/docs/evidence/backlog.md b/docs/evidence/backlog.md index 432e0ad..67ff637 100644 --- a/docs/evidence/backlog.md +++ b/docs/evidence/backlog.md @@ -38,6 +38,7 @@ row per line, appended, so two runs closing two rows do not land on the same lin | B-131 | **We shipped `iteration refund` and nothing said the money is not refunded with it.** A recoverable provider error does not consume an attempt — correct — but the provider still billed the call, and a response that arrived and then failed to parse was generated, metered and charged. Cost accounting written on the success path under-counts on exactly the runs that go worst. | 2026-08-31 harness-engineering harvest, queue position 6 (ARCH) | 2 | 0 | 1 | **2.0** | **closed 2026-08-31, shipped in v0.21.0** — `llm-proxy-billing.md` gains *Refund the iteration, charge the money* with the observed failure (cost added after a successful parse, compensated inside the exception handler, in a harness whose `cost_limit` is the primary bound with the step limit off), the `finally` rule, and why under-counting is the worse asymmetry. §2's bullet now carries the pairing in one clause. | | B-132 | **`graph-engineering.md` argued every rule from failure and cited nothing.** The failures were right and unattributed, so a reader had no way to weigh them against their own experience, and the file had no answer to *why not just write better prompts*. | 2026-08-31 harness-engineering harvest, queue position 7 (ARCH) | 1 | 0 | 1 | **1.0** | **closed 2026-08-31, shipped in v0.22.0** — §11a carries MAST: 14 modes in three groups from ~150 traces across seven frameworks at Cohen's kappa 0.88, mapped onto the five-field node contract (interface and owner, the edge payload, the check). The answer to *better prompts* is the paper's negative result: patching bought ChatDev **15.6%**, and the modes are architectural. Written as a per-node review checklist. | | B-133 | **`governance.md`'s four-boundary table put per-tool authorisation at the moment of invocation, and never named the stronger control one layer earlier.** Schema gating was §3's rule and argued as assembly hygiene rather than as safety; nothing said a tool absent from the schema cannot be argued for or probed, that sub-agent isolation needs `message_history = None` beside the filtering, or that an approval system without persistence converts itself into no approval system. | 2026-08-31 harness-engineering harvest, queue position 8 (ARCH) | 2 | 0 | 1 | **2.0** | **closed 2026-08-31, shipped in v0.22.0** — *The cheapest control is absence*, with both isolation mechanisms, the 40%-to-under-5% context figures, and the rule underneath them (**eager for what you own and always need, lazy for what is foreign and might not be used**). Approval fatigue filed as a safety failure that leaves clean logs. | +| B-134 | **§7 named four things to instrument and no format to carry them, and the obvious standard omits two of the four.** `grep -ri opentelemetry` over the pack returned one table cell in `agent-interop/references/mcp.md`. Meanwhile `gen_ai.evaluation.result` has no attribute for who produced a score — the field §7 requires by name — and the conventions carry no cost attribute at all while splitting usage across eleven token fields. | 2026-08-31 harness-engineering harvest, observability bundle (10 findings, one source) | 2 | 0 | 2 | **1.0** | **closed 2026-09-01, shipped in v0.23.0** — `references/otel-genai.md`: the span-name formulas and closed 17-value operation enum, the two Required attributes that make conformance a weak floor, the missing scorer field with OpenInference's `annotation.annotator_kind` as the extension to carry, the three content tiers and the upload hook that runs independently of opt-in **and** of the sampling decision, the eleven token attributes with the both-directions cost error, the disjoint OpenInference/OpenLLMetry vocabularies under one trace id, and the three senses of *replay*. Status recorded honestly: the spec moved repositories, its Schema URL reads `TODO`, and every `gen_ai.*` attribute is Development. | ## Open, and why diff --git a/docs/evidence/verification.md b/docs/evidence/verification.md index aaccc91..9a1352c 100644 --- a/docs/evidence/verification.md +++ b/docs/evidence/verification.md @@ -10,6 +10,24 @@ This file exists because its absence read as zero exposure. `sshlg-skills` board --- +## Shipped state — v0.23.0 (2026-09-01) + +Measured on the release-candidate tree before the tag exists. The observability bundle's +ten findings, which had one home between them and no coverage here. + +| REQ | What ships | How it was confirmed | Confirmed | +|---|---|---|---| +| OT-1 | §7 gains the wire format it never named | before: `grep -ri opentelemetry plugins/agent-stack/skills/` → one table cell in `agent-interop/references/mcp.md`; after: `references/otel-genai.md` plus a declared row and a pointer from §7 | **observed** | +| OT-2 | The spec's status is recorded rather than assumed | the file states the move to a separate repository, `Schema URL: TODO`, and that **every** `gen_ai.*` attribute is Development while only borrowed core attributes are Stable — with the instruction to pin and expect migration | **observed** | +| OT-3 | The field §7 requires and the standard omits is named, with its extension | `gen_ai.evaluation.result` carries no scorer attribute; OpenInference's `annotation.annotator_kind` (HUMAN/LLM/CODE) matches SKILL.md:296's `human`/`llm_judge`/`code_check` exactly, and the file says carry it as a documented extension rather than drop the requirement | **observed** | +| OT-4 | The content default is stated **and checked against a shipping implementation** | the spec's own default is to record nothing; OpenLLMetry's `is_content_tracing_enabled()` returns `(os.getenv("TRACELOOP_TRACE_CONTENT") or "true")` — unset means **true** — read from `traceloop-sdk/traceloop/sdk/config/__init__.py` on 2026-08-31, so the most-used implementation ships the inverse of the spec | **observed** | +| OT-5 | The upload hook's two properties are carried, because they decide the design | it runs independently of the opt-in flags **and** regardless of the sampling decision, so it is the last scrubbing point and fires on spans nobody reads; and a masking hook cannot change the span name, which is built from tool and MCP target names | **observed** | +| OT-6 | Cost is stated as a join that goes wrong in both directions | no cost attribute exists; usage splits across eleven token fields; `input + output` bills cache reads at full price and misses reasoning tokens and cache writes; `gen_ai.client.token.usage` carries a **MUST NOT report** when counts are unobtainable, matching `audit.md`'s absence-over-zero rule | **observed** | +| OT-7 | Two overloaded words are disambiguated | *OpenTelemetry-based* — Phoenix emits OpenInference, OpenLLMetry emits `gen_ai.*` plus `traceloop.*`, neither a subset of the other; *replay* — durable execution, trace playground, and this skill's fixture replay, which `audit.md` had asked for without naming | **observed** | +| OT-8 | The body stayed inside the working limit | auditor `0 GAP, 14 PASS`, body **4631/4750**; the finding went into a reference and one pointer sentence | **observed** | + +--- + ## Shipped state — v0.22.0 (2026-08-31) Measured on the release-candidate tree before the tag exists. Queue positions 7–8, both ARCH. diff --git a/package.json b/package.json index b916b22..ed149b3 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@ssheleg/agent-stack", - "version": "0.22.0", + "version": "0.23.0", "scripts": { "test": "python3 test/validate.py && python3 test/plant_guard_test.py && node test/installer_test.js" }, diff --git a/plugins/agent-stack/.claude-plugin/plugin.json b/plugins/agent-stack/.claude-plugin/plugin.json index 41e913d..f3625fe 100644 --- a/plugins/agent-stack/.claude-plugin/plugin.json +++ b/plugins/agent-stack/.claude-plugin/plugin.json @@ -3,7 +3,7 @@ "name": "agent-stack", "displayName": "Agent Stack", "description": "Four skills: agent-orchestrator — tool-calling loops, pipelines with checkpoints, provider routing with fallback, memory architecture, plus the wallet side of reselling LLM access; agent-evals — run/trace/thread evals, LLM judges, and fixtures grown from production; agent-interop — MCP servers and clients, A2A agent cards, the MCP Registry, and gateways; agent-harness — system prompts, tool shaping, workflow-vs-agent, and auditing an agent system.", - "version": "0.22.0", + "version": "0.23.0", "author": { "name": "ssheleg", "url": "https://x.com/sshlg93" diff --git a/plugins/agent-stack/skills/agent-evals/SKILL.md b/plugins/agent-stack/skills/agent-evals/SKILL.md index 8c201a6..3a8aa7d 100644 --- a/plugins/agent-stack/skills/agent-evals/SKILL.md +++ b/plugins/agent-stack/skills/agent-evals/SKILL.md @@ -288,6 +288,10 @@ predictive. ## 7. What to instrument first +**The wire format for all of this is `references/otel-genai.md`** — and it is worth +reading before the first span, because two of the things this section requires (a score's +source, a trustworthy cost) are the two the standard does not carry. + None of the above runs without these, and they are the part people skip: - **A durable trace store, queryable by id, filterable by score and time.** The live @@ -330,6 +334,7 @@ None of the above runs without these, and they are the part people skip: | Load | When | |---|---| | [`references/statistics.md`](references/statistics.md) | a number is about to change a decision — how many runs before a difference is real, `pass@k` vs `pass^k` and why trials are not independent, paired comparison, the harness as a variable, and what a given piece of evidence authorises next | +| [`references/otel-genai.md`](references/otel-genai.md) | you are **instrumenting an agent for someone else to read** — span-name formulas and the closed operation enum, the evaluation event that has no field for who scored, the three content tiers and the upload hook that runs even on dropped spans, the eleven token attributes and why cost is a join you can get wrong in both directions, and why "OpenTelemetry-based" is not one vocabulary | --- diff --git a/plugins/agent-stack/skills/agent-evals/references/otel-genai.md b/plugins/agent-stack/skills/agent-evals/references/otel-genai.md new file mode 100644 index 0000000..5a2ba8b --- /dev/null +++ b/plugins/agent-stack/skills/agent-evals/references/otel-genai.md @@ -0,0 +1,177 @@ +# OpenTelemetry GenAI — the wire format for §7, and what it does not carry + +**Load this when instrumenting an agent for someone else to read**: choosing span names and +attributes, deciding what of a prompt reaches a backend, wiring token usage into a bill, or +judging what an "OpenTelemetry-based" tool actually gives you. + +§7 says *what* to instrument in four bullets. This is the format that carries it — and the +three places it will not carry what this skill requires. + +## Contents + +- [Read this before encoding any of it](#read-this-before-encoding-any-of-it) +- [Span names are formulas, and the operation list is closed](#span-names-are-formulas-and-the-operation-list-is-closed) +- [The evaluation event, missing the field §7 requires](#the-evaluation-event-missing-the-field-7-requires) +- [Content: three tiers, and a hook that runs when nothing else does](#content-three-tiers-and-a-hook-that-runs-when-nothing-else-does) +- [Tokens are eleven numbers and money is none of them](#tokens-are-eleven-numbers-and-money-is-none-of-them) +- ["OpenTelemetry-based" is not one vocabulary](#opentelemetry-based-is-not-one-vocabulary) +- [Replay is three different guarantees](#replay-is-three-different-guarantees) + +--- + +## Read this before encoding any of it + +**The conventions moved.** They no longer live in the main `semantic-conventions` repository +— that page now says only that they have moved and is no longer maintained. They have their +own repository, whose README lists **Schema URL: `TODO`**. + +**Nothing in them is stable.** Every document is marked `Status: Development`. On the +inference span the only attributes marked `Stable` are the ones borrowed from core semconv — +`error.type`, `server.address`, `server.port`, `exception.*`. **Every single `gen_ai.*` +attribute is `Development`.** + +So: adopt it, because a moving standard beats a private vocabulary that will never be read +by anyone else's tooling — and **pin the version you adopted and expect to migrate**. Treat +any code that branches on a `gen_ai.*` attribute as code with an expiry date, and re-read +the spec before quoting a field name from this file. + +## Span names are formulas, and the operation list is closed + +Span names are computed, not free text: + +| Span | Name | Kind | +|---|---|---| +| inference | `{gen_ai.operation.name} {gen_ai.request.model}` | `CLIENT` | +| tool call | `execute_tool {gen_ai.tool.name}` | `INTERNAL` | +| agent invocation | `invoke_agent {gen_ai.agent.name}` | `CLIENT` | +| planning | `plan {gen_ai.agent.name}` | `INTERNAL` | +| agent creation | `create_agent {gen_ai.agent.name}` | — | +| MCP | `{mcp.method.name} {target}`, target being the tool or prompt name | — | + +`gen_ai.operation.name` is a **closed 17-value enum** — `chat`, `text_completion`, +`generate_content`, `embeddings`, `retrieval`, `fetch_response`, `execute_tool`, +`create_agent`, `invoke_agent`, `plan` and the rest. A value outside it is not an extension, +it is a name a backend cannot group by. + +**Only two attributes are Required on an inference span.** Everything else that matters — +the model that actually answered, token counts, finish reasons — is Recommended or +Conditionally Required. A conformant instrumentation can therefore be almost empty, which is +the reason to specify what *you* need rather than to trust conformance as a floor. + +One behaviour worth knowing before you count spans: **automatic retries collapse into one +span.** A span is one logical operation, not one HTTP request, so a retry storm is invisible +at this layer and has to be measured somewhere else. + +## The evaluation event, missing the field §7 requires + +The spec defines `gen_ai.evaluation.result`, parented to the span being evaluated — or +carrying `gen_ai.response.id` when the span id is not available — with: + +- `gen_ai.evaluation.name` (**Required**) +- `gen_ai.evaluation.score.value` +- `gen_ai.evaluation.score.label` — low-cardinality: `pass`, `fail`, `relevant`, … +- `gen_ai.evaluation.explanation` + +**There is no attribute for who or what produced the score.** + +That is precisely the field this skill requires: §7 says a score binds to a run with a +`source` of `human` | `llm_judge` | `code_check`, *because a score with no source cannot be +calibrated, audited, or trusted differently from its neighbours*. The standard omits it. + +OpenInference has it, under a different name: **`annotation.annotator_kind`** — `HUMAN`, +`LLM`, `CODE`, or custom — with `evaluation.annotator_kind` beside it. The three values are +our three values. + +**So carry it yourself.** Add the attribute under your own namespace and document that it is +an extension; do not drop the requirement because the schema has no slot for it. A judge +score and a human label that are indistinguishable on the wire will be averaged by somebody +downstream, and that average is the thing §5's calibration exists to prevent. + +## Content: three tiers, and a hook that runs when nothing else does + +Prompt and completion content is governed by three named patterns, chosen by environment: + +1. **Record nothing** — the default. +2. **Record on span attributes** — *"best suited for … pre-production environments"*. +3. **Store externally, record a reference on the span** — *"recommended in production + environments where telemetry volume is a concern or sensitive data needs to be handled + securely. Using external storage enables separate access controls."* + +**The spec's own default is to capture nothing**, and it says instrumentations SHOULD NOT +capture content by default. Verify that in your stack rather than assuming it: the most +widely used `gen_ai.*` implementation ships the opposite. OpenLLMetry's +`is_content_tracing_enabled()` reads + +```python +return (os.getenv("TRACELOOP_TRACE_CONTENT") or "true").lower() == "true" +``` + +— an unset variable is **true**, so prompts are captured unless you turn them off. Measured +2026-08-31 against `traceloop-sdk/traceloop/sdk/config/__init__.py`. An agent's trace carries +prompts, prompts carry secrets, and the default here works against you. + +**The upload hook is the last scrubbing point, and it runs where nothing else does.** Two +properties make it the mechanism rather than a convenience: it *"SHOULD operate independently +of the opt-in flags"*, and instrumentations *"SHOULD invoke it regardless of the span sampling +decision"*. It therefore fires on spans nobody will ever look at — which is right, because +the content has already left the process by then, and wrong to reason about as *"we only keep +sampled traces"*. + +**A masking hook cannot change the span name.** Span names are built from tool names and MCP +targets, so a redaction design that scrubs attributes and leaves the name is a design that +leaks through the one field it never inspected. Decide naming and redaction together. + +## Tokens are eleven numbers and money is none of them + +**There is no cost attribute anywhere in the GenAI semantic conventions.** The spec +standardises tokens and never money, so cost is always a join against a price table living +outside the trace — and that join is where the number goes wrong. + +Because usage is not one number. It is eleven: `gen_ai.usage.input_tokens`, `output_tokens`, +`reasoning.output_tokens`, `cache_read.input_tokens`, `cache_write.input_tokens`, and +per-modality `text.*` / `image.*` / `audio.*` splits including +`image.cache_read.input_tokens`. + +**A cost computed from `input_tokens + output_tokens` alone is wrong in both directions.** It +bills cache reads at full price — they are the cheap ones — and it misses reasoning tokens and +cache writes entirely. `references/kv-cache.md` in `agent-orchestrator` is the other half of +this: the cache read is the case worth getting right, because at scale it is most of the +traffic. + +And `gen_ai.client.token.usage` carries a hard **MUST NOT report** when the counts are not +obtainable. A zero is a claim; absence is the honest value. That is the same rule +`agent-harness/references/audit.md` states for cost attribution — *missing attribution beats +wrong attribution, and absence is a named state rather than a fabricated zero*. + +## "OpenTelemetry-based" is not one vocabulary + +Two widely used tools both describe themselves as OpenTelemetry-based and emit **disjoint** +attribute sets: + +| | Emits | +|---|---| +| **Phoenix** | OpenInference — `openinference.span.kind` (required; `LLM`, `CHAIN`, `RETRIEVER`, `RERANKER`, `TOOL`, `AGENT`, `GUARDRAIL`, `EVALUATOR`, `PROMPT`, `EMBEDDING`), plus `llm.*`, `input.value`, `input.mime_type`, `document.*`, `annotation.*` | +| **OpenLLMetry** | `gen_ai.*` plus `traceloop.*` | + +**Neither set is a subset of the other, and neither is a superset of the GenAI +conventions.** A backend query written against one returns nothing against the other, and a +trace assembled from two services using each is **two disconnected halves under one trace +id** — which reads as a gap in the system rather than a gap in the vocabulary. + +So *"we use OpenTelemetry"* answers a transport question and no semantic one. Ask which +attribute set, and pick one per system rather than per service. + +## Replay is three different guarantees + +The word appears in three products in this space and means three incompatible things. Our +own `agent-harness/references/audit.md` asks *"can a past run be replayed"* without saying +which: + +| Sense | What re-runs | Cost | Answers | +|---|---|---|---| +| **Durable execution** (Temporal) | nothing — recorded results are replayed and only the failed step retries | free, deterministic | can I resume without redoing 20 web searches | +| **Trace playground** (Phoenix) | the model call, against the live provider, with an edited prompt | a real call | would a different prompt have done better | +| **Fixture replay** (this skill, §2 single-step) | an assertion over a stored run | free | did the decision at this point change | + +They are not interchangeable, and a runbook that says *"replay the run"* has not said what it +means. Name the sense.