v0.23.0 — the wire format for §7, and the three things it will not carry

.claude-plugin/marketplace.json

@@ -12,7 +12,7 @@
       "displayName": "Agent Stack",
       "source": "./plugins/agent-stack",
       "description": "Four skills: agent-orchestrator — tool-calling loops, pipelines with checkpoints, provider routing with fallback, memory architecture, plus the wallet side of reselling LLM access; agent-evals — run/trace/thread evals, LLM judges, and fixtures grown from production; agent-interop — MCP servers and clients, A2A agent cards, the MCP Registry, and gateways; agent-harness — system prompts, tool shaping, workflow-vs-agent, and auditing an agent system.",
-      "version": "0.22.0",
+      "version": "0.23.0",
       "author": {
         "name": "ssheleg",
         "url": "https://x.com/sshlg93"

CHANGELOG.md

@@ -1,3 +1,52 @@
+## v0.23.0 — the wire format for §7, and the three things it will not carry
+
+`agent-evals` §7 said *what* to instrument in four bullets and named no format.
+`references/otel-genai.md` is the format — and the reason it is worth its own file is not
+the field list but the three places the standard refuses to carry what this skill requires.
+
+**Read the status before encoding any of it.** The conventions have moved out of the main
+`semantic-conventions` repository into their own, whose README lists **Schema URL: `TODO`**,
+and every document is `Status: Development`. On the inference span the only `Stable`
+attributes are the ones borrowed from core semconv; **every `gen_ai.*` attribute is
+Development**. Adopt it anyway — a moving standard beats a private vocabulary nobody else's
+tooling reads — but pin the version and treat any branch on a `gen_ai.*` name as code with
+an expiry date.
+
+**The evaluation event has no field for who produced the score.** `gen_ai.evaluation.result`
+carries a name, a value, a low-cardinality label and an explanation, and nothing about the
+scorer — while §7 requires exactly that (`human` | `llm_judge` | `code_check`), because a
+score with no source cannot be calibrated or audited. OpenInference has the field under
+another name, `annotation.annotator_kind`, with our three values. The reference says to carry
+it as a documented extension rather than drop the requirement: a judge score and a human
+label that are indistinguishable on the wire get averaged by somebody downstream, which is
+what §5's calibration exists to prevent.
+
+**There is no cost attribute at all**, and usage is eleven numbers rather than one —
+`input_tokens`, `output_tokens`, `reasoning.output_tokens`, `cache_read.input_tokens`,
+`cache_write.input_tokens`, plus per-modality splits. **A cost from `input + output` alone is
+wrong in both directions**: it bills cache reads at full price and misses reasoning tokens
+and cache writes entirely. `gen_ai.client.token.usage` carries a hard **MUST NOT report**
+when counts are unobtainable — absence over a fabricated zero, the rule `audit.md` already
+states for attribution.
+
+**And two seams that decide a design rather than a field.** The content-upload hook *"SHOULD
+operate independently of the opt-in flags"* and *"SHOULD be invoked regardless of the span
+sampling decision"* — so it is the last scrubbing point and it fires on spans nobody will
+read; a masking hook meanwhile **cannot change the span name**, which is built from tool and
+MCP target names, so redaction and naming are one decision. Verified against a shipping
+implementation rather than assumed: OpenLLMetry's `is_content_tracing_enabled()` returns
+**true when its environment variable is unset**, the opposite of the spec's default, so
+prompts are captured unless you turn them off (`traceloop-sdk/traceloop/sdk/config/__init__.py`,
+read 2026-08-31).
+
+**Finally, two words that mean three things each.** *"OpenTelemetry-based"* answers a
+transport question and no semantic one: Phoenix emits OpenInference, OpenLLMetry emits
+`gen_ai.*` plus `traceloop.*`, neither is a subset of the other, and a trace assembled from
+both is two disconnected halves under one trace id. And *"replay"* is durable execution
+(recorded results, nothing re-runs), a trace playground (the call re-runs against the live
+provider), or this skill's fixture replay (an assertion over a stored run) — `audit.md` asks
+whether a run can be replayed without saying which, and the reference names the three.
+
 ## v0.22.0 — the failures named from outside, and the control that is absence
 
 **`graph-engineering.md` was argued entirely from what breaks, with no citation behind it.**

docs/evidence/backlog.md

@@ -38,6 +38,7 @@ row per line, appended, so two runs closing two rows do not land on the same lin
 | B-131 | **We shipped `iteration refund` and nothing said the money is not refunded with it.** A recoverable provider error does not consume an attempt — correct — but the provider still billed the call, and a response that arrived and then failed to parse was generated, metered and charged. Cost accounting written on the success path under-counts on exactly the runs that go worst. | 2026-08-31 harness-engineering harvest, queue position 6 (ARCH) | 2 | 0 | 1 | **2.0** | **closed 2026-08-31, shipped in v0.21.0** — `llm-proxy-billing.md` gains *Refund the iteration, charge the money* with the observed failure (cost added after a successful parse, compensated inside the exception handler, in a harness whose `cost_limit` is the primary bound with the step limit off), the `finally` rule, and why under-counting is the worse asymmetry. §2's bullet now carries the pairing in one clause. |
 | B-132 | **`graph-engineering.md` argued every rule from failure and cited nothing.** The failures were right and unattributed, so a reader had no way to weigh them against their own experience, and the file had no answer to *why not just write better prompts*. | 2026-08-31 harness-engineering harvest, queue position 7 (ARCH) | 1 | 0 | 1 | **1.0** | **closed 2026-08-31, shipped in v0.22.0** — §11a carries MAST: 14 modes in three groups from ~150 traces across seven frameworks at Cohen's kappa 0.88, mapped onto the five-field node contract (interface and owner, the edge payload, the check). The answer to *better prompts* is the paper's negative result: patching bought ChatDev **15.6%**, and the modes are architectural. Written as a per-node review checklist. |
 | B-133 | **`governance.md`'s four-boundary table put per-tool authorisation at the moment of invocation, and never named the stronger control one layer earlier.** Schema gating was §3's rule and argued as assembly hygiene rather than as safety; nothing said a tool absent from the schema cannot be argued for or probed, that sub-agent isolation needs `message_history = None` beside the filtering, or that an approval system without persistence converts itself into no approval system. | 2026-08-31 harness-engineering harvest, queue position 8 (ARCH) | 2 | 0 | 1 | **2.0** | **closed 2026-08-31, shipped in v0.22.0** — *The cheapest control is absence*, with both isolation mechanisms, the 40%-to-under-5% context figures, and the rule underneath them (**eager for what you own and always need, lazy for what is foreign and might not be used**). Approval fatigue filed as a safety failure that leaves clean logs. |
+| B-134 | **§7 named four things to instrument and no format to carry them, and the obvious standard omits two of the four.** `grep -ri opentelemetry` over the pack returned one table cell in `agent-interop/references/mcp.md`. Meanwhile `gen_ai.evaluation.result` has no attribute for who produced a score — the field §7 requires by name — and the conventions carry no cost attribute at all while splitting usage across eleven token fields. | 2026-08-31 harness-engineering harvest, observability bundle (10 findings, one source) | 2 | 0 | 2 | **1.0** | **closed 2026-09-01, shipped in v0.23.0** — `references/otel-genai.md`: the span-name formulas and closed 17-value operation enum, the two Required attributes that make conformance a weak floor, the missing scorer field with OpenInference's `annotation.annotator_kind` as the extension to carry, the three content tiers and the upload hook that runs independently of opt-in **and** of the sampling decision, the eleven token attributes with the both-directions cost error, the disjoint OpenInference/OpenLLMetry vocabularies under one trace id, and the three senses of *replay*. Status recorded honestly: the spec moved repositories, its Schema URL reads `TODO`, and every `gen_ai.*` attribute is Development. |
 
 ## Open, and why
 

docs/evidence/verification.md

@@ -10,6 +10,24 @@ This file exists because its absence read as zero exposure. `sshlg-skills` board
 
 ---
 
+## Shipped state — v0.23.0 (2026-09-01)
+
+Measured on the release-candidate tree before the tag exists. The observability bundle's
+ten findings, which had one home between them and no coverage here.
+
+| REQ | What ships | How it was confirmed | Confirmed |
+|---|---|---|---|
+| OT-1 | §7 gains the wire format it never named | before: `grep -ri opentelemetry plugins/agent-stack/skills/` → one table cell in `agent-interop/references/mcp.md`; after: `references/otel-genai.md` plus a declared row and a pointer from §7 | **observed** |
+| OT-2 | The spec's status is recorded rather than assumed | the file states the move to a separate repository, `Schema URL: TODO`, and that **every** `gen_ai.*` attribute is Development while only borrowed core attributes are Stable — with the instruction to pin and expect migration | **observed** |
+| OT-3 | The field §7 requires and the standard omits is named, with its extension | `gen_ai.evaluation.result` carries no scorer attribute; OpenInference's `annotation.annotator_kind` (HUMAN/LLM/CODE) matches SKILL.md:296's `human`/`llm_judge`/`code_check` exactly, and the file says carry it as a documented extension rather than drop the requirement | **observed** |
+| OT-4 | The content default is stated **and checked against a shipping implementation** | the spec's own default is to record nothing; OpenLLMetry's `is_content_tracing_enabled()` returns `(os.getenv("TRACELOOP_TRACE_CONTENT") or "true")` — unset means **true** — read from `traceloop-sdk/traceloop/sdk/config/__init__.py` on 2026-08-31, so the most-used implementation ships the inverse of the spec | **observed** |
+| OT-5 | The upload hook's two properties are carried, because they decide the design | it runs independently of the opt-in flags **and** regardless of the sampling decision, so it is the last scrubbing point and fires on spans nobody reads; and a masking hook cannot change the span name, which is built from tool and MCP target names | **observed** |
+| OT-6 | Cost is stated as a join that goes wrong in both directions | no cost attribute exists; usage splits across eleven token fields; `input + output` bills cache reads at full price and misses reasoning tokens and cache writes; `gen_ai.client.token.usage` carries a **MUST NOT report** when counts are unobtainable, matching `audit.md`'s absence-over-zero rule | **observed** |
+| OT-7 | Two overloaded words are disambiguated | *OpenTelemetry-based* — Phoenix emits OpenInference, OpenLLMetry emits `gen_ai.*` plus `traceloop.*`, neither a subset of the other; *replay* — durable execution, trace playground, and this skill's fixture replay, which `audit.md` had asked for without naming | **observed** |
+| OT-8 | The body stayed inside the working limit | auditor `0 GAP, 14 PASS`, body **4631/4750**; the finding went into a reference and one pointer sentence | **observed** |
+
+---
+
 ## Shipped state — v0.22.0 (2026-08-31)
 
 Measured on the release-candidate tree before the tag exists. Queue positions 7–8, both ARCH.

package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ssheleg/agent-stack",
-  "version": "0.22.0",
+  "version": "0.23.0",
   "scripts": {
     "test": "python3 test/validate.py && python3 test/plant_guard_test.py && node test/installer_test.js"
   },

plugins/agent-stack/.claude-plugin/plugin.json

@@ -3,7 +3,7 @@
   "name": "agent-stack",
   "displayName": "Agent Stack",
   "description": "Four skills: agent-orchestrator — tool-calling loops, pipelines with checkpoints, provider routing with fallback, memory architecture, plus the wallet side of reselling LLM access; agent-evals — run/trace/thread evals, LLM judges, and fixtures grown from production; agent-interop — MCP servers and clients, A2A agent cards, the MCP Registry, and gateways; agent-harness — system prompts, tool shaping, workflow-vs-agent, and auditing an agent system.",
-  "version": "0.22.0",
+  "version": "0.23.0",
   "author": {
     "name": "ssheleg",
     "url": "https://x.com/sshlg93"

plugins/agent-stack/skills/agent-evals/SKILL.md

@@ -288,6 +288,10 @@ predictive.
 
 ## 7. What to instrument first
 
+**The wire format for all of this is `references/otel-genai.md`** — and it is worth
+reading before the first span, because two of the things this section requires (a score's
+source, a trustworthy cost) are the two the standard does not carry.
+
 None of the above runs without these, and they are the part people skip:
 
 - **A durable trace store, queryable by id, filterable by score and time.** The live
@@ -330,6 +334,7 @@ None of the above runs without these, and they are the part people skip:
 | Load | When |
 |---|---|
 | [`references/statistics.md`](references/statistics.md) | a number is about to change a decision — how many runs before a difference is real, `pass@k` vs `pass^k` and why trials are not independent, paired comparison, the harness as a variable, and what a given piece of evidence authorises next |
+| [`references/otel-genai.md`](references/otel-genai.md) | you are **instrumenting an agent for someone else to read** — span-name formulas and the closed operation enum, the evaluation event that has no field for who scored, the three content tiers and the upload hook that runs even on dropped spans, the eleven token attributes and why cost is a join you can get wrong in both directions, and why "OpenTelemetry-based" is not one vocabulary |
 
 ---