diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index 694bed6..1d0d3dd 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -12,7 +12,7 @@ "displayName": "Agent Stack", "source": "./plugins/agent-stack", "description": "Four skills: agent-orchestrator — tool-calling loops, pipelines with checkpoints, provider routing with fallback, memory architecture, plus the wallet side of reselling LLM access; agent-evals — run/trace/thread evals, LLM judges, and fixtures grown from production; agent-interop — MCP servers and clients, A2A agent cards, the MCP Registry, and gateways; agent-harness — system prompts, tool shaping, workflow-vs-agent, and auditing an agent system.", - "version": "0.19.1", + "version": "0.20.0", "author": { "name": "ssheleg", "url": "https://x.com/sshlg93" diff --git a/CHANGELOG.md b/CHANGELOG.md index 15e7b88..077d6c4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,39 @@ +## v0.20.0 — what a trajectory cannot carry across a vendor, and where the capability goes + +Three findings from the harvest, all landing on the same question: **which model does what, +and what survives when that changes mid-run.** §6 shipped three traps about routing, and all +three quietly assume the *request* is what moves. + +- **A trajectory carries a vendor credential, and it may not be attached to the reasoning.** + Tool calls and results are portable — different structure, same meaning, re-render and + send. Reasoning is portable *text* plus a **non-portable credential** the vendor attaches + to prove the reasoning is its own, and vendors disagree on what they demand: one end + validates nothing, the other rejects any credential it did not issue. The credential + sometimes sits on the **tool call** rather than the reasoning — which is why *"just strip + all reasoning before failing over"*, the policy that sounds safest, is the one that + produces a 400. Store trajectories in a neutral internal format, keep the text, discard + the credential, re-render per vendor at send time, and put the failover boundary + **between turns**. A fallback chain never exercised mid-trajectory has not been tested: a + green health probe answers a question about the endpoint, not about your history. +- **Capability is not spent evenly — the planner is the bottleneck.** *Plan-and-Act* + (arXiv:2503.09572) found that with good enough planning a relatively simple executor + suffices, and with a wrong decomposition every downstream executor is building on a false + premise; their 54% on WebArena-Lite came from improving the **planner**, not the executor. + So the strongest model and the most carefully written prompt go to the **manager**. It + also says where to look when a multi-agent system underperforms: **a weak plan is + invisible in every executor's transcript**, because each one did its own step correctly. +- **Steps the agent cannot see buy nothing.** Standard agents have no budget awareness, so + at **300 steps** they still plateau at roughly what they achieve at **30**. A + max-iteration guard is the floor of this rather than the mechanism — it stops the spend + and never changes the behaviour that led there. + +**And the second displacement in two releases.** The body was at 4609/4750 after v0.19.0 +bought that headroom back; these three lines would have left **5 tokens**. §1's context +dataclass and sub-agent base class moved to `references/patterns.md`, beside the loop +listing that went there in v0.19.0, landing the body at **4631/4750**. That is now twice in +a row that an addition has cost a displacement, which is the auditor's own signal — *the +answer then is a split, not a trim* — and it is filed rather than absorbed again. + ## v0.19.1 — the class the umbrella had been catching for us, twice `B-126`'s board row shipped in v0.19.0 with **nine cells against the eight its header diff --git a/docs/evidence/backlog.md b/docs/evidence/backlog.md index 651f0b9..17e61f9 100644 --- a/docs/evidence/backlog.md +++ b/docs/evidence/backlog.md @@ -32,6 +32,8 @@ row per line, appended, so two runs closing two rows do not land on the same lin | B-124 | **`agent-evals` had no `references/` directory and no statistics layer: 315 lines saying what to assert, nothing saying how many times.** A skill whose job is answering *did it get better* could not distinguish a result from noise — a grep over SKILL.md for the vocabulary of repetition and variance returned **one** hit across 315 lines, and it was about escalation. Separately, §5's *"Right tools, right order"* is the exact practice Anthropic calls *"too rigid"*. | 2026-08-31 harness-engineering harvest; the statistics gap was reported independently by **four** of the twenty agents | 3 | 0 | 2 | **1.5** | **closed 2026-08-31, shipped in v0.18.0** — `references/statistics.md`: the noise band with its table, `pass@k` vs `pass^k` and the 91-point gap, the τ-bench data showing trials are correlated so `pass^k` cannot be exponentiated, paired comparison with 3–5 seeds, the harness as a 6 pp variable with a floor AND a ceiling, and the evidence ladder ending on *4/4 on a slice is not 100% system-wide*. Every figure recomputed before it was written; the correlation finding is ours, from recomputing rather than restating. §5 moved **between** its two measured edges — Anthropic's *too rigid* and Claw-Eval's 44% of safety violations missed by trajectory-blind grading — rather than flipping to the other extreme. | | B-118 (umbrella) | **The committed social card clipped its eyebrow at the canvas, and the check on it counted bytes.** The eyebrow is generated from the umbrella's `role` cell for this pack — 91 characters — and at the smallest scale the renderer will use it needs **1354px of a 1200px canvas**, so the last **11** characters were never drawn: the card read *"…AND THE WALLET UNDER"* and dropped `" LLM RESALE"`. `test/social_preview.py` checked the PNG signature, the chunk order, the byte count and the 1200x630 dimensions — none of which is a reading about text — so it was green over the clipped card for as long as one existed. | umbrella board row B-118, measured by its v1.11.0 run 2026-08-31; re-measured here before acting | 3 | 0 | 2 | **1.5** | **closed 2026-08-31, shipped in v0.18.2** — the finding was re-derived rather than restated: decoding the committed PNG put accent ink at **x=1199 of 1200**, and replaying the renderer's own metric gave a pen finishing at x=1354 with 11 characters past the edge. `fitScale` was already at its floor of 2, so no scale fixes it and the text had to shorten. The new role string is 60 characters — `orchestration, prompts, evals, protocols, and the LLM wallet` — which names all four skills where the old one named three, and paints to **x=919, 196px (19%) inside the content box**. The sensor is the durable half: `test/card_ink.py` decodes the image and requires every painted pixel to sit inside the content box, with the one deliberate exception (the top-left accent bar) required to be **present**, so a blank or mis-decoded image cannot pass by having nothing in the gutters. Watched failing on the previously committed card — *"accent paints at x=1199, y=190 — 84px past the right edge of the content box … and reaches the canvas edge, so characters were cut off entirely"* — while the old check printed `OK` on that same file. A negative self-test in `validate.yml` plants one pixel in the last column of a repo copy and requires the entry point to refuse it, with `plant_guard.py verify` confirming the plant landed. The decoder lives in its own module deliberately: `test/social_preview.py` is shared by nine repositories and the umbrella stops comparing copies below 0.90 similarity **for all nine at once**, so the shared file gained two lines and measures **0.9133** against its eight siblings. Umbrella-side follow-up is the coordinator's and is named in the ledger. | | B-126 | **The pack had zero coverage of the prompt cache — the harvest's most corroborated gap, 14 findings from 9 independent bundles** — and SKILL.md §10 told the reader to rebuild the prompt per request without saying what that costs. Separately, four of the compaction ladder's five rungs were labelled `free` when every one of them edits history and invalidates the prefix from its replacement point. | 2026-08-31 harness-engineering harvest, cluster #1 by computed priority | 3 | 0 | 2 | **1.5** | **closed 2026-08-31, shipped in v0.19.0** — `references/kv-cache.md`: the directional rule, the 100k-conversations timestamp incident (TTFT 0.5s→3–5s, bill nearly doubled), the 2^N cache populations a runtime condition in front of the boundary creates, the named invalidators including the role-switch and hot-plugin trades, and four optimisations that pay cache for nothing. §10 corrected; the ladder gained a Cache column. Headroom bought by **displacement** — the body was at 4749/4750 and the Data Verification Protocol moved to where §10's own text sends prompt content. **The ceiling was enforced in CI and invisible locally**: the *House skill audit* job failed this very release on the headroom GAP, while `npm test` measured nothing — `validate.py` now hard-fails at 5000 and reports at 4750 on every local run, watched failing on a planted 1,200-char body. The §2 listing was displaced to `patterns.md`, taking the body from 4749/4750 to **4609/4750**. | +| B-128 | **§6's three routing traps all assume the REQUEST is what moves, and a mid-run failover moves a trajectory.** Reasoning is portable text plus a non-portable vendor credential — sometimes attached to the tool call rather than the reasoning — so the safest-sounding policy, strip all reasoning before failing over, is the one that 400s. Separately, nothing said where model capability should go: the planner is the bottleneck, not the executor. And a max-iteration guard stops a spend without changing the behaviour that led there. | 2026-08-31 harness-engineering harvest, queue positions 3-4 by computed priority (both ARCH) | 2 | 0 | 2 | **1.0** | **closed 2026-08-31, shipped in v0.20.0** — `llm-proxy-billing.md` gains three sections: what a trajectory cannot carry across a vendor (with the neutral-format rule and the between-turns failover boundary), where the capability goes (Plan-and-Act arXiv:2503.09572, 54% on WebArena-Lite from the planner), and budget awareness (300 steps performing like 30). §6 gains a fourth trap naming the assumption the other three share; §2 gains the budget line beside the iteration refund. | +| B-129 | **`agent-orchestrator`'s SKILL.md body is structurally full: two releases running, an addition has cost a displacement.** v0.19.0 displaced the §2 loop listing to buy room for the cache correction (4749 → 4609); v0.20.0 displaced §1's context dataclass and base class for three routing lines that would otherwise have left 5 tokens (→ 4631). The house auditor states the remedy in its own GAP text — *the answer then is a split, not a trim* — and each pass has instead paid with someone else's listing. | 2026-08-31, observed across v0.19.0 and v0.20.0 | 1 | 0 | 3 | **0.3** | **open.** The displacements were each defensible on their own (a listing belongs in a reference, the body carries what is read every time), but the pattern is a body at capacity rather than three good edits. A split means deciding which of the 13 numbered sections are one skill and which are another — a routing decision, not a formatting one, so it is filed rather than improvised at the end of an unrelated release. | ## Open, and why diff --git a/docs/evidence/verification.md b/docs/evidence/verification.md index b259cef..095e12c 100644 --- a/docs/evidence/verification.md +++ b/docs/evidence/verification.md @@ -10,6 +10,21 @@ This file exists because its absence read as zero exposure. `sshlg-skills` board --- +## Shipped state — v0.20.0 (2026-08-31) + +Measured on the release-candidate tree before the tag exists. Queue positions 3–4 of the +2026-08-31 harvest, both ARCH class, plus the budget finding that shares their subject. + +| REQ | What ships | How it was confirmed | Confirmed | +|---|---|---|---| +| MR-1 | §6 names the assumption its three existing traps share, and the reference carries the mechanism | the new fourth bullet states that the three above assume the *request* is portable; `llm-proxy-billing.md` → *What a trajectory cannot carry across a vendor* has the credential mechanism, the tool-call attachment that makes *strip all reasoning* the failing policy, the neutral-format rule and the between-turns boundary | **observed** | +| MR-2 | Capability allocation is stated, with its source | `references/llm-proxy-billing.md` → *Where the capability goes* cites Plan-and-Act (arXiv:2503.09572) and its 54% on WebArena-Lite from improving the planner; §6 carries the one-line rule | **observed** | +| MR-3 | Budget awareness sits beside the guard it corrects | §2's bullet list gains it after the iteration refund; the reference explains why a max-iteration guard is the floor and not the mechanism (300 steps performing like 30) | **observed** | +| MR-4 | The body stayed inside the working limit, and the headroom was bought rather than borrowed | the three lines alone left **4745/4750 — 5 tokens**; §1's context dataclass and sub-agent base class were displaced to `references/patterns.md`, landing at **4631/4750**. Auditor: `0 GAP, 14 PASS` | **observed** | +| MR-5 | The table-shape guard shipped in v0.19.1 caught its author, on its first use by someone other than its plant | both new board rows were inserted after the last row of *any* table and landed in the three-column *Open, and why* — `python3 test/validate.py` refused with `row has 8 cells against the 3 its own header declares`, twice. **Third occurrence of the class and the first caught by this repository rather than the umbrella**, which is exactly what it was written for | **watched failing** | + +--- + ## Shipped state — v0.19.1 (2026-08-31) Measured on the release-candidate tree before the tag exists. A class caught by the umbrella twice diff --git a/package.json b/package.json index 6c92453..b648862 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@ssheleg/agent-stack", - "version": "0.19.1", + "version": "0.20.0", "scripts": { "test": "python3 test/validate.py && python3 test/plant_guard_test.py && node test/installer_test.js" }, diff --git a/plugins/agent-stack/.claude-plugin/plugin.json b/plugins/agent-stack/.claude-plugin/plugin.json index fa1db1c..8e99023 100644 --- a/plugins/agent-stack/.claude-plugin/plugin.json +++ b/plugins/agent-stack/.claude-plugin/plugin.json @@ -3,7 +3,7 @@ "name": "agent-stack", "displayName": "Agent Stack", "description": "Four skills: agent-orchestrator — tool-calling loops, pipelines with checkpoints, provider routing with fallback, memory architecture, plus the wallet side of reselling LLM access; agent-evals — run/trace/thread evals, LLM judges, and fixtures grown from production; agent-interop — MCP servers and clients, A2A agent cards, the MCP Registry, and gateways; agent-harness — system prompts, tool shaping, workflow-vs-agent, and auditing an agent system.", - "version": "0.19.1", + "version": "0.20.0", "author": { "name": "ssheleg", "url": "https://x.com/sshlg93" diff --git a/plugins/agent-stack/skills/agent-orchestrator/SKILL.md b/plugins/agent-stack/skills/agent-orchestrator/SKILL.md index e15136c..984f062 100644 --- a/plugins/agent-stack/skills/agent-orchestrator/SKILL.md +++ b/plugins/agent-stack/skills/agent-orchestrator/SKILL.md @@ -40,23 +40,12 @@ OrchestratorAgent.run(AgentContext) ### Shared Context Object -Pass a single immutable-ish context object to all sub-agents: +Pass a single immutable-ish context object to every sub-agent. It carries the request +(`project_id`, `user_question`, `chat_history`), the machinery (`llm_router`, `tracker`, +`workflow_id`), the resolved provider and model, and one `extra` dict for pipeline flags. +The full dataclass is in [`references/patterns.md`](references/patterns.md) → *The +orchestrator's shared context and sub-agent protocol*. -```python -@dataclass -class AgentContext: - project_id: str - user_question: str - chat_history: list[Message] - llm_router: LLMRouter # provider abstraction with retry/fallback - tracker: WorkflowTracker # SSE event emitter for real-time UI - workflow_id: str # unique ID for this request - connection_config: ... | None # external resource config - user_id: str | None - preferred_provider: str | None # e.g. "openrouter" - model: str | None # e.g. "/" - extra: dict[str, Any] # pipeline_action, flags, overrides -``` **Key principles:** - Sub-agents never modify context — they return typed results @@ -65,20 +54,10 @@ class AgentContext: ### Sub-Agent Protocol -Every sub-agent extends a base class: - -```python -class BaseAgent(ABC): - @abstractmethod - async def run(self, context: AgentContext, **kwargs) -> AgentResult: ... - - @property - @abstractmethod - def name(self) -> str: ... +Every sub-agent extends one base class with a single abstract `run(context) -> Result`, +so the orchestrator never learns what any of them does internally. The class is in the same +reference section. - @staticmethod - def accum_usage(total, usage): ... # merge token counters -``` Typed result subclasses per agent (e.g. `SQLAgentResult` with `query`, `results`, `attempts`). @@ -108,6 +87,7 @@ was gathered rather than returning nothing. The full listing is in - **Token limit recovery**: On `LLMTokenLimitError`, compress to 60% and retry once. If still fails, return partial answer - **Max iterations guard**: Always have a hard limit. On exhaustion, compose best-effort answer from data gathered so far - **Iteration refund**: a recoverable provider error is not charged to that guard +- **Budget awareness**: tell the model what is left, or 300 steps performs like 30 --- @@ -197,6 +177,11 @@ fallback chain and per-provider retry with exponential backoff, and its - **Model selection has three levels** — the request, the tenant, the system default — and a tenant override that silently loses to a request parameter is how a cheap model ends up billed at a premium one's rate. +- **Those three assume the REQUEST is portable; the trajectory is not.** Reasoning carries + a vendor credential — sometimes on the tool call — so mid-turn failover can 400, and + *strip all reasoning* is what causes it. Fail over between turns. +- **Capability is not spent evenly**: the planner is the bottleneck, so the strongest model + goes to the manager, not to whichever agent does the most work. ## 7. Multi-Layer Memory System Four layers, each with a different lifetime and a different reason to exist: diff --git a/plugins/agent-stack/skills/agent-orchestrator/references/llm-proxy-billing.md b/plugins/agent-stack/skills/agent-orchestrator/references/llm-proxy-billing.md index 05aa7e0..0ec6245 100644 --- a/plugins/agent-stack/skills/agent-orchestrator/references/llm-proxy-billing.md +++ b/plugins/agent-stack/skills/agent-orchestrator/references/llm-proxy-billing.md @@ -260,3 +260,57 @@ model" is otherwise unanswerable. See `patterns.md` for the retry, health-check and error-hierarchy patterns these routing calls sit inside. + +### What a trajectory cannot carry across a vendor + +The routing above assumes the **request** is portable. Mid-run failover is a different +problem, because by then there is an accumulated history and not all of it can move. + +- **Tool calls and results are portable.** They differ in structure between vendors and + mean the same thing, so re-rendering them is enough. +- **Reasoning is not.** It is portable *text* plus a **non-portable credential** the vendor + attaches to prove the reasoning is its own. Vendors disagree on what they demand: one end + validates nothing, the other rejects any credential it did not issue. +- **The credential is not always attached to the reasoning.** It may sit on the *tool call* + — which is why the apparently safe policy *"just strip all reasoning before failing + over"* is exactly what fails at some vendors, and fails as a 400 rather than as + degradation. + +Design rules that follow: + +- Store trajectories in a **neutral internal format**: keep the text, discard the + credential, re-render per vendor at send time. +- Decide the failover boundary deliberately. **Between turns** is cheap and safe; **inside + a turn**, after reasoning has been emitted, is where the credential problem lives. +- A fallback chain that has never been exercised **mid-trajectory** has not been tested. + A green health probe answers a question about the endpoint, not about your history. + +### Where the capability goes — not evenly + +The intuitive allocation is to spend evenly across agents, or to give the strongest model +to whichever agent does the most work. Both are wrong for a planner–executor pair. + +*Plan-and-Act* (arXiv:2503.09572) found the **planner is the bottleneck of the whole +system**: with good enough planning a relatively simple executor suffices, and with a wrong +decomposition every downstream executor is building on a false premise. Their 54% on +WebArena-Lite came from improving the **planner's** planning, not the executor's execution. + +So: **give the strongest model and the most carefully written prompt to the manager**, and +let the executors be cheaper. It also sets where to look when a multi-agent system +underperforms — a weak plan is invisible in every executor's transcript, because each one +did its own step correctly. + +### Budget awareness — steps the agent cannot see buy nothing + +Raising a step budget does not by itself buy more work. Google's *Budget-Aware Tool-Use +Enables Effective Agent Scaling* reports that standard agents have **no budget awareness**, +so at **300 steps** they still conduct shallow searches and plateau at roughly what they +achieve at **30**. + +Spending a larger budget requires telling the model where it is in that budget, so it can +shift strategy — broad exploration early, narrowing later. The multi-agent form is the +manager allocating step budget per sub-task rather than handing every executor the same cap. + +A max-iteration guard that only composes a partial answer at exhaustion is the *floor* of +this, not the mechanism: it stops the spend, and it never changes the behaviour that led +there. diff --git a/plugins/agent-stack/skills/agent-orchestrator/references/patterns.md b/plugins/agent-stack/skills/agent-orchestrator/references/patterns.md index 5a3ad38..c6249bd 100644 --- a/plugins/agent-stack/skills/agent-orchestrator/references/patterns.md +++ b/plugins/agent-stack/skills/agent-orchestrator/references/patterns.md @@ -200,6 +200,51 @@ for attempt in range(1, max_retries + 1): --- +## The orchestrator's shared context and sub-agent protocol + +`SKILL.md` §1 states the two rules — one context object down, typed results back — and this +is the shape they describe. It moved here in v0.20.0 for the same reason §2's loop listing +did in v0.19.0: the body carries what is read every time, a reference carries what is read +once. + +```python +@dataclass +class AgentContext: + project_id: str + user_question: str + chat_history: list[Message] + llm_router: LLMRouter # provider abstraction with retry/fallback + tracker: WorkflowTracker # SSE event emitter for real-time UI + workflow_id: str # unique ID for this request + connection_config: ... | None # external resource config + user_id: str | None + preferred_provider: str | None # e.g. "openrouter" + model: str | None # e.g. "/" + extra: dict[str, Any] # pipeline_action, flags, overrides +``` + +**Key principles:** +- Sub-agents never modify context — they return typed results +- Provider/model preferences flow down from user → project defaults → app defaults +- `extra` carries pipeline state, flags like `_skip_complexity`, session ids + +```python +class BaseAgent(ABC): + @abstractmethod + async def run(self, context: AgentContext, **kwargs) -> AgentResult: ... + + @property + @abstractmethod + def name(self) -> str: ... + + @staticmethod + def accum_usage(total, usage): ... # merge token counters +``` + +The single abstract method is what keeps the orchestrator ignorant of any sub-agent's +internals; a second one is how that boundary starts leaking. + + ## The tool-calling loop, in full `SKILL.md` §2 states the six steps and the guard; this is the listing they describe. It