Skip to content

v0.19.0 — the prompt cache is an architectural constraint, and the gate that never watched the body - #19

Merged
sshlg merged 1 commit into
mainfrom
feat/kv-cache-iso
Aug 31, 2026
Merged

v0.19.0 — the prompt cache is an architectural constraint, and the gate that never watched the body#19
sshlg merged 1 commit into
mainfrom
feat/kv-cache-iso

Conversation

@sshlg

@sshlg sshlg commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Fourteen findings from nine independent bundles of the 2026-08-31 harvest said the same thing, and this pack had zero coverage of it:

grep -rniE 'kv.cache|prefix cache|prompt cach|cache hit|cacheable' plugins/agent-stack/skills/
→ 0 hits

It was the harvest's most corroborated gap and came out first on computed priority.

references/kv-cache.md

Spined on one directional rule — the earlier a change lands, the more cache it invalidates — from which the append-only discipline stops being style and becomes arithmetic.

A Current time: {{now}} line at 100,000 conversations/day TTFT 0.5 s → 3–5 s, monthly bill nearly doubled. Nothing else changed.
N binary runtime conditions in front of the cache boundary 2^N cache populations. Three innocuous ones — OS × debug × locale — make eight cold caches out of one warm one.
Sorting tools by usage frequency costs the prefix, and buys almost no tool-selection accuracy. Cost for nothing.
"Manual concatenation breaks the cache" False. Caching is over bytes. The real cost of flattening is an out-of-distribution format.

Two invalidators are architecture rather than mistakes, and the file says so: a role switch that replaces the system prompt buys a hard tool boundary and pays the prefix for it, where a skill keeps the prefix and gives an instruction the model may ignore; and a hot-loaded plugin owes its documentation a KV-cache impact, not only a behavioural one.

The architectural error it corrects

§10 said the prompt is rebuilt per request from live capability flags "so the two can never disagree" — right for correctness, silent on cost. It now states that rebuilding is free only while it is byte-identical, and that a capability change mid-session is appended rather than written into the prefix.

Four of the compaction ladder's five rungs were labelled free, and none of them is. Every rung edits history, so every rung invalidates from its replacement point. context-engineering.md gains a Cache column and the scheduling rule that follows: compact in batch at a threshold, never every round.

Displacement, not deletion — and why nobody saw the ceiling

The body sat at 4749 of the 4750-token working limit. One token. So the §10 correction was paid for by moving the Data Verification Protocol into agent-harness/references/system-prompt.md — which is where §10's own text says prompt content belongs, so a protocol sitting in the wiring section had been contradicting the boundary its section drew.

This gate never measured the body. The limit is real doctrine that make-skill enforces and agent-stack did not, so a file could sit one token from the edge in silence. test/validate.py now hard-fails at 5000 and reports at 4750, on every run pass or fail — a warning printed only beside a failure is invisible on exactly the runs where it can still be acted on. The estimator errs high against the house auditor (~4947 vs ~4804) deliberately, documented in the code so it is not "reconciled" into the looser one.

Cut in an isolated worktree

agent_sync.py status showed two leasesB-118 and B-126-kv-cache — with both sessions' uncommitted work in one checkout on one branch. My six files were backed up, a worktree created from origin/main, and the shared checkout restored to clean main. git show --name-only 62bb910 ∩ my file list = empty, so the separation held at the commit level too.

Evidence

Check Result
npm test rc=0
Budget plant (1,200 chars appended) refused — body ~5547 tokens, past the 5000 budget
Ladder table header, separator and every row at 5 cells
Reference two-way check passes — a link to a missing file and an unlinked orphan both fail it

Board B-126. Ledger KV-1 … KV-6.

🤖 Generated with Claude Code

Fourteen findings from nine independent bundles of the 2026-08-31 harvest said
the same thing, and this pack had zero coverage: a grep for kv-cache / prefix
cache / prompt cach / cache hit / cacheable over all four skills returned nothing.

references/kv-cache.md is the missing layer, spined on one directional rule —
the earlier a change lands, the more cache it invalidates:
- a `Current time: {{now}}` line at 100k conversations/day took TTFT from 0.5s to
  3-5s and nearly doubled the monthly bill
- N binary runtime conditions in front of the boundary produce 2^N cache
  populations; three innocuous ones make eight cold caches out of one warm one
- the named invalidators, including the two that are architecture rather than
  mistakes: a role switch buys a hard tool boundary and pays the prefix for it;
  a hot-loaded plugin owes its docs a KV-cache impact
- four things that look like optimisations and are not, each with the
  measurement that settles it

SKILL.md section 10 said the prompt is rebuilt per request "so the two can never
disagree" — right for correctness, silent on cost. It now says rebuilding is free
only while byte-identical. The compaction ladder called four of five rungs free;
none is, and it gains a Cache column.

Headroom was bought by displacement, not deletion: the body sat at 4749 of the
4750 working limit, so the Data Verification Protocol moved to
agent-harness/references/system-prompt.md, where section 10's own text sends
prompt content.

And the ceiling was invisible because this gate never measured the body.
validate.py now hard-fails at the 5000 budget and reports at the 4750 working
limit on every run, pass or fail. Watched failing on a planted 1,200-char body.
The estimator errs high against the house auditor on purpose.

Cut in an isolated worktree: agent-sync showed two leases and two sessions'
uncommitted work in one checkout on one branch.

B-126, KV-1..6. Gate: npm test rc=0.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@sshlg
sshlg force-pushed the feat/kv-cache-iso branch from 1e7ff0a to 902f568 Compare August 31, 2026 17:08
@sshlg
sshlg merged commit c23ff6d into main Aug 31, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant