feat(cache): add an opt-in 1h cache_control ttl - #42
Merged
Conversation
A cache_control breakpoint with no ttl gets the 5m tier. That is short enough to be self-defeating on long turns: the entry is written during prefill, so a turn that runs longer than 5m outlives its own cache and the next turn pays a full cold prefill of the whole conversation. Observed live -- a 341s turn left a 353s gap, the next turn read 0 cached tokens and re-prefilled 223K. extend_cache_ttl fills in ttl="1h" at breakpoints that carry none. An explicit ttl is left alone, since a client that names one has priced the trade itself. Off by default: extended writes bill at a higher rate than 5m ones while reads cost the same, so the premium lands on every write and the saving only on an expiry that would otherwise have happened. On a conversation doing many small incremental writes between rare expiries that is a net loss.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A
cache_controlbreakpoint with nottlgets the 5-minute tier. That is short enough to be self-defeating on long turns: the entry is written during prefill, so a turn that itself runs longer than five minutes has already outlived its own cache by the time it finishes, and the next turn pays a full cold prefill of the whole conversation.Observed live on this proxy. A 341s turn left a 353s gap to the next one, whose prefix was byte-identical (
first differing message index: Noneafter strippingcache_controlmarkers) — and it still read 0 cached tokens and re-prefilled 223K. Every one of the other 196 large requests in the same window had a gap under 131s and hit cache. That cold prefill then pushed the turn past an upstream ceiling at ~633s, so it came back 502stream_interruptedhaving burned $4.19 and returned nothing.extend_cache_ttlfills inttl = "1h"at breakpoints that carry none. An explicitttlis left exactly as sent, since a client that names one has priced the trade itself.Verified end to end against Copilot:
ephemeral_1hephemeral_5mttl: 5mDurability confirmed separately: a 1h entry read back in full at t+400s, past the point a 5m entry would have been gone. A bogus
ttl: "99h"is rejected upstream withInput should be '5m' or '1h', so the field is genuinely parsed rather than dropped.Off by default, deliberately. Extended writes bill at a higher rate than 5m ones while reads cost the same, so the premium is charged on every write and the saving only lands on an expiry that would otherwise have happened. Modelled against this session's actual token counts, blanket 1h would have cost about 30% more — many small incremental writes between rare expiries. It pays off when turns routinely run past five minutes.
Also worth noting: Claude Code never sets
ttlitself — 146 system and 73 message breakpoints observed, all bare{"type": "ephemeral"}— so the proxy is the only place to intervene.155 lib + 18 bin tests pass; clippy clean.