Skip to content

fix(verdict-cache): bounded per-entry LRU so a return to a recent state HITS (JEF-390)#206

Merged
thejefflarson merged 1 commit into
mainfrom
thejefflarson/jef-390-multi-slot-lru-verdict-cache-return-to-a-recent-state-should
Jul 8, 2026
Merged

fix(verdict-cache): bounded per-entry LRU so a return to a recent state HITS (JEF-390)#206
thejefflarson merged 1 commit into
mainfrom
thejefflarson/jef-390-multi-slot-lru-verdict-cache-return-to-a-recent-state-should

Conversation

@thejefflarson

Copy link
Copy Markdown
Owner

Closes JEF-390.

Problem

The verdict cache was single-slot per entry (cached: Option<(String, Verdict)>). A workload whose evidence oscillates between states A and B re-judged on every flip: judging B overwrote A's cached verdict, and flipping back to A missed — hammering the slow CPU model on a state it had already decided.

Change (engine/src/engine/state/verdict_store.rs)

  • Replaced the single slot with a bounded per-entry LRU of (fingerprint → decisive Verdict) — a new VerdictLru (an ordered Vec with move-to-front + truncate; no crate).
  • Default 32 slots, env-configurable via PROTECTOR_VERDICT_CACHE_SLOTS. Unset / unparseable / 0 → default; a positive value is honoured but floored at 2 (MIN_VERDICT_CACHE_SLOTS) so the cache can always retain one prior state. The cap is resolved once at construction, not per insert.
  • cached_for hits on any fingerprint in the entry's LRU and promotes it to most-recently-used; cache_decisive inserts most-recently-used and evicts the least-recently-used past the cap.

Invariants preserved (no regression)

  • Only DECISIVE verdicts are cached. cache_decisive is the sole insert path and the engine calls it only in its decisive arm; an Uncertain is never added and still arms the JEF-234 EntryBackoff (verified by test).
  • Exact-fingerprint hit = byte-identical evidence (the fingerprint is the whole-prompt hash), so a served verdict is exactly as valid as when judged — wider retention introduces no new staleness.
  • Journal-restore (JEF-141), display carry-forward, journaled dedup, recency (JEF-201), and the Mutex lock discipline are untouched. This ticket only widens the cached gate from 1 slot to N.

Tests

Added engine/src/engine/state/verdict_store_tests.rs (sibling file to stay under the 1,000-line cap), all driven through a small harness that mirrors the engine judging loop:

  1. A→B→A serves A's verdict from cache on the return (2 judgements, not 3).
  2. Beyond the cap the LRU evicts the least-recently-used state and re-judges it — and a revisit before eviction promotes the survivor (proves LRU, not FIFO).
  3. Uncertain is not cached and the entry backs off (JEF-234); a later decisive verdict clears the backoff and populates the LRU.
  4. The env knob parses: defaults on unset/invalid/0/negative, honours+trims positive values, floors 1 to the minimum.

Checks

  • cargo fmt — clean
  • cargo clippy --all-targets (warnings = errors) — clean
  • cargo nextest run778 passed (4 new), file-size guard green

🤖 Generated with Claude Code

https://claude.ai/code/session_01VtjoJttCvBY4dzCoE4f9vP

…udged state HITS (JEF-390)

The verdict cache was a single slot per entry, so a workload whose evidence
oscillates between states A and B re-judged on every flip: judging B overwrote
A's cached verdict, and flipping back to A missed. Widen the `cached` gate from
one slot to a bounded per-entry LRU of (fingerprint -> decisive Verdict) —
default 32 slots, env-configurable via PROTECTOR_VERDICT_CACHE_SLOTS
(unset/unparseable/0 -> default; positive values floored at 2 so the cache can
always retain one prior state).

`cached_for` now hits on ANY fingerprint in the entry's LRU and promotes it to
most-recently-used; `cache_decisive` inserts most-recently-used and evicts the
least-recently-used past the cap. Dependency-light: an ordered Vec with
move-to-front + truncate, no crate.

Invariants preserved: only DECISIVE verdicts are ever inserted — an Uncertain is
never cached and still arms the JEF-234 exponential backoff; an exact-fingerprint
hit is byte-identical evidence (the fingerprint is the whole-prompt hash), so a
served verdict is exactly as valid as when judged, just more retained — no new
staleness. Journal-restore, display carry-forward, journaled dedup, recency, and
the lock discipline are untouched.

Tests (sibling verdict_store_tests.rs to stay under the 1,000-line cap): A->B->A
serves A on the return (2 judgements, not 3); beyond the cap the LRU evicts the
least-recently-used state and re-judges it (proving LRU, not FIFO); Uncertain is
not cached and the entry backs off; the env knob defaults on invalid/zero and
floors small values.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VtjoJttCvBY4dzCoE4f9vP
@thejefflarson thejefflarson merged commit 57d3219 into main Jul 8, 2026
5 checks passed
@thejefflarson thejefflarson deleted the thejefflarson/jef-390-multi-slot-lru-verdict-cache-return-to-a-recent-state-should branch July 8, 2026 06:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant