Skip to content

feat: add runtime-controlled request-local caching#52

Draft
lan17 wants to merge 3 commits into
mainfrom
agent/request-local-cache
Draft

feat: add runtime-controlled request-local caching#52
lan17 wants to merge 3 commits into
mainfrom
agent/request-local-cache

Conversation

@lan17

@lan17 lan17 commented Jul 16, 2026

Copy link
Copy Markdown
Owner

Summary

Adds an opt-in request-local cache layer whose lifetime is the outermost enabled DialCache scope. Runtime policy can enable it per invocation with requestLocal: true, while existing local and Redis TTL/ramp behavior remains unchanged.

Closes #51#51

Architecture and design

Request lifetime

  • The outermost enable() creates a lightweight request holder.
  • Nested enable() and disable() calls reuse that holder while retaining async-scope-local enabled state.
  • Request-local value and in-flight maps are allocated together, lazily, only when an invocation resolves requestLocal: true.
  • When the outer callback settles, the holder closes and any maps are cleared. Detached async work inherited from that boundary becomes pass-through and cannot repopulate request-local state.

Runtime policy

  • DialCacheKeyConfig now accepts optional requestLocal?: boolean; the compatibility default is off.
  • Request-local policy has no TTL or ramp and is intentionally not part of the TTL/ramp-oriented CacheLayer enum.
  • Each enabled cached-function invocation fetches runtime configuration once and passes that snapshot through the selected chain.
  • A false decision bypasses request-local reads and writes without evicting an earlier scoped value, so a later true decision in the same scope can reuse it.

Lookup and coalescing

request-scoped single-flight
  -> request-local value
  -> process-scoped single-flight
  -> local LRU
  -> Redis
  -> fallback

Only active portions of the chain participate:

  • Request-local work coalesces before the request-local lookup.
  • After a request-local miss, active local/Redis work enters the existing process-wide coalescer before the first shared layer.
  • With request-local off, active local/Redis work still coalesces process-wide.
  • When every layer is off, calls remain independent pass-through executions.
  • Single-flight state is removed after fulfillment or rejection; only successful resolved values are memoized.
  • Cached undefined is preserved with membership-based lookup semantics.

Both value maps and flight maps continue to use the existing canonical DialCacheKey.urn. This deliberately avoids changing key encoding in this PR; the centralized typed-key work remains tracked by #14.

Invalidation, retention, and ownership

  • Targeted invalidation remains remote-only. invalidateRemote() does not evict or fence request-local or process-local values.
  • Request-local storage has no entry cap or eviction policy; the documented operating assumption is a short-lived scope with bounded key cardinality.
  • Request-local and process-local values are stored natively without cloning, freezing, or serializer round-tripping.
  • API and README guidance now make cached return values caller-immutable by contract and document layer-dependent reference identity.

The broader ownership test matrix remains tracked by #18.

API and observability

  • Adds DialCacheKeyConfig.requestLocal.
  • Uses the fixed metric-layer value request_local without adding request-local policy to CacheLayer or expanding the root value exports.
  • Adds bounded coalescing scope values request_local | process to custom adapter labels.
  • Preserves the existing dialcache_coalesced_counter{use_case,key_type} schema.
  • Adds dialcache_scoped_coalesced_counter{use_case,key_type,scope}.
  • Adds request-local request, miss, and get observations with layer="request_local".

Metric evolution and config observability follow-ups remain tracked by:

Documentation and maintainer tooling

  • Documents the minimal request-local-only configuration first, then separates request-local, process-local, and Redis behavior.
  • Covers HTTP request-boundary usage, nested scope behavior, lazy allocation, retention, invalidation, coalescing, and immutable value ownership.
  • Adds the repository-only pnpm benchmark:request-local maintainer command for sequential request-local hits plus request- and process-scoped fan-out.
  • Keeps the published package limited to runtime artifacts; package-consumer checks verify the public ESM/CJS/type surface.

Validation

Run with Node v22.22.0:

  • corepack pnpm check
    • 123 unit tests passed
    • coverage: 96.55% statements, 90.55% branches, 98.03% functions, 96.61% lines
    • ESM/CJS declarations and packed-package consumer checks passed
  • corepack pnpm test:integration
    • 46 Redis integration tests passed
  • corepack pnpm benchmark:request-local
    • 50,000 sequential request-local hits: one fallback
    • 1,000 request-local coalesced calls: one fallback
    • 1,000 process-coalesced calls: one fallback
  • Generated root declarations contain no request-local implementation helpers or extra layer/scope exports.
  • Fresh core and whole-diff cleanup reviews returned no findings.
  • git diff --check

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a runtime-controlled request-local cache layer

1 participant