Beyond codecs: layer kinds for filters/caches in the flat model, and fast-op hook placement

[thorwhalen]

Intent record (maintainer, 2026-08-21). Formalizes the P3 bullet already noted in #90, and gives #24's fast-path hooks a designated home. The flat model (#86, dol/_interface_wrap.py) today represents only pure per-role codecs; filt_iter, cached_keys, and friends are not codecs and still nest. Intent: extend the flat stack with first-class layer kinds so the flat-model guarantees (inverse mapping, one boundary hop, pickle-by-construction) can extend to mixed stacks — and place bulk/fast-op hooks where they are correct by construction.

Layer-kind vocabulary (intent, grounded in a trans.py survey)

• pure-codec — per-role Codec(encoder, decoder); needs two flags the survey surfaced: one-way (e.g. add_decoder, conditional_data_trans install only a decoder) and store-aware (FirstArgIsMapping / self-convention transforms receive the store — and which store they receive is exactly the wrap_kvs will wrap the instance but self of instance is not wrapped #18/Key-transform wrappers delegate capability methods with the unmapped key (url_for etc. silently address the wrong object) #83 fault line, so this must be explicit, never silent).
• contextual-codec — postget/preset: f(k, v) -> v, value transform conditioned on the key. Cannot fuse into per-role pipelines; needs its own kind with an explicit composition order relative to plain value codecs.
• filter — filt_iter: changes the key set, not the key form. Carries the predicate — and should carry the declarative payload (suffix/prefix/regex strings) rather than only a compiled closure, so a fast-op planner can push the filter down to a backend query (S3 prefix LIST, SQL LIKE, mongo find — the Ideas for extensions of dol -- namely when we will re-architecture it #24 headline case). filter_regex already stashes its pattern on the produced callable; keep that pattern alive for all declarative constructors.
• keys-cache (cached_keys: cache container + updatability flags), value-cache (cache_vals), error-memo (catch_and_cache_error_keys — the hardest: its observable key set changes over the store's lifetime).
• interceptor — effectful policies: confirm_overwrite (reads the store and prompts — syntactically a preset, semantically not a codec), add_missing_key_handling.
• Explicitly OUT of the stack (so the flat guarantees stay honest): path/reshape layers (add_path_get/add_path_access/autoviv/flatten — second key language, recursive descent) and spec-surface edits (read-only masks, aliases, added methods) which the flat model represents as edits to the interface spec, not stack entries.

Fast-op hooks (#24) — placement intent

Spec-registered batch operations that traverse the layer kinds: the leaf advertises capabilities (batch-get / batch-write / batch-delete / contains-batch / native-len / native-filter); each layer kind declares how it participates (codec: encode the batch; filter: apply — or push down — its predicate; caches: batch-maintain state). A bulk op is fast-dispatchable iff every layer above the leaf participates; otherwise fall back to the per-item loop. This makes fast-ops an argument for the layer-kind vocabulary rather than a separate feature.

The update() case (#56) — what we verified

• Store never overrides update; the MutableMapping mixin runs per-item self[k] = v through the wrapped __setitem__ — transform-correct, never bulk.
• The commented-out raw self.store.update(...) delegation in base.py would bypass all transforms — correctly disabled; do not resurrect it.
• Backend-native bulk writes are structurally unreachable through today's wrappers (update is found by MRO before __getattr__ delegation, and delegate_to subtracts dir(wrapper_cls)).
• cached_keys' generated update (forwards per item, batch-maintains its cache once) is the in-tree proof that bulk ops require layer cooperation.
• So: keep the mixin loop as the universal correct fallback; add the fast path only where the compiled per-role encoders are visible — the flat proxy / spec route (the prototype's tests already probe an update_all(**kv: VT) shape). A nested wrapper structurally cannot do encode-all-then-one-backend-call.

Links

#24 (the hook idea), #56 (fast update/sync), #86 + misc/docs/dol_flat_model.md (the flat model), #90 (the P3 bullet this formalizes). Sequenced as P3 in misc/docs/dol_roadmap.md — after the P2 facade, since layer kinds extend the same stack.