From d6838f65ddcbabfd05995d13b4d6e06401d3524c Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 5 Oct 2026 06:29:16 +0000 Subject: [PATCH 1/3] =?UTF-8?q?board:=20storage=20portability=20=E2=80=94?= =?UTF-8?q?=20Quack=20owns=20query=20semantics,=20Rubicon=20owns=20durable?= =?UTF-8?q?=20commit,=20storage=20supplies=20capabilities?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Architecture-only follow-up to #1328. Records the read boundary (a backend binds once before execution and never approximates Quack semantics), the write boundary (Rubicon amortizes transient folds into one durable sparse commit; 64k parallelism and Kanban are never storage requirements), NodeGuid × Version vs backend physical history, the existing seams that already bind before execution (quack::lower, mask-risc validate, Activation::resolve_for_context), and non-binding backend mappings for Lance/MOCA, RocksDB, Iceberg, DuckDB and S3. Flags an open conflict: the proposed durable field-granular merge-on-read write is not what contract::alpha implements (row-granular claims, unclaimed = None never base, discardable). Not renamed or resolved here. No code, no trait, no backend. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01G22yT6htkcdyXsihxxXdrg --- .../2026-10-05-quack-storage-portability.md | 165 ++++++++++++++++++ .claude/board/entries/README.md | 3 +- 2 files changed, 167 insertions(+), 1 deletion(-) create mode 100644 .claude/board/entries/2026-10-05-quack-storage-portability.md diff --git a/.claude/board/entries/2026-10-05-quack-storage-portability.md b/.claude/board/entries/2026-10-05-quack-storage-portability.md new file mode 100644 index 000000000..c18834035 --- /dev/null +++ b/.claude/board/entries/2026-10-05-quack-storage-portability.md @@ -0,0 +1,165 @@ +# 2026-10-05 — Storage portability: Quack owns query semantics, Rubicon owns durable commit, storage supplies capabilities + +Follow-up to `2026-10-05-quack-two-world-frontend.md` (#1328, merged). It applies the same resolve-once lifecycle to the storage boundary. **Contract and architecture only: no backend, no storage trait, no new type.** + +## DECISION — the three boundaries + +``` +QUACK makes reads portable: stable numeric query semantics across backends +RUBICON makes writes amortizable: stable durable-transition semantics across backends +BACKEND supplies capabilities: physical read and write capabilities, at both boundaries +``` + +- **SCOPE:** architecture for lance-graph's query and persistence boundaries. +- **BASIS:** the two-world contract (describe once, resolve once, canonicalize once, execute numeric) and the #1326 lifecycle, applied to physical execution. +- **REVISIT WHEN:** a second backend adapter is actually built. At that point the capability list below must be measured against it, not extended speculatively. + +``` + DEVELOPER WORLD + SQL / Rust / Java / SAP / IAM + | + v + RESOLUTION MEMBRANE + names -> ids, source -> numeric + canonical LE at byte boundaries + | + v + QUACK + stable numeric semantics + | + physical binding (once) + / | \ + Lance/MOCA Iceberg DuckDB / RocksDB + | + v + transient execution + Kanban / folds / masks <- never a storage requirement + | + | 0..many operations + v + RUBICON + durable transition + | + v + sparse immutable commit + NodeGuid × Version × field mask × payload + | + backend write mapping + / | | \ + MOCA RocksDB Iceberg S3 +``` + +Not every backend takes part in every layer, and not in the same way. + +## DECISION — Quack ↔ backend: the read boundary + +- **Storage does not own Quack semantics.** `quack::Query` is the stable logical and numeric query contract: no strings, no catalog lookup, canonical little-endian only where raw bytes are read. A backend may execute some or all of it physically. +- **A backend never approximates Quack semantics.** + - Each part of a query either runs exactly on the backend, or runs beside the backend over numeric lanes the backend supplies. + - Otherwise binding fails before execution. + - "Close enough" pushdown is a correctness bug, not an optimization. +- **Binding happens once, before execution** (the #1326 law): `Query + schema + backend capabilities → resolved physical plan → repeated execution`. There is no capability check per row, and no dynamic dispatch per fold where the choice can be made beforehand. +- **Storage format is replaceable; query semantics are not.** The same `Query` may run over Lance native lanes, Arrow batches, DuckDB vectors, Iceberg/Parquet columns or decoded RocksDB values, provided the adapter reproduces the same numeric result. The canonical LE rule fixes byte interpretation where raw bytes cross a boundary. It does not require identical files. +- **No universal `Storage` trait.** Capabilities run in two independent, asymmetric directions: + - **read:** projection, `EqU32`, ordered range, count, group reduce, semijoin, strided reads, mask-native execution, predicate pushdown; + - **write:** immutable object append, atomic batch, compare-and-swap, snapshot/generation commit, manifest publication, compaction/rewrite. + + These are recorded as vocabulary, not as an enum. No type is added until a second backend gives one something real to describe. + +## MEASURED — the capability seams that already exist in code + +The binding-before-execution shape is already the code's shape. No new abstraction is needed to state the rule. + +| seam | what it already does | +|---|---| +| `lance_graph_quack::lower(&Query) -> Result` | The current physical binding (Quack → mask-risc). It refuses unsupported shapes before execution (`GroupedBlend`, `EmptyJunction`, `HavingAggOutOfRange`, …) rather than answering approximately. | +| mask-risc `validate` (`check_lane` / `LaneKind`, run by `execute_into` and `reference_execute_into` before any row) | Lane-kind and bounds checks are total and happen before the first row. An `Err` leaves scratch untouched. This is the "no capability check per row" property. | +| `lance_graph_contract::hotplug::Activation::resolve_for_context` → `ResolvedReading` | Storage reading resolved once per population, with named `ActivationDrift` failures. | + +A future backend adapter is a sibling of `lower`: `Query → (backend-native part, local numeric remainder)` or `Err`, decided once. + +## DECISION — Rubicon: the write boundary + +``` +transient state: Kanban / thoughts / folds + | potentially ~1M folds, in memory + v + RUBICON the commit decision + | + v + durable sparse commit one amortized transition + | + v + backend "commit this generation" +``` + +- **The backend receives an already-amortized durable transition.** The contract is "commit this durable generation/change set", not "persist every logical operation". +- **None of the following is a backend requirement:** 64k concurrent writers, thought scheduling, Kanban state, Rubicon phases, one write per fold. The backend cannot tell whether a commit came from 1 fold or 10⁶. +- **The name is anchored in existing code.** In the contract, Rubicon is the commit decision point (`rubicon_witness`, `action.rs`: an action is `Pending` until the Rubicon commit boundary). **No durable-commit type exists yet.** This entry names where one belongs; it does not build one. +- **Fire-and-forget, with replay.** Intermediate fold states need not persist. Crash recovery needs one of: + - a stable base plus deterministic input; + - a compact replay/event input; + - a periodic checkpoint; + - an equivalent durable recipe (`C0 + input batch + deterministic recipe → C1`). + + None of these needs one write per fold. R2IL and reasoning machinery stay out of scope. + +## DECISION — identity layers stay separate + +| layer | meaning | owner | +|---|---|---| +| `NodeGuid` | semantic object identity | ours | +| Version / generation | logical object generation | ours | +| backend snapshot, sequence, file or fragment id | physical history | the backend | + +`NodeGuid × Version` is the semantic lineage. It is never aliased to an Iceberg snapshot id, an S3 object name, a RocksDB sequence number or a Lance fragment id. + +## OPEN — "Alpha" as the durable sparse write: a conflict with the existing contract + +The proposed durable record is a sparse write over an immutable address: +- `NodeGuid × Version × mask × ChangedPayload`; +- merge on read: `effective = base ⊕ Δ(v1) ⊕ … ⊕ Δ(vN)`, with the mask deciding which coordinates each Δ overrides; +- only the coordinates a query needs are resolved, so projection pruning and sparse merge cooperate. + +That semantics is sound. **It is not, however, what `lance_graph_contract::alpha` / `alpha_tunnel` (the split-tunnel) implement today.** The two disagree on four points: + +| | proposed durable sparse write | existing `contract::alpha` | +|---|---|---| +| granularity | per coordinate/field (mask over a row's fields) | per row: `claim` materializes one whole 512-byte `NodeRow` (`alpha.rs:20-23`) | +| what the mask ranges over | fields of one address | `AlphaMask` is a population bitset over the overlay's addresses (`alpha.rs:60-62`) | +| read of an unwritten coordinate | falls back to the base (merge on read) | `None` = "not attended", explicitly **never** the base row (`alpha.rs:25-28`) | +| durability | the durable commit form, versioned | "ephemer daneben, verwerfbar": discardable whole, no bake row, no digest (`alpha.rs:11-17`) | + +`.claude/knowledge/reference-frame-vs-motion.md` (`ISS-LXA-ALPHA-FIT`) also forbids a value that a bake measured from living in alpha, and allows motion to be "discardable whole, or versioned as runtime state". So a *versioned* motion overlay is permitted. A base-falling-back field merge is a different read contract. + +**Not resolved here, and not renamed.** Calling the durable sparse commit "Alpha" would give one name two incompatible read semantics: "absent = not attended" versus "absent = inherit the base". The decision belongs to the operator: +- (a) Alpha gains a second, explicitly separate *durable* reading, with merge-on-read living beside the existing overlay read; +- (b) the durable sparse write gets its own name, and Alpha stays the discardable attention overlay. + +Until then, this entry calls it the **sparse commit**. Neither `alpha.rs` nor the split-tunnel writer is touched. + +## WORKING-MODEL — backend mappings (vision, not implementation) + +None of these is built, promised equivalent, or measured. Each is a capability profile to verify. + +- **Lance / MOCA.** + - Read: `Quack → lower → mask-risc → ndarray` over native lanes (exists). + - Write: sparse commit → sparse delta beside the spine → merge on read → native compaction → optional push to S3. +- **RocksDB.** + - Read: exact key/range lookups, with the remaining numeric work local. + - Write: sparse commit → versioned keys in one `WriteBatch` → newest-visible read or merge operator → RocksDB compaction. +- **Iceberg.** + - Read: predicate and projection pushdown only where exact; the remainder runs locally. + - Write: sparse commit → immutable delta/data files → snapshot/manifest publication → merge on read → later rewrite/compaction. + - **This is an adapter mapping to investigate, not a claim that Iceberg implements the sparse-commit contract one to one.** Candidates are a base table plus a sparse delta relation, or native row-level delete/update where its semantics match exactly. Choosing needs a dedicated, measured experiment. +- **DuckDB.** + - `Quack → relational subset translation → DuckDB`. + - It is useful both as an execution backend and as a differential semantic oracle for Quack. `crates/lance-graph-quack/tests/duckdb_differential.rs` already plays that role. +- **S3 / object storage.** + - Requires no in-place append. + - Minimal mapping: a chain of immutable commit objects, plus an optional manifest/head/generation pointer. "Append" means appending immutable objects to the logical history, not appending bytes to one object. + - With local MOCA objects as the hot tier, `~10⁶ transient folds → Rubicon → one or a few durable objects → S3` replaces 10⁶ remote writes with a handful. **Not measured; no performance claim.** + +## Scope guards held + +No RocksDB, Iceberg, DuckDB or S3 backend, and no `Storage`/`Backend` trait or capability enum. Kanban and Rubicon stay out of Quack, and no write-concurrency requirement is placed on storage. `NodeGuid`, version semantics, #1326, the IAM contracts, `alpha.rs` and the split-tunnel writer are unchanged. Alpha is not exposed to developer queries. diff --git a/.claude/board/entries/README.md b/.claude/board/entries/README.md index f4e6edc24..663abe3ff 100644 --- a/.claude/board/entries/README.md +++ b/.claude/board/entries/README.md @@ -25,12 +25,13 @@ index row, (3) no duplicate entry id. Checks 1 and 2 are deliberately opposite directions; the stranding this convention prevents shows up in exactly one of them, never both. -215 entries, 2026-08-06 .. 2026-10-05. +216 entries, 2026-08-06 .. 2026-10-05. | date | entry id | finding | file | |---|---|---|---| | 2026-10-05 | `text-to-numeric-boundary-inventory` | | [2026-10-05-text-to-numeric-boundary-inventory.md](2026-10-05-text-to-numeric-boundary-inventory.md) | | 2026-10-05 | `quack-two-world-frontend` | | [2026-10-05-quack-two-world-frontend.md](2026-10-05-quack-two-world-frontend.md) | +| 2026-10-05 | `quack-storage-portability` | | [2026-10-05-quack-storage-portability.md](2026-10-05-quack-storage-portability.md) | | 2026-10-04 | `D-LXC-22` | | [2026-10-04-wordnet-clam-chaoda-scope-and-grammar-read-params.md](2026-10-04-wordnet-clam-chaoda-scope-and-grammar-read-params.md) | | 2026-10-04 | `D-HPS-1` | | [2026-10-04-spog-slab-hotplug-resolution.md](2026-10-04-spog-slab-hotplug-resolution.md) | | 2026-10-04 | `D-HPS-2` | | [2026-10-04-resolve-once-population-execution.md](2026-10-04-resolve-once-population-execution.md) | From 1d981277257e611610d5ef3c63d4d16f575ac29b Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 5 Oct 2026 06:33:42 +0000 Subject: [PATCH 2/3] =?UTF-8?q?board:=20correct=20storage=20portability=20?= =?UTF-8?q?against=20the=20source=20audit=20=E2=80=94=20Alpha=20unchanged,?= =?UTF-8?q?=20durable=20boundary=20is=20the=20cycle=20seal?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Alpha is authoritative and not redefined: a discardable transient overlay of whole-row attention claims (stamp only, no payload, no field data). Rubicon in code is the Planning -> CognitiveWork phase crossing, not the durable write; the amortizing write boundary is persist_sink's cycle seal (one WAL write per cycle) behind the existing WalSink seam, implemented by LanceCycleWriter, at row granularity. Field-level dirtiness is tracked nowhere; SparseDelta is architecture vocabulary only, derivable at DetachedCycleBatch::freeze by comparing final images against base_version. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01G22yT6htkcdyXsihxxXdrg --- .../2026-10-05-quack-storage-portability.md | 194 ++++++++++-------- 1 file changed, 104 insertions(+), 90 deletions(-) diff --git a/.claude/board/entries/2026-10-05-quack-storage-portability.md b/.claude/board/entries/2026-10-05-quack-storage-portability.md index c18834035..ca4b65d06 100644 --- a/.claude/board/entries/2026-10-05-quack-storage-portability.md +++ b/.claude/board/entries/2026-10-05-quack-storage-portability.md @@ -1,18 +1,18 @@ -# 2026-10-05 — Storage portability: Quack owns query semantics, Rubicon owns durable commit, storage supplies capabilities +# 2026-10-05 — Storage portability: Quack owns query semantics, a durable commit boundary owns writes, storage supplies capabilities -Follow-up to `2026-10-05-quack-two-world-frontend.md` (#1328, merged). It applies the same resolve-once lifecycle to the storage boundary. **Contract and architecture only: no backend, no storage trait, no new type.** +Follow-up to `2026-10-05-quack-two-world-frontend.md` (#1328, merged). It applies the same resolve-once lifecycle to the storage boundary. **Contract and architecture only: no backend, no storage trait, no new type. Alpha is unchanged and is not redefined.** -## DECISION — the three boundaries +## DECISION — the boundaries ``` -QUACK makes reads portable: stable numeric query semantics across backends -RUBICON makes writes amortizable: stable durable-transition semantics across backends -BACKEND supplies capabilities: physical read and write capabilities, at both boundaries +QUACK makes reads portable: stable numeric query semantics across backends +DURABLE COMMIT makes writes amortizable: many transient operations -> one durable transition +BACKEND supplies capabilities: physical read and write capabilities, at both boundaries ``` - **SCOPE:** architecture for lance-graph's query and persistence boundaries. -- **BASIS:** the two-world contract (describe once, resolve once, canonicalize once, execute numeric) and the #1326 lifecycle, applied to physical execution. -- **REVISIT WHEN:** a second backend adapter is actually built. At that point the capability list below must be measured against it, not extended speculatively. +- **BASIS:** the two-world contract (describe once, resolve once, canonicalize once, execute numeric), the #1326 lifecycle, and the source audit below. +- **REVISIT WHEN:** a second backend adapter is actually built. Measure the capability list against it then; do not extend it speculatively. ``` DEVELOPER WORLD @@ -32,19 +32,22 @@ BACKEND supplies capabilities: physical read and write capabilities, at both Lance/MOCA Iceberg DuckDB / RocksDB | v - transient execution - Kanban / folds / masks <- never a storage requirement + Quack numeric execution | - | 0..many operations v - RUBICON - durable transition + transient working state never a storage requirement + + Alpha overlay (current: transient, discardable, row-level attention) + + Kanban / folds (Rubicon = Planning -> CognitiveWork crossing, pre-execution) | + | 0..many casts v - sparse immutable commit - NodeGuid × Version × field mask × payload + durable commit boundary (current: persist_sink cycle seal, one WAL write per cycle) | - backend write mapping + v + durable net change (current: full 512-byte image per dirty row; + future candidate: SparseDelta, field-level) + | + backend write mapping (current seam: WalSink::commit_cycle) / | | \ MOCA RocksDB Iceberg S3 ``` @@ -54,112 +57,123 @@ Not every backend takes part in every layer, and not in the same way. ## DECISION — Quack ↔ backend: the read boundary - **Storage does not own Quack semantics.** `quack::Query` is the stable logical and numeric query contract: no strings, no catalog lookup, canonical little-endian only where raw bytes are read. A backend may execute some or all of it physically. -- **A backend never approximates Quack semantics.** - - Each part of a query either runs exactly on the backend, or runs beside the backend over numeric lanes the backend supplies. - - Otherwise binding fails before execution. - - "Close enough" pushdown is a correctness bug, not an optimization. -- **Binding happens once, before execution** (the #1326 law): `Query + schema + backend capabilities → resolved physical plan → repeated execution`. There is no capability check per row, and no dynamic dispatch per fold where the choice can be made beforehand. -- **Storage format is replaceable; query semantics are not.** The same `Query` may run over Lance native lanes, Arrow batches, DuckDB vectors, Iceberg/Parquet columns or decoded RocksDB values, provided the adapter reproduces the same numeric result. The canonical LE rule fixes byte interpretation where raw bytes cross a boundary. It does not require identical files. +- **A backend never approximates Quack semantics.** Each part of a query either runs exactly on the backend or runs beside it over numeric lanes the backend supplies. Otherwise binding fails before execution. +- **Binding happens once, before execution** (the #1326 law): `Query + schema + backend capabilities → resolved physical plan → repeated execution`. There is no capability check per row and no per-fold dispatch where the choice can be made beforehand. +- **Storage format is replaceable; query semantics are not.** The same `Query` may run over Lance lanes, Arrow batches, DuckDB vectors, Iceberg/Parquet columns or decoded RocksDB values, provided the numeric result is reproduced exactly. - **No universal `Storage` trait.** Capabilities run in two independent, asymmetric directions: - **read:** projection, `EqU32`, ordered range, count, group reduce, semijoin, strided reads, mask-native execution, predicate pushdown; - **write:** immutable object append, atomic batch, compare-and-swap, snapshot/generation commit, manifest publication, compaction/rewrite. - These are recorded as vocabulary, not as an enum. No type is added until a second backend gives one something real to describe. + These are recorded as vocabulary, not as an enum. -## MEASURED — the capability seams that already exist in code +## MEASURED — source audit (2026-10-05) -The binding-before-execution shape is already the code's shape. No new abstraction is needed to state the rule. +### Alpha: authoritative and unchanged -| seam | what it already does | -|---|---| -| `lance_graph_quack::lower(&Query) -> Result` | The current physical binding (Quack → mask-risc). It refuses unsupported shapes before execution (`GroupedBlend`, `EmptyJunction`, `HavingAggOutOfRange`, …) rather than answering approximately. | -| mask-risc `validate` (`check_lane` / `LaneKind`, run by `execute_into` and `reference_execute_into` before any row) | Lane-kind and bounds checks are total and happen before the first row. An `Err` leaves scratch untouched. This is the "no capability check per row" property. | -| `lance_graph_contract::hotplug::Activation::resolve_for_context` → `ResolvedReading` | Storage reading resolved once per population, with named `ActivationDrift` failures. | +Files: `crates/lance-graph-contract/src/alpha.rs` and `alpha_tunnel.rs` (the split-tunnel). Types: `AlphaAddr`, `AlphaStamp`, `AlphaMask`, `AlphaAllocation`, `AlphaOverlay`, `AlphaClaim`, `AlphaError`, `AlphaTunnel`. -A future backend adapter is a sibling of `lower`: `Query → (backend-native part, local numeric remainder)` or `Err`, decided once. +Current semantics: +- **Discardable transient overlay** (`alpha.rs:11-16`, "ephemer daneben, verwerfbar": no bake, no digest; `discard(self)`). +- **Whole-row claims.** `claim` materializes one 512-byte `NodeRow`: key copied verbatim from the base, edges zeroed, and an `AlphaStamp { cycle, seq, rung, visits }` in value slot 0. **The rest of the value slab stays zero.** +- **Population tracking.** `AlphaMask` is a bitset over the overlay's addresses (`alpha.rs:60-61`). +- An unclaimed address reads `None` ("not attended") and never falls back to the base (`alpha.rs:25-28`). -## DECISION — Rubicon: the write boundary +**Alpha holds no field-level information and no payload.** There is no API that writes data into a claimed row's value. `claim` writes only the stamp, `get` and `rows` are read-only, and `alpha_tunnel` only merges stamps by rung. -``` -transient state: Kanban / thoughts / folds - | potentially ~1M folds, in memory - v - RUBICON the commit decision - | - v - durable sparse commit one amortized transition - | - v - backend "commit this generation" -``` +Alpha may *inform* a future durable change: it records where attention went in a cycle. It is not, and does not become, that change. Durable storage mappings are never called "Alpha storage". + +### Rubicon: implemented code, but as the pre-execution phase crossing, not as the durable write -- **The backend receives an already-amortized durable transition.** The contract is "commit this durable generation/change set", not "persist every logical operation". -- **None of the following is a backend requirement:** 64k concurrent writers, thought scheduling, Kanban state, Rubicon phases, one write per fold. The backend cannot tell whether a commit came from 1 fold or 10⁶. -- **The name is anchored in existing code.** In the contract, Rubicon is the commit decision point (`rubicon_witness`, `action.rs`: an action is `Pending` until the Rubicon commit boundary). **No durable-commit type exists yet.** This entry names where one belongs; it does not build one. -- **Fire-and-forget, with replay.** Intermediate fold states need not persist. Crash recovery needs one of: - - a stable base plus deterministic input; - - a compact replay/event input; - - a periodic checkpoint; - - an equivalent durable recipe (`C0 + input batch + deterministic recipe → C1`). +Rubicon exists in code (category A), but **not as the durable commit boundary**: +- `contract/src/kanban.rs` defines `RubiconTransitionError` and the Rubicon lifecycle transitions over `KanbanColumn`. +- `contract/src/rubicon_witness.rs` (`RubiconVerdict`, `RubiconReading`) reads which side of the Planning → CognitiveWork crossing a thought is on, from its focus mask. It reads; it never moves anything. +- `planner/src/owner_adapter.rs` preserves "the Rubicon crossing itself (`Planning → CognitiveWork`)" bit for bit. +- `contract/src/action.rs`: an `ActionState` stays `Pending` until the cycle decides the result sound, then commits out. +- `cognitive-compiler::RubiconPhase` is a separate phase enum. - None of these needs one write per fold. R2IL and reasoning machinery stay out of scope. +So in current code Rubicon is the Heckhausen commitment point **before** execution (deliberation → implementation). It is not the point where transient work becomes durable. -## DECISION — identity layers stay separate +### The actual durable write boundary (category B for the brief's "Rubicon") -| layer | meaning | owner | -|---|---|---| -| `NodeGuid` | semantic object identity | ours | -| Version / generation | logical object generation | ours | -| backend snapshot, sequence, file or fragment id | physical history | the backend | +The amortizing write membrane exists, under different names: -`NodeGuid × Version` is the semantic lineage. It is never aliased to an Iceberg snapshot id, an S3 object name, a RocksDB sequence number or a Lance fragment id. +| file / symbol | role | +|---|---| +| `planner/src/batch_writer.rs` `BatchWriter::cast` | ephemeral intent records; payload is a descriptor `(mailbox, dirty row-range, cycle)`, never delta bytes | +| `planner/src/persist_sink.rs` `persist_cycle`, `DetachedCycleBatch::freeze`, `SweepSlot`, `CycleFrame` | cast ⊂ chunk ⊂ cycle. The casts are stable-ordered, then folded per row (last state wins). **One WAL write per cycle, one `DatasetVersion`.** Intent-only casts produce `NoChange` and zero writes. | +| `planner/src/persist_sink.rs` `trait WalSink { commit_cycle, scan_sealed, … }` | the existing backend **write-capability seam**: one amortized durable commit per cycle, reconciliation by `(cycle, batch_hash)` | +| `lance-graph/src/graph/cycle_sink.rs` `LanceCycleWriter: WalSink` | the real Lance implementation: the sole, non-`Clone` writer | +| `lance-graph-supervisor/src/cycle_driver.rs` | drives it: planner casts → one `persist_cycle` | + +This already realizes the properties the brief asks of a write membrane: +- the backend sees one commit per cycle, never one per thought, and does not know how many thoughts contributed; +- 64k parallel producers are fire-and-forget casters, never writers; +- recovery reads sealed landings through `recover_and_apply`; the intent records are not a WAL. + +### What changed inside a row: tracked nowhere at field level + +- **Granularity is the row.** `SweepSlot.row: u64` names the dirty SoA row, and `LanceCycleWriter` stores one `FixedSizeBinary(512)` **final image per dirty row per cycle** (`cycle_sink.rs:105-123`). The batch-writer descriptor is a dirty *row range*. +- **No field, coordinate or lane dirtiness exists in the substrate write path:** + - no mask beside Alpha; + - no split-tunnel metadata beyond the stamp; + - no per-lane tracking; + - no comparison against the base row. +- `lance-graph-hydrate::dirty::is_dirty` is per **dataset**: it compares `version_id()` against the version the caller obtained when it hydrated. +- The only attribute-level change representation is domain-local and in memory: `lance-graph-dir-sim`'s `Overlay` (per-attribute override maps) and `Change::SetAttribute { from, to }`. That is a directory-simulation model, not the substrate write path. + +### NodeGuid × Version: not the write path's identity today + +| layer | where it lives today | +|---|---| +| semantic object identity | `NodeGuid`: the first 16 bytes of the 512-byte payload; the write path keys by SoA `row: u64`, not by `NodeGuid` | +| semantic generation | `(cycle, batch_hash)` per committed batch: `FrameMeta`, `LandedSlot` ("semantic identity lives in `(cycle, batch_hash)`") | +| physical history | Lance `DatasetVersion`, deliberately not re-derivable per row | -## OPEN — "Alpha" as the durable sparse write: a conflict with the existing contract +There is no per-node `Version`. The separation the brief wants (semantic lineage ≠ backend history) already holds as `(cycle, batch_hash)` ≠ `DatasetVersion`. A future `NodeGuid × Version` lineage would extend that, never replace it with Lance, Iceberg, RocksDB or S3 identifiers. -The proposed durable record is a sparse write over an immutable address: -- `NodeGuid × Version × mask × ChangedPayload`; -- merge on read: `effective = base ⊕ Δ(v1) ⊕ … ⊕ Δ(vN)`, with the mask deciding which coordinates each Δ overrides; -- only the coordinates a query needs are resolved, so projection pruning and sparse merge cooperate. +## OPEN — SparseDelta (architecture vocabulary only, not a type) -That semantics is sound. **It is not, however, what `lance_graph_contract::alpha` / `alpha_tunnel` (the split-tunnel) implement today.** The two disagree on four points: +A future durable sparse-change contract can represent changed coordinates over immutable semantic identity and support merge-on-read: -| | proposed durable sparse write | existing `contract::alpha` | -|---|---|---| -| granularity | per coordinate/field (mask over a row's fields) | per row: `claim` materializes one whole 512-byte `NodeRow` (`alpha.rs:20-23`) | -| what the mask ranges over | fields of one address | `AlphaMask` is a population bitset over the overlay's addresses (`alpha.rs:60-62`) | -| read of an unwritten coordinate | falls back to the base (merge on read) | `None` = "not attended", explicitly **never** the base row (`alpha.rs:25-28`) | -| durability | the durable commit form, versioned | "ephemer daneben, verwerfbar": discardable whole, no bake row, no digest (`alpha.rs:11-17`) | +``` +SparseDelta { semantic_identity: NodeGuid, version, changed_coordinates, payload } -- representation unresolved +effective = base ⊕ Δ(v1) ⊕ … ⊕ Δ(vN), resolving only the coordinates a query needs +``` -`.claude/knowledge/reference-frame-vs-motion.md` (`ISS-LXA-ALPHA-FIT`) also forbids a value that a bake measured from living in alpha, and allows motion to be "discardable whole, or versioned as runtime state". So a *versioned* motion overlay is permitted. A base-falling-back field merge is a different read contract. +`changed_coordinates` could be a field bitmap, a lane bitmap, byte ranges, a typed coordinate list or a backend-native delta. **No choice is made, and nothing is implemented.** -**Not resolved here, and not renamed.** Calling the durable sparse commit "Alpha" would give one name two incompatible read semantics: "absent = not attended" versus "absent = inherit the base". The decision belongs to the operator: -- (a) Alpha gains a second, explicitly separate *durable* reading, with merge-on-read living beside the existing overlay read; -- (b) the durable sparse write gets its own name, and Alpha stays the discardable attention overlay. +**The smallest real seam is `DetachedCycleBatch::freeze` → `WalSink::commit_cycle`.** At freeze time the batch already holds the coalesced final 512-byte image per dirty row, and `CycleFrame.base_version` names the sealed predecessor. A SparseDelta is therefore derivable at exactly one place, by comparing each final image against the same row at `base_version`. That comparison is the missing field-dirtiness signal; today it does not exist. It could later become a refinement of `WalSink`'s image rows, not a new write path. -Until then, this entry calls it the **sparse commit**. Neither `alpha.rs` nor the split-tunnel writer is touched. +Alpha does not supply that signal (it carries no payload), and the split-tunnel writer is not touched. ## WORKING-MODEL — backend mappings (vision, not implementation) -None of these is built, promised equivalent, or measured. Each is a capability profile to verify. +None of these is built, promised equivalent, or measured. None is "Alpha storage". - **Lance / MOCA.** - Read: `Quack → lower → mask-risc → ndarray` over native lanes (exists). - - Write: sparse commit → sparse delta beside the spine → merge on read → native compaction → optional push to S3. + - Write: `WalSink` / `LanceCycleWriter` (exists, full-row images). A SparseDelta refinement would be a native delta/chunk beside the base, merged on read and compacted natively, with an optional push to S3. - **RocksDB.** - - Read: exact key/range lookups, with the remaining numeric work local. - - Write: sparse commit → versioned keys in one `WriteBatch` → newest-visible read or merge operator → RocksDB compaction. + - Read: exact key/range lookups, with the rest computed locally. + - Write: a `WalSink` whose `commit_cycle` is one versioned `WriteBatch`, read newest-visible, with RocksDB compaction. - **Iceberg.** - - Read: predicate and projection pushdown only where exact; the remainder runs locally. - - Write: sparse commit → immutable delta/data files → snapshot/manifest publication → merge on read → later rewrite/compaction. - - **This is an adapter mapping to investigate, not a claim that Iceberg implements the sparse-commit contract one to one.** Candidates are a base table plus a sparse delta relation, or native row-level delete/update where its semantics match exactly. Choosing needs a dedicated, measured experiment. + - Read: pushdown only where exact, with the rest computed locally. + - Write: one cycle → immutable data/delta files + one snapshot/manifest publication. + - **An adapter mapping to investigate, not a one-to-one claim.** Candidates are a base table plus a delta relation, or native row-level delete/update where its semantics match exactly. This needs a dedicated, measured experiment. - **DuckDB.** - - `Quack → relational subset translation → DuckDB`. - - It is useful both as an execution backend and as a differential semantic oracle for Quack. `crates/lance-graph-quack/tests/duckdb_differential.rs` already plays that role. + - `Quack → relational subset translation`, with a transactional mapping for commits. + - Also a differential semantic oracle for Quack; `quack/tests/duckdb_differential.rs` already does this. - **S3 / object storage.** - - Requires no in-place append. - - Minimal mapping: a chain of immutable commit objects, plus an optional manifest/head/generation pointer. "Append" means appending immutable objects to the logical history, not appending bytes to one object. - - With local MOCA objects as the hot tier, `~10⁶ transient folds → Rubicon → one or a few durable objects → S3` replaces 10⁶ remote writes with a handful. **Not measured; no performance claim.** + - Write: one cycle → one immutable commit object, plus an optional manifest/head/generation pointer. "Append" means appending immutable objects to the logical history; S3 needs no in-place append. + - With local MOCA objects as the hot tier, ~10⁶ transient folds → one or a few durable objects instead of 10⁶ remote writes. **Not measured; no performance claim.** + +## Replay / fire-and-forget + +Intermediate fold states are not persisted. This is already the code's rule: `BatchWriter` intent records are explicitly not a WAL, and recovery is a pinned-reference read of sealed landings. A cheaper future replay (checkpoint `C0` + input batch + deterministic recipe → `C1`) remains conceptual; R2IL stays out of scope. ## Scope guards held -No RocksDB, Iceberg, DuckDB or S3 backend, and no `Storage`/`Backend` trait or capability enum. Kanban and Rubicon stay out of Quack, and no write-concurrency requirement is placed on storage. `NodeGuid`, version semantics, #1326, the IAM contracts, `alpha.rs` and the split-tunnel writer are unchanged. Alpha is not exposed to developer queries. +- No RocksDB, Iceberg, DuckDB or S3 backend, and no `Storage` trait or capability enum. +- No `SparseDelta` type; Alpha is unchanged and not redefined; the split-tunnel and cycle writers are not redesigned. +- Kanban and Rubicon stay out of Quack, and no write-concurrency requirement is placed on storage. +- No change to `NodeGuid`, version semantics, #1326 or IAM. From 9b297086fac42cce3def84366bd98d5688240f0a Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 5 Oct 2026 06:43:04 +0000 Subject: [PATCH 3/3] board: add the Evaluation / revision.rs phase between transient execution and the cycle seal Rubicon commits intent, revision judges the result, the cycle seal commits state. Records advance_on_revision's routing (accepted -> Commit; eligible without counterfactual or NoIncrease -> Plan -> Planning; Suspend holds), that revision.rs and advance_on_revision have no production caller, and that today's seal is driven by artifact casts with Kanban moves applied post-seal, so it is not gated on revision acceptance. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01G22yT6htkcdyXsihxxXdrg --- .../2026-10-05-quack-storage-portability.md | 53 ++++++++++++++++--- 1 file changed, 46 insertions(+), 7 deletions(-) diff --git a/.claude/board/entries/2026-10-05-quack-storage-portability.md b/.claude/board/entries/2026-10-05-quack-storage-portability.md index ca4b65d06..ac9e91212 100644 --- a/.claude/board/entries/2026-10-05-quack-storage-portability.md +++ b/.claude/board/entries/2026-10-05-quack-storage-portability.md @@ -6,7 +6,7 @@ Follow-up to `2026-10-05-quack-two-world-frontend.md` (#1328, merged). It applie ``` QUACK makes reads portable: stable numeric query semantics across backends -DURABLE COMMIT makes writes amortizable: many transient operations -> one durable transition +CYCLE SEAL makes writes amortizable: many transient operations -> one durable transition BACKEND supplies capabilities: physical read and write capabilities, at both boundaries ``` @@ -35,13 +35,26 @@ BACKEND supplies capabilities: physical read and write capabiliti Quack numeric execution | v - transient working state never a storage requirement - + Alpha overlay (current: transient, discardable, row-level attention) - + Kanban / folds (Rubicon = Planning -> CognitiveWork crossing, pre-execution) + Planning + | Rubicon: intent becomes action (Planning -> CognitiveWork) + v + CognitiveWork / Action + | + v + transient execution never a storage requirement + folds + Alpha overlay (transient, discardable, row-level attention) | - | 0..many casts v - durable commit boundary (current: persist_sink cycle seal, one WAL write per cycle) + Evaluation / revision.rs epistemic judgment; performs no write + |-- NoIncrease, or IncreaseEligible with docket incomplete + | -> Plan -> Planning (re-deliberate, carrying the witness) + |-- Suspend -> held in Evaluation (tension open, pending grounding) + `-- IncreaseEligible + counterfactual Necessary -> Commit (accepted) + | + | 0..many casts (BatchWriter::cast) + v + cycle seal: DetachedCycleBatch::freeze -> one WAL write per cycle + | performs no epistemic evaluation | v durable net change (current: full 512-byte image per dirty row; @@ -54,6 +67,32 @@ BACKEND supplies capabilities: physical read and write capabiliti Not every backend takes part in every layer, and not in the same way. +**Rubicon commits intent. Revision judges the result. The cycle seal commits state.** These are three distinct boundaries; none absorbs another. + +## MEASURED — the lifecycle as wired today (audit 2026-10-05) + +| step | code | status | +|---|---|---| +| Rubicon: intent becomes action | `KanbanColumn` Planning → CognitiveWork (`contract/src/kanban.rs`), read by `rubicon_witness` | implemented; not redefined here | +| revision: judge the result | `revision.rs` `GadamerRevision::revise` → `RevisionDelta { kind, evidential_effect, … }`; `RevisionVerdict { effect, counterfactual }`; `is_acceptable()` = `IncreaseEligible ∧ Necessary` | pure policy. Its output types "deliberately stop before actual-world mutation" (`revision.rs:450`). **No production caller**: only tests and `examples/probe_revision_attention_view.rs` | +| route on the verdict | `KanbanColumn::advance_on_revision` (`kanban.rs:262`) | contract only; **no production caller** | +| the seal | `persist_cycle` / `DetachedCycleBatch::freeze` → `WalSink::commit_cycle` | implemented and driven by `supervisor/cycle_driver.rs` | + +How the verdict feeds back, per `advance_on_revision`: +- **`IncreaseEligible` and counterfactual `Necessary`** → `advance()` → `Commit`. The cycle settles. +- **`IncreaseEligible` with the docket incomplete** → `revise()` → `Plan`. Eligible is not accepted. +- **`NoIncrease`** (including `Echo` and `ClosedCycle`) → `Plan`. Understanding rose and evidence did not, so the cycle re-deliberates carrying the witness. +- **`Suspend`** → `None`. The mailbox stays in Evaluation; the tension stays open pending grounding. +- **Revision never prunes.** `Prune` is the MUL gate's `Block`, a different act on a different arm. + +Feedback therefore goes `Plan → Planning` and re-crosses the Rubicon. It does not jump straight back into CognitiveWork (`next_phases`: `Evaluation → {Commit, Plan, Prune}`, `Plan → {Planning}`). + +**Gap between this chain and today's wiring (recorded, not fixed):** +- The seal is **not gated on revision acceptance.** A cycle seals whatever artifact casts it holds. Kanban moves, including `Evaluation → Commit`, ride in `SweepSlot::paired_move` and are applied after the seal (`recover_and_apply` → `try_advance_phase`). A pure Kanban step writes nothing (`cycle_sink.rs:34-40`). +- The `Commit` column's "calcify" step is itself **declared, not implemented** (`kanban.rs:47-54`). + +So "accepted → `BatchWriter::cast`" is the intended ordering, not a wired one. Making the seal wait for acceptance would be a deliberate change to the caster/driver, not something this entry does. + ## DECISION — Quack ↔ backend: the read boundary - **Storage does not own Quack semantics.** `quack::Query` is the stable logical and numeric query contract: no strings, no catalog lookup, canonical little-endian only where raw bytes are read. A backend may execute some or all of it physically. @@ -93,7 +132,7 @@ Rubicon exists in code (category A), but **not as the durable commit boundary**: So in current code Rubicon is the Heckhausen commitment point **before** execution (deliberation → implementation). It is not the point where transient work becomes durable. -### The actual durable write boundary (category B for the brief's "Rubicon") +### The actual durable write boundary: the cycle seal, not Rubicon The amortizing write membrane exists, under different names: