From 89382f43b7490e3d6f51982a9b6ae185c1adf4b3 Mon Sep 17 00:00:00 2001 From: meh Date: Mon, 7 Sep 2026 16:49:45 +0700 Subject: [PATCH 1/4] Require arctic 0.1.11 and ps-reclaim 0.1.4 ps-reclaim 0.1.4 closes a use-after-free. A retirement published between the participant scan and the extraction of garbage was judged against a decision taken before it existed, so a live reader's object could be reclaimed under it. The fix is a sequence cutoff captured under the garbage mutex before the scan. It also stops bounded drains being quadratic: `extract_if(..).take(k)` compacts the unchecked tail on drop, so a 128,000 backlog at `advance_up_to(256)` moved 23.29 ms of records under that mutex and now moves 1.28 ms. Both were already being picked up by resolution, because `^0.1` allows them. Raising the floors says they are required rather than merely permitted, which is what a correctness fix means. arctic 0.1.11 is the release where `smr-ps-reclaim` stopped implying `std`. It changes nothing here - WorkTable links `std` and takes ps-reclaim directly with default features, so feature unification gives it `std` either way - but the floor is what lets a no_std consumer downstream rely on it. cargo test 927 passed, 0 failed cargo clippy --all-targets clean --- Cargo.toml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Cargo.toml b/Cargo.toml index e32eb50c..e18d14ee 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -37,7 +37,7 @@ versioned-row-publication = ["worktable_codegen/versioned-row-publication"] # pointer-only fast path. Publication is append-only and asserted at each swap. arc-swap = "1" async-trait = "0.1" -arctic = { package = "arctic-wt", version = "^0.1, >=0.1.9", default-features = false, features = ["smr-ps-reclaim"] } +arctic = { package = "arctic-wt", version = "^0.1, >=0.1.11", default-features = false, features = ["smr-ps-reclaim"] } congee = { package = "congee-wt", version = "^0.4, >=0.4.4" } convert_case = "0.6" crc32fast = "1" @@ -59,7 +59,7 @@ prettytable-rs = "0.10" psc-nanoid = { version = "3", features = ["rkyv", "packed"] } rkyv = { version = "0.8", features = ["uuid-1"] } reqwest = { version = "0.12", optional = true, default-features = false, features = ["rustls-tls-webpki-roots", "charset", "http2"] } -ps-reclaim = { version = "^0.1, >=0.1.3" } +ps-reclaim = { version = "^0.1, >=0.1.4" } rustc-hash = "2" rusty-s3 = { version = "0.10", optional = true } smart-default = "0.7" From 93b938e1ade51e6d2f107ac65c396571dace2b53 Mon Sep 17 00:00:00 2001 From: meh Date: Mon, 7 Sep 2026 16:50:25 +0700 Subject: [PATCH 2/4] Write down what a data page has to say about itself beta.19 changed the on-disk format and every `.wt.data` on this machine had to be thrown away and rebuilt on 6 September 2026, because nothing could read the old shape. That is a regeneration event, and the reason it happened is that a data page cannot be read without the index that points into it. This test states the requirement for beta.20 as an executable assertion rather than a paragraph in a release note: a page that describes itself can be read by a reader that has never seen the writer's index. --- tests/slotted_page_requirement.rs | 203 ++++++++++++++++++++++++++++++ 1 file changed, 203 insertions(+) create mode 100644 tests/slotted_page_requirement.rs diff --git a/tests/slotted_page_requirement.rs b/tests/slotted_page_requirement.rs new file mode 100644 index 00000000..32af729f --- /dev/null +++ b/tests/slotted_page_requirement.rs @@ -0,0 +1,203 @@ +//! What a data page has to say about itself, for beta.20. +//! +//! # Where this comes from +//! +//! beta.19 changed the on-disk format and every existing `.wt.data` had to be +//! thrown away and rebuilt, because nothing could read the old shape. That is a +//! regeneration event, and it happened on 6 September 2026 across every store on +//! this machine. +//! +//! It does not have to happen again. The reason it did is that a data page +//! cannot be read without the index that points into it: +//! +//! ```ignore +//! pub struct DataPage { +//! pub length: u32, +//! pub data: [u8; DATA_LENGTH], +//! } +//! ``` +//! +//! Rows are bump allocated into `data` and `length` is a high water mark. There +//! are no delimiters, so nothing can tell where one row ends and the next +//! begins. `empty_links_list` in the `SpaceInfoPage` records freed ranges and is +//! explicitly lossy: `bound_empty_links_list` truncates it when it outgrows the +//! info page and logs "space leak, not corruption". +//! +//! **The schema is already there and this is not asking for it again.** +//! `SpaceInfoPage` carries `row_schema`, `primary_key_fields` and +//! `secondary_index_types`, and `ensure_schema` refuses a mismatch by name. A +//! reader already knows how to decode a row. What it cannot do is find one. +//! +//! # The two things, in order of how much they matter +//! +//! 1. **A row directory in the data page**, the usual slotted layout: an +//! `(offset, length)` per row growing down from the end of the page, with a +//! count. Then a page describes itself, a reader needs no index, and the CRC +//! on that page validates the directory together with the rows it points at. +//! +//! 2. **A reader for the format beta.19 writes**, so beta.20 is an upgrade +//! rather than another regeneration. One already exists and is switched off: +//! `src/page/iterators.rs` in DataBucket, where `LinksIterator` walks index +//! pages for links and `DataIterator` follows them, decoding through +//! `row_schema`. It is 226 lines, commented out at `src/page/mod.rs:4`, and +//! enabling it produces nine errors that are bit rot rather than design: +//! `crate::IndexData` and `super::SpaceInfo` were renamed, and one call site +//! predates the API going async. +//! +//! # How the two fit together +//! +//! `DATA_VERSION` is 2 today and lives in every page's `GeneralHeader`, so it is +//! per page rather than per file. +//! +//! - **beta.20 ships both.** It writes 3 and reads 2 and 3. +//! - **beta.21 ships neither of the old ones.** The v2 path is deleted. +//! +//! So v2 is a one way ramp rather than dual support: a store is loaded through +//! it once, written back as v3, and never read that way again. It does not need +//! to be fast and it never needs append, which is most of why it is cheap. +//! +//! Two things to settle rather than discover: +//! +//! - Once a page has a directory and an index, both know where a row is and they +//! can disagree. One has to be authoritative. The directory is the better +//! candidate: it is local to the page and validated by the same CRC, where the +//! index is a separate structure with a different topology per backend. Under +//! `validate-reads` a load can compare the two and name a disagreement instead +//! of silently preferring one. +//! - Whether one file may hold both v2 and v3 pages. Per page versioning allows +//! it, which makes migration an append rather than a rewrite, but then no +//! reader may assume uniformity. +//! +//! # What is missing here, and is the next piece of work +//! +//! A committed `.wt.data` written by beta.19, so the ramp can be tested against +//! a real old file rather than against one this build just wrote. Until that +//! fixture exists, `a_store_reopens_without_being_rebuilt` below only proves the +//! current version reopens, which is the weaker half. + +use worktable::prelude::*; +use worktable::worktable; + +worktable!( + name: SlottedRow, + version: 1, + persist: true, + columns: { + id: u64 primary_key autoincrement, + blob: String, + }, +); + +/// Enough rows to fill more than one page, so a directory would be doing real +/// work rather than describing a single row. +const ROWS: u64 = 4_000; + +async fn filled(dir: &str) -> SlottedRowWorkTable { + let _ = std::fs::remove_dir_all(dir); + std::fs::create_dir_all(dir).expect("a directory"); + + let engine = SlottedRowPersistenceEngine::new(DiskConfig::new_with_table_name( + dir, + SlottedRowWorkTable::name_snake_case(), + SlottedRowWorkTable::version(), + )) + .await + .expect("an engine"); + let table = SlottedRowWorkTable::load(engine).await.expect("a table"); + + for n in 0..ROWS { + table + .insert(SlottedRowRow { + id: table.get_next_pk().into(), + blob: format!("row {n}, long enough to make the page boundaries interesting"), + }) + .await + .expect("a row"); + } + table.wait_for_ops().await.expect("the queue drains"); + table +} + +fn data_file(dir: &str) -> std::path::PathBuf { + std::path::Path::new(dir) + .join(SlottedRowWorkTable::name_snake_case()) + .join(".wt.data") +} + +/// A data page should say where its rows are, without an index. +/// +/// **This is the beta.20 requirement.** The check goes straight at the bytes on +/// purpose. Reading the page through the engine would prove only that the index +/// still works, and the index is exactly what a self describing page is supposed +/// to make unnecessary. +/// +/// Written against bytes rather than against an API that does not exist yet, so +/// this file compiles today and fails on the missing behaviour rather than on a +/// missing symbol. +#[tokio::test] +#[ignore = "beta.20: a data page carries no row directory"] +async fn a_data_page_says_where_its_rows_are() { + let dir = "tests/data/slotted_page/self_describing"; + let table = filled(dir).await; + table.close().await.expect("the table closes"); + + let bytes = std::fs::read(data_file(dir)).expect("the file"); + assert!( + bytes.len() > PAGE_SIZE, + "the fixture has to span pages: {} bytes", + bytes.len() + ); + + // A slotted page keeps its directory at the end: a row count in the last + // four bytes, then that many (offset, length) pairs growing back up. Any + // layout would do; what matters is that something in the page delimits the + // rows. Today the tail is write padding, so this reads zero. + let mut described = 0usize; + for page in bytes.chunks_exact(PAGE_SIZE).skip(1) { + let mut tail = [0u8; 4]; + tail.copy_from_slice(&page[PAGE_SIZE - 4..]); + described += u32::from_le_bytes(tail) as usize; + } + + assert_eq!( + described, ROWS as usize, + "no page says how many rows it holds, so the {ROWS} rows in this file \ + cannot be found without the index. A row directory in the data page is \ + what makes a page readable on its own, and what makes the next format \ + change an upgrade instead of a regeneration." + ); + let _ = std::fs::remove_dir_all(dir); +} + +/// A store reopens without being deleted first. +/// +/// **Not ignored, and passing.** It guards the property at the current version, +/// so a format change that breaks reopening trips here rather than in somebody's +/// deploy. It is the weaker half of the requirement: proving beta.20 can read +/// beta.19 needs a beta.19 file committed as a fixture, which does not exist +/// yet. +#[tokio::test] +async fn a_store_reopens_without_being_rebuilt() { + let dir = "tests/data/slotted_page/reopen"; + let table = filled(dir).await; + table.close().await.expect("the table closes"); + + let engine = SlottedRowPersistenceEngine::new(DiskConfig::new_with_table_name( + dir, + SlottedRowWorkTable::name_snake_case(), + SlottedRowWorkTable::version(), + )) + .await + .expect("an engine"); + let reopened = SlottedRowWorkTable::load(engine) + .await + .expect("a store reopens rather than needing to be rebuilt"); + + assert_eq!( + reopened.select_all().execute().expect("a read").len(), + ROWS as usize, + "no rows are lost reopening a store" + ); + reopened.close().await.expect("the table closes"); + let _ = std::fs::remove_dir_all(dir); +} From 5dcbf21b195dfcda7a9cee05e58b18ea00e1e2ac Mon Sep 17 00:00:00 2001 From: meh Date: Mon, 7 Sep 2026 16:50:25 +0700 Subject: [PATCH 3/4] Map paper two onto evidence that already exists The CIDR submission deferred the lock-discipline scaling comparison, crash consistency, protocol checking, the cost of monomorphization, and baselines beyond redb and LMDB. This plans the paper those deferrals point at, and pins each candidate contribution to the code and the measurements that back it, so the writing starts from what has landed rather than from an outline. --- docs/paper-2-plan.md | 115 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 115 insertions(+) create mode 100644 docs/paper-2-plan.md diff --git a/docs/paper-2-plan.md b/docs/paper-2-plan.md new file mode 100644 index 00000000..474414be --- /dev/null +++ b/docs/paper-2-plan.md @@ -0,0 +1,115 @@ +# Paper two: plan and evidence map + +Written 2026-09-06 against master `d5656e6` (1.0.0-beta.19). Companion to the CIDR 2027 +submission (beta.6, submitted for the 2026-08-04 deadline; notification 2026-10-06). + +## Thesis candidates + +The CIDR paper argued *compile-time engine specialization*. It deferred (§5, §7): the lock +discipline scaling comparison, crash consistency, formal checking of the protocols, the cost +of monomorphization, and external baselines beyond redb/LMDB. Paper two should be the paper +those deferrals point at, not a re-statement of the thesis. + +**A. Lifecycle paper (recommended).** "Row lifecycle in a specialized engine: exact-cell +locking, epoch reclamation, and reactive vacuum without a transaction manager." Everything +in it has landed since beta.6 and has numbers. Contributions: + +1. Exact-cell synchronization replacing hashed stripes and the table-global barrier + (`src/in_memory/data.rs` `CellLocks`; `docs/versioned-row-publication.md`). +2. Quiescent-state reclamation via a one-word `!Send` guard (`ps-reclaim`), replacing a + global reader counter; the `Send`-guard use-after-free found on the way is a good + cautionary section (`docs/TODO.md` "ps-reclaim 0.1.1"). +3. Reactive vacuum planned from the free-range registry, gated by a mutation lease and a + quiet epoch; root-cause note that all four reclaim bugs came from indexes storing + physical addresses (`docs/vacuum-design-directions.md`, `src/table/vacuum/`). +4. The 1-32 thread grid across three index backends: the lock-discipline scaling result the + CIDR paper promised (`docs/beta18-validation.md`). +5. `partition_by` with the loom model of the slot protocol (`src/partition/loom_tests.rs`), + the first model-checked component; HFT review that produced `partition_ref` + (`wt-review.md`). + +**B. Persistence paper.** "Index persistence by replaying the tree's own CDC stream, and what +it costs." Needs the durability work first: structural CDC is ~28% of a 126 ns insert +(`docs/wti-dirty-generation-persistence-plan.md`); ART logical WAL exists +(`src/persistence/space/art_index.rs`) but data pages and WTI still end at `flush()`; the +watermark/fsync design is proposal only (`docs/durability-visibility-proposal.md`). Not +ready before a Q1 2027 deadline unless the journal lands. + +**C. Schema-as-IR paper (PL venue).** `worktable_dsl`: schema read as data, diff with a +per-change cost model, declarations baked into generated code, `wt-dsl` CLI, migration +engine with on-disk version detection (`dsl/`, `docs/migration.md`). Fits PEPM / OOPSLA +better than a DB venue; overlaps with the PEPM plan in the other session. + +## Evidence already in hand (all M4 Max, local trees; needs a pinned Linux rerun) + +| Claim | Number | Source | +|---|---|---| +| Hot-page writer, exact-cell vs hashed | 322 ns vs 1,536 ns | beta18-validation | +| Per-page vs table-global barrier | +25% @4, +65% @8 disjoint writers | versioned-row-publication | +| Generation retire, beta.18 vs 15 | 33.6x (WTI), 16.1x (Arctic) | beta18-validation | +| Read scaling best/1T at 16 threads | 3.6x / 3.3x / 2.6x per backend | beta18-validation | +| 10%-write mix ceiling | peaks at 4 threads | beta18-validation (also beta.13/15) | +| Reactive vacuum foreground penalty | -1.1..-5.1% vs 16-41% unpaced | beta17/18-validation | +| Vacuum reclamation | 18/18 cells, 196 pages, 100% | beta17-validation | +| Point lookup vs stripped index | 15.65 ns vs 9.25 ns bare Arctic vs 33.5 Vec+BTreeMap | beta18-validation | +| Memory overhead | 0.14 B/row over control | beta18-validation | +| Partition route | 0.73 ns Vec vs 9.5 ns string hash; `partition_ref` 3.35 ns | TODO.md, partition docs | +| Persisted insert, beta.15 to 18 | -33..-42% (Arctic 6,130 to 3,753 ns/row) | beta18-validation | +| CDC share of insert | ~35 ns of 126 ns | wti-dirty-generation plan | + +Not in hand: `paper-bench/results/` (never committed), `compile_cost.sh` never run, no +sled/SQLite/DashMap baselines, no beta.19 rerun of Table 2. + +## Consumer workloads (surveyed 2026-09-06, all under ~/code) + +| Repo | Shape | What it gives the paper | Open? | +|---|---|---|---| +| `agentcode` | 11 tables, 8 persisted, Arctic on nearly every index, u128 keys. Stress = 8 concurrent 800-file `update_latency` processes. | The concurrency bug that motivates the paper: `docs/known-defects.md:70-137`, torn/corrupt page header in secondary-index batch apply at beta.11, 6/8 runs failing, 28/28 after moving to Arctic. Map it to the beta.18 fix ("torn reads and premature physical-link reuse") and show the same harness clean on beta.19. Also: Arctic vs WTI at 20k rows, insert 2.18M/s vs 0.89M/s, lookup 17.0M/s vs 5.0M/s (`docs/benchmarks/index-backends.md`); state 42.4 to 22.2 MB after u128 keys (`state-growth.md`); the request for a non-unique fixed-width index that became `ArcticMultiIndex`. | Proprietary | +| `agencyzero` | 19 persisted tables, String PKs, migration engine, `LoadMode::Recovery`, single-writer flock, QA fixture of 248 projects (~30 MB store). | The reclamation case study: `docs/store-recovery.md` records four production corruptions, including a variable-width index page that forgot fragmentation across restart (174 live entries, 2,664 B dead, 64 B tail overlap) and beta.5 whole-row rebuilds churning `pr_project_idx` (38 disagreeing rows) fixed by in-place updates. Production migrations via schema fingerprint. This is the "why indexes must not store physical addresses" story with real data. | Private (GitHub) | +| `karen` | 2 persisted tables, tiny (~700 rows). | Row-level, queryable, durable learned session state instead of one blob: the per-turn Confirm write-through gives 15-20 points top-1. One paragraph of motivation, not evaluation. Uses `unload_gracefully`. | Closed | +| `ekopathrs` | No `worktable!` at all. Uses `worktable-vec::AtomicKeyTable` for two in-memory profiling tables. | The honest negative: `docs/STORAGE-REVIEW.md` rejects full WorkTable (318-package resolve, no `no_std`) for 399 entries. Cite as the boundary of the design space; `worktable-vec` is the lock-free, `no_std` sibling. | Private | + +Use agentcode as the headline stress workload in §4 alongside the shadow-state harness; use +agencyzero as the recovery/fragmentation case study; mention karen and ekopathrs in one +paragraph each in the experience section. Get written OK before naming private repos. + +## Gaps to close for option A + +- Rerun the beta.18 grid and `paper-bench` on a quiet pinned x86 box; commit `results/`. +- Lock-discipline ablation as a proper figure: field vs row vs table lock, 1-32 threads, + skewed keys (`paper-bench/src/bin/contention`). +- Semi-formal statement of the cell-lock + reclamation invariants; ideally extend loom + beyond partitions to the cell/retire path (the CIDR reviewers will ask). +- Wart sweep: 9 `todo!()` sites remain (`codegen/.../queries/in_place.rs`, `update.rs`, + `src/features/s3_support.rs`), `Avaiable` typo in 2 files. +- Merge or explicitly exclude `feat/columnar-fields-indexes` (branch, Aug 6, unmerged). + +## Target: EDBT 2027, 3rd cycle (verified 2026-09-06) + +- Submission **2026-10-07, 5pm PST** (31 days out). Author feedback 11-19, notification + Acc/Rej/Revise 12-05, revised paper 2027-01-04, final 01-27, camera-ready 02-10. + Conference Lille, April 6-9, 2027. +- Paper types: Research long (12p) or short (6p, title prefixed "[Short Paper]"), + Experiments & Analysis, Vision (6p). Topics list includes "Concurrency control, recovery, + and transaction management", "Storage, indexing, and physical database design", + "Data management on modern hardware", "Benchmarking and performance evaluation". +- The revise cycle matters: a paper that gets "revise" on Dec 5 has until Jan 4 to add + the pinned-Linux rerun, so the Oct 7 draft can ship on the M4 grid with the caveat stated. +- CIDR notification is Oct 6, one day before: paper two cannot depend on the outcome and + must not overlap the CIDR text (still under review until then). Option A is disjoint by + construction; cite the CIDR paper as "under submission". + +Alternatives if A slips: ICDE 2027 R2 (2026-11-11), PVLDB rolling (monthly to 2027-03-01), +SIGMOD R4 (2026-10-17). DaMoN 2027 CFP not posted. + +## 31-day schedule for option A (long paper) + +| Week | Dates | Deliverable | +|---|---|---| +| 1 | Sep 7-13 | Freeze the claim list. Run `paper-bench` contention (field/row/table/inplace, 1-32 tasks) and beta.18 grid on the pinned Linux box if available, else M4 with three rotated passes; commit `results/`. Decide long vs short by Sep 13 based on whether the scaling figure holds. | +| 2 | Sep 14-20 | Draft §2 protocols (cell lock, retire, vacuum lease) with invariants stated; §3 partition + loom. Wart sweep PR (`todo!()`, `Avaiable`). | +| 3 | Sep 21-27 | Draft §4 evaluation from `results/`; figures; related work (Hekaton, epoch/QSBR: Fraser, Hart et al., Bw-tree, OLC, DaMoN vacuum/compaction lineage). | +| 4 | Sep 28-Oct 4 | Full read-through, internal review, page trim to 12. | +| 5 | Oct 5-7 | Buffer. Submit by Oct 6 evening local time (Oct 7 5pm PST is 07:00 Oct 8 in Bangkok, but do not use it). | + +Short-paper fallback (6p): contributions 1-3 only, one scaling figure, one vacuum figure. From 1cd8301be257ab38124e37939fabff62f4128579 Mon Sep 17 00:00:00 2001 From: meh Date: Mon, 7 Sep 2026 17:12:45 +0700 Subject: [PATCH 4/4] Cut the page run with as_chunks, which is what CI's clippy asks for `chunks_exact` with a constant chunk size is a lint on a newer clippy than the one installed here, so the local run was clean and CI was not. `as_chunks` also gives fixed-size arrays rather than slices, which is what the loop wanted. --- tests/slotted_page_requirement.rs | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/slotted_page_requirement.rs b/tests/slotted_page_requirement.rs index 32af729f..8b21460d 100644 --- a/tests/slotted_page_requirement.rs +++ b/tests/slotted_page_requirement.rs @@ -153,7 +153,8 @@ async fn a_data_page_says_where_its_rows_are() { // layout would do; what matters is that something in the page delimits the // rows. Today the tail is write padding, so this reads zero. let mut described = 0usize; - for page in bytes.chunks_exact(PAGE_SIZE).skip(1) { + let (pages, _) = bytes.as_chunks::(); + for page in pages.iter().skip(1) { let mut tail = [0u8; 4]; tail.copy_from_slice(&page[PAGE_SIZE - 4..]); described += u32::from_le_bytes(tail) as usize;