Take WorkTable off tokio::fs, let a persisted table choose its page size, and drop prettytable by pathscale · Pull Request #102 · pathscale/WorkTable

pathscale · 2026-09-09T03:59:43Z

One PR for the repo. Supersedes #101, whose branch is an ancestor of this one.

Needs pathscale/DataBucket#75, which this cannot compile without.

Off `tokio::fs`

src/fsx.rs holds the file type and the operations no IO trait carries: sync_all, sync_data, set_len, metadata, and the openers. The whole crate goes through it, so swapping backends is swapping one file, which is how these numbers were taken. It is std::fs behind AllowStdIo, which belongs to no runtime.

Three arms against perf-benchmarks, medians of 7 and 9 interleaved, each arm verified to link a distinct checkout:

	before (#101)	this	total vs master	without this
reopen arctic	5.72 ms	0.380x	0.357x	0.939x
reopen wti-84char	17.87 ms	0.568x	0.568x	1.000x
index-ops insert arctic	8,554,016/s	1.031x	1.087x	1.031x
index-ops lookup arctic	34,334,764/s	0.997x	1.006x	0.995x
index-ops insert wti	2,331,172/s	1.027x	1.025x	1.012x
index-ops lookup wti	5,286,111/s	0.993x	0.988x	1.024x

Cold reopen is 2.8x faster on arctic and 1.8x on wti. The four in-memory metrics move between 0.988x and 1.087x against a null arm spanning 0.995x to 1.031x, so they are flat: the win is IO, which is what taking tokio::fs off the read path should do.

Because the calls block, a persistence engine should own a thread rather than share a runtime's worker pool. They were never waiting on the disk through a runtime anyway: the persistence path measures 89 voluntary context switches across 25,000 inserts.

`page_size` works for a persisted table

It was refused because the seeks computed every offset from a hardcoded constant while the generated table threaded the configured one. Both take the stride as a parameter now. What remains is a 512-byte floor, since a page on disk carries a 28-byte header.

Three things had to be found before a table at 8192 would reload, and every one was silent.

SpaceLogicalIndex and its three siblings wrapped SpaceIndex without passing a stride, and the = DEFAULT_PAGE_STRIDE default I had put on that parameter made it compile. The index file was written at 16384 and read at 8192, and the only symptom was a page failing to parse a long way from the cause. So the defaults are gone. Every instantiation names its stride and a wrapper that forgets to thread it no longer compiles, which turned the remaining two into compiler errors instead of a hunt.

The other two: a slot bounds check against the crate default (fixed in DataBucket#75), and one of two WorkTable<...> emissions in the in-memory generator hardcoding INNER_PAGE_SIZE where the other used the table's constant.

tests/persistence/custom_page_size.rs writes a persisted table at 8192, requires the file to span several pages of that size, and reloads every row. The length check is the half that matters: a table that fell back to the default would still read its own writes.

`no_std`: the crate's own source, and one dependency gone

#![cfg_attr(not(feature = "std"), no_std)] behind a default-on std feature. This does not make the crate build for a bare-metal target; eleven dependencies still fail, and the list is in the thread below.

One of them is gone. SystemInfo's Display is a diagnostic dump with one caller in the repository that does not print it, and prettytable-rs was deciding whether this crate can be embedded: it reaches csv, then memchr, which fails without std in 505 places. Every alternative measured is no better, so the columns are padded by hand. Twelve crates leave with it, 127 in the graph to 115.

Documentation

docs/page-size.md lists every location across the four crates that decides a page size, and the three bugs above, because the next such parameter will hide in the same kind of place.

docs/wt-user-guide.typ and its PDF. The declaration and calls it opens with are examples/guide_check.rs, compiled and run by the ordinary build, which immediately caught the guide claiming insert was synchronous.

Verified

927 tests pass, clippy is clean, and --no-default-features is clean.

🤖 Generated with Claude Code

ps-reclaim 0.1.4 closes a use-after-free. A retirement published between the participant scan and the extraction of garbage was judged against a decision taken before it existed, so a live reader's object could be reclaimed under it. The fix is a sequence cutoff captured under the garbage mutex before the scan. It also stops bounded drains being quadratic: `extract_if(..).take(k)` compacts the unchecked tail on drop, so a 128,000 backlog at `advance_up_to(256)` moved 23.29 ms of records under that mutex and now moves 1.28 ms. Both were already being picked up by resolution, because `^0.1` allows them. Raising the floors says they are required rather than merely permitted, which is what a correctness fix means. arctic 0.1.11 is the release where `smr-ps-reclaim` stopped implying `std`. It changes nothing here - WorkTable links `std` and takes ps-reclaim directly with default features, so feature unification gives it `std` either way - but the floor is what lets a no_std consumer downstream rely on it. cargo test 927 passed, 0 failed cargo clippy --all-targets clean

beta.19 changed the on-disk format and every `.wt.data` on this machine had to be thrown away and rebuilt on 6 September 2026, because nothing could read the old shape. That is a regeneration event, and the reason it happened is that a data page cannot be read without the index that points into it. This test states the requirement for beta.20 as an executable assertion rather than a paragraph in a release note: a page that describes itself can be read by a reader that has never seen the writer's index.

The CIDR submission deferred the lock-discipline scaling comparison, crash consistency, protocol checking, the cost of monomorphization, and baselines beyond redb and LMDB. This plans the paper those deferrals point at, and pins each candidate contribution to the code and the measurements that back it, so the writing starts from what has landed rather than from an outline.

`chunks_exact` with a constant chunk size is a lint on a newer clippy than the one installed here, so the local run was clean and CI was not. `as_chunks` also gives fixed-size arrays rather than slices, which is what the loop wanted.

`scripts/ci-local.sh` runs `cargo fmt --all --check` as its first step and CI did not, which made the local script stricter than CI rather than equal to it, so formatting drift could reach master unnoticed. Same command, same arguments. Measured before adding it: the tree already passes, exit 0 with no diffs, so this job is green as written rather than a red first run to clean up.

The log carried beta.18 and beta.19 and nothing before them, so the history a consumer needs in order to judge an upgrade was only in the commits. Entries are derived from this repository's own history.

`BatchOperation::validate` refuses an event stream with a hole in it. A hole is normally transient, and one that survives the whole deferral budget is not, but the stall message could not tell the two apart: it reported the range and nothing else. Two very different bugs produce that line. Either an id was assigned by the index and its event was dropped instead of queued, in which case nothing will ever deliver it, or the operation was queued and batch collection keeps assembling batches that exclude it. Distinguishing them needs a record of what was queued. Assignment happens inside the index and cannot be hooked from here, but every event reaching persistence passes through `Queue`, so an id present in the stream and absent from the ledger was assigned and never queued. That is the leak signature. Producers are named by `std::panic::Location`, which is why `push_at` takes the caller location and why the old `push` and `push_many` wrappers are deleted rather than kept: calling them loses exactly the location the ledger exists to record. This observes and does not fix. `docs/TODO.md` records the three leak sites the instrument points at, all in generated code: the `NotFound` arm of the update query returns without building an acknowledge while its `AlreadyExists` sibling builds one, and two `res?;` sites in update and delete propagate before acknowledging. The TODO's congee section is corrected while here. The crossbeam-epoch port was already done in congee-wt 0.4.4, and it was a design change rather than a rename: per-tree `Domain`, batched retires, guard provenance checked, and a new `Guard` that is `!Send` and tree-scoped, which constrains future callers.

The ledger from the previous commit points at three sites in generated persisted query code where an event the index had already assigned an id to was dropped instead of queued. A dropped event is a hole `BatchOperation::validate` refuses forever, so the table stalls and the message names a range rather than a cause. All three now do what the rollback arms already did: build an `Acknowledge` carrying the orphaned events and apply it before propagating. The `IndexError::NotFound` arm of the update insert path acknowledges the events its sibling `AlreadyExists` arm was already acknowledging, and the two bare `res?` sites, in `gen_process_diffs_remove_on_index` and after `delete_row_cdc`, became `if let Err(e) = res` blocks that acknowledge and then return. The events are moved into the acknowledge rather than cloned, and that is load-bearing rather than a preference. Cloning does not compile: the events type is still an inference variable at that point, pinned only by the `op.extend_secondary_key_events` call further down, and method resolution for `.clone()` needs the type known where the call is written. It fails as an `E0282` reported against the `worktable!` invocation with no inner span, which is expensive to diagnose. The comment in `docs/TODO.md` records that so nobody pays for it twice. The write failures are not forcible through the public API, so the wiring is pinned on the emitted tokens the way the surrounding tests already do it. Both assertions check the acknowledge is emitted *before* the return or the extend, not merely that one appears somewhere in the output. The in-memory generator has the same two `NotFound` arms and is deliberately untouched: there is no persistence stream behind them to gap.

`DataPages::pages` was a `Vec` behind an `ArcSwap`, and both append paths cloned the whole thing to add one page. Every page in it is an `Arc`, so the clone was one atomic increment per existing page, each touching a separately allocated page header: a cache miss apiece. Appending page N cost O(N), and filling a table cost O(N^2). It hid behind row size, because row size is what decides how many pages a table has. At 256 bytes a page holds sixty-odd rows and twenty thousand rows is three hundred pages, where the copy is invisible and per-row cost is flat. At 4 KiB a page holds three, the same rows are six thousand pages, and per-row cost climbed from 2.95 to 30.08 microseconds across the load. The pages now live in fixed-size chunks, so an append copies one chunk and the spine of chunk pointers rather than every page. Readers still take an `ArcSwap` snapshot and never block, and page access still goes through the directory first, so the read path is unchanged. Measured over twenty thousand 4 KiB rows, per-row cost goes flat - 1.53 to 1.95 microseconds against 2.95 to 30.08 - and inserting into an unpersisted table goes from 254 MB/s to 3062. A persisted table gains far less, 208 to 279 MB/s, because with the copy gone persistence is what the load now waits on. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The directory holds stable page pointers so a point access does not take an `ArcSwap` snapshot, and it reached 64 * 64 = 4,096 pages, which is 64 MiB at the default page size. Past that `publish` returned early and every access fell back to the snapshot. A table of 4 KiB rows crosses that after twelve thousand rows, which is not a large table. Raising it to 1,024 roots reaches 1 GiB for an 8 KiB array of pointers. On its own it made no measurable difference to insert cost, because the page list copy was the term that mattered; this moves a ceiling rather than removing a cost, and it is committed separately so the two are not confused. The tests are the harness the page list work was measured with: whether insert cost scales with row size, whether it grows as the table fills, what the floor is with few pages, and what the persistence path sustains end to end. They are all `#[ignore]`d, since they are measurements and not assertions. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

An insert that fits in the page already open and one that has to add a page are different operations with different costs, and averaging them hides the second behind the first. How much it hides depends on row size: at 256 bytes one insert in sixty allocates, so allocation is a tail event, while at 4 KiB one in three does and it is a third of all traffic. Split by whether the page count moved, the page list change reads as what it is - a tail-latency fix that leaves the common path alone. On 4 KiB rows the allocating population goes from 59.67 to 7.08 microseconds at p50, 133.38 to 14.46 at p99, and 451.54 to 42.33 at its worst, while the existing-page population does not move. On 256-byte rows the same change is worth about a third, on the one insert in sixty that pays it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The chunked page list handed back an owned `Arc` from `get`, which costs an atomic increment and the matching decrement on drop. The link-based read is a read path and needs the page only for the length of one call, so it paid both for nothing. It measured. A delete went from 665 to 751 ns, while a select by primary key - which goes through the page directory and never touches this - did not move. Borrowing through `with_page` puts delete back at 654 ns, and the criterion cases either side of it are unchanged or better: insert 442 -> 86 ns, select by primary key 21.0 -> 20.1 ns. Found by running the benchmark suite that already existed rather than the ad-hoc probes the rest of this work was measured with. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

It was named "bulk load", which reads as loading a table from disk. It inserts 25,000 rows. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Collapses what was PR #102 onto the arctic, event-ledger and page-list work already on this branch, so WorkTable carries one pull request. #97, #99 and #100 are folded in; #58 is not, being 224 commits behind master and already conflicted, which is its own job. `fsx` and every persistence signature go through `nagoya::io`, so the production path names no runtime. A persisted table can set `page_size`, which was refused before because the seeks computed offsets from a crate constant while the generated table threaded the configured one. Four places where the folded branches disagreed, each resolved by keeping both rather than either: * `arctic` and `ps-reclaim` keep #97's raised version floors and gain the `default-features = false` the no_std work needs. * `allocated_bytes` keeps the page list as #100 left it and takes `core::mem` from the no_std work; the `.load()` in the older branch belonged to a shape #100 replaced. * `batch.rs` and `task.rs` take the `core`/`alloc` imports, plus the `Arc` and `Location` the no_std lists had dropped and the code still uses. * The s3 test goes back to tokio's extension traits. It talks to a tokio `TcpStream` it starts itself, so the sweep that took the storage path off tokio should never have touched it. `event_ledger` is new here and was written against `std`. Its bookkeeping moves to `core` and `alloc`; only the two parts that genuinely need an operating system are gated, the backtrace capture and reading `WT_EVENT_LEDGER`, so without `std` the ledger is simply never enabled. `futures/std` goes in this crate's own `std` feature rather than on the dependency line, where a `--no-default-features` build would still have turned it on. cargo test --workspace --all-targets 933 passed ... --all-features 935 passed ... --features versioned-row-publication 1029 passed cargo clippy --workspace --all-targets [--all-features] clean cargo check --no-default-features clean

pathscale · 2026-09-09T09:27:20Z

Collapsed, 9 September

This now carries #97, #99 and #100 as well, rebased linear. 14 commits, no merge commit. Those three are closed.

#58 is deliberately not in here. It is from 5 August, 224 commits behind master, and already reports dirty. Folding a branch that stale into a persistence change is its own job, not part of this one.

Where the folded branches disagreed

Four conflicts, each resolved by keeping both sides rather than picking one. Three of them would have silently reverted work if taken wholesale:

arctic and ps-reclaim keep Require arctic 0.1.11 and ps-reclaim 0.1.4, and write down the beta.20 page requirement #97's raised floors (>=0.1.11, >=0.1.4) and gain the default-features = false the no_std work needs. Taking either side alone loses one of them.
allocated_bytes keeps the page list as Stop copying the page list every time a page is appended #100 left it and takes core::mem. The .load() on the other side belonged to a shape Stop copying the page list every time a page is appended #100 had replaced; the unconflicted line directly above it already read self.pages.len().
batch.rs and task.rs take the core/alloc import lists, plus Arc and Location, which those lists had dropped and the code still uses 2 and 8 times respectively.
The s3 test goes back to tokio::io's extension traits. It drives a tokio TcpStream from a TcpListener it starts itself, so the sweep that took the storage path off tokio should never have rewritten it. That one was mine.

`event_ledger` needed real work

#99's new file was written against std, after the no_std sweep, so nothing had reconciled them. Its bookkeeping moves to core and alloc, and only the two capabilities that genuinely need an operating system are gated: the Backtrace capture and reading WT_EVENT_LEDGER. Without std the ledger is never enabled, which is the honest answer for a diagnostic an environment variable turns on.

futures/std went into this crate's own std feature rather than onto the dependency line, where a --no-default-features build would still have turned it on. That is the same trap this branch already fixed for nine other dependencies.

QA, run as CI runs it

cargo test --workspace --all-targets                        933 passed, 0 failed
cargo test --workspace --all-targets --all-features         935 passed, 0 failed
cargo test --workspace --all-targets --features versioned-row-publication
                                                           1029 passed, 0 failed
cargo clippy --workspace --all-targets -- -D warnings        clean
cargo clippy --workspace --all-targets --all-features -- -D warnings clean
cargo check --no-default-features                            clean

It could not resolve anywhere before this

[patch.crates-io] now pins data_bucket, nagoya and ps-st3 by git. All three are needed here: [patch] only takes effect from the workspace root, so a dependency's own patches do not reach its consumer. data_bucket also moves to ^0.6, because the concrete error type and the stride parameter shipped as 0.6.0 and ^0.5 cannot resolve it.

Merge order

ps-st3 #4  ->  0.5.1  ->  nagoya #1  ->  0.1.0  ->  DataBucket #75  ->  0.6.0  ->  this

Delete the patch block entries as each publishes.

pathscale · 2026-09-09T20:29:32Z

Folded into #105 as commits 1 to 14, unchanged. #105 is based on master and contains this whole stack.

meh added 8 commits September 7, 2026 16:49

Backfill the change log to 0.3.10

89b5860

The log carried beta.18 and beta.19 and nothing before them, so the history a consumer needs in order to judge an upgrade was only in the commits. Entries are derived from this repository's own history.

This was referenced Sep 9, 2026

Concrete errors, one write per run of pages, and the page stride as a parameter pathscale/DataBucket#75

Open

Make WorkTable's own source std-free #101

Closed

pathscale changed the title ~~Go through fsx for the filesystem, and carry a page stride~~ Take WorkTable off tokio::fs, let a persisted table choose its page size, and drop prettytable Sep 9, 2026

meh and others added 6 commits September 9, 2026 16:03

Call the insert benchmark what it measures

f30eff4

It was named "bulk load", which reads as loading a table from disk. It inserts 25,000 rows. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

pathscale force-pushed the feat/tunable-page-stride branch from e898e4c to ac66ae0 Compare September 9, 2026 09:26

This was referenced Sep 9, 2026

Find and fix three persistence event-gap leak sites #99

Closed

Require arctic 0.1.11 and ps-reclaim 0.1.4, and write down the beta.20 page requirement #97

Closed

Stop copying the page list every time a page is appended #100

Closed

pathscale mentioned this pull request Sep 9, 2026

WorkTable 1.9.0-alpha1: off tokio and tokio::fs, columnar side indexes, and a schema-selected runtime #105

Open

pathscale closed this Sep 9, 2026

Navigation Menu

Take WorkTable off tokio::fs, let a persisted table choose its page size, and drop prettytable - #102

Take WorkTable off tokio::fs, let a persisted table choose its page size, and drop prettytable#102
pathscale wants to merge 14 commits into
masterfrom
feat/tunable-page-stride

pathscale commented Sep 9, 2026 •

edited

Loading

Uh oh!

pathscale commented Sep 9, 2026

Uh oh!

pathscale commented Sep 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

pathscale commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Off tokio::fs

page_size works for a persisted table

no_std: the crate's own source, and one dependency gone

Documentation

Verified

Uh oh!

pathscale commented Sep 9, 2026

Collapsed, 9 September

Where the folded branches disagreed

event_ledger needed real work

QA, run as CI runs it

It could not resolve anywhere before this

Merge order

Uh oh!

pathscale commented Sep 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

pathscale commented Sep 9, 2026 •

edited

Loading

Off `tokio::fs`

`page_size` works for a persisted table

`no_std`: the crate's own source, and one dependency gone

`event_ledger` needed real work