Require arctic 0.1.11 and ps-reclaim 0.1.4, and write down the beta.20 page requirement - #97
Closed
pathscale wants to merge 4 commits into
Closed
Require arctic 0.1.11 and ps-reclaim 0.1.4, and write down the beta.20 page requirement#97pathscale wants to merge 4 commits into
pathscale wants to merge 4 commits into
Conversation
ps-reclaim 0.1.4 closes a use-after-free. A retirement published between the
participant scan and the extraction of garbage was judged against a decision
taken before it existed, so a live reader's object could be reclaimed under it.
The fix is a sequence cutoff captured under the garbage mutex before the scan.
It also stops bounded drains being quadratic: `extract_if(..).take(k)` compacts
the unchecked tail on drop, so a 128,000 backlog at `advance_up_to(256)` moved
23.29 ms of records under that mutex and now moves 1.28 ms.
Both were already being picked up by resolution, because `^0.1` allows them.
Raising the floors says they are required rather than merely permitted, which
is what a correctness fix means.
arctic 0.1.11 is the release where `smr-ps-reclaim` stopped implying `std`.
It changes nothing here - WorkTable links `std` and takes ps-reclaim directly
with default features, so feature unification gives it `std` either way - but
the floor is what lets a no_std consumer downstream rely on it.
cargo test 927 passed, 0 failed
cargo clippy --all-targets clean
added 2 commits
September 7, 2026 16:57
beta.19 changed the on-disk format and every `.wt.data` on this machine had to be thrown away and rebuilt on 6 September 2026, because nothing could read the old shape. That is a regeneration event, and the reason it happened is that a data page cannot be read without the index that points into it. This test states the requirement for beta.20 as an executable assertion rather than a paragraph in a release note: a page that describes itself can be read by a reader that has never seen the writer's index.
The CIDR submission deferred the lock-discipline scaling comparison, crash consistency, protocol checking, the cost of monomorphization, and baselines beyond redb and LMDB. This plans the paper those deferrals point at, and pins each candidate contribution to the code and the measurements that back it, so the writing starts from what has landed rather than from an outline.
`chunks_exact` with a constant chunk size is a lint on a newer clippy than the one installed here, so the local run was clean and CI was not. `as_chunks` also gives fixed-size arrays rather than slices, which is what the loop wanted.
pathscale
pushed a commit
that referenced
this pull request
Sep 9, 2026
Collapses what was PR #102 onto the arctic, event-ledger and page-list work already on this branch, so WorkTable carries one pull request. #97, #99 and #100 are folded in; #58 is not, being 224 commits behind master and already conflicted, which is its own job. `fsx` and every persistence signature go through `nagoya::io`, so the production path names no runtime. A persisted table can set `page_size`, which was refused before because the seeks computed offsets from a crate constant while the generated table threaded the configured one. Four places where the folded branches disagreed, each resolved by keeping both rather than either: * `arctic` and `ps-reclaim` keep #97's raised version floors and gain the `default-features = false` the no_std work needs. * `allocated_bytes` keeps the page list as #100 left it and takes `core::mem` from the no_std work; the `.load()` in the older branch belonged to a shape #100 replaced. * `batch.rs` and `task.rs` take the `core`/`alloc` imports, plus the `Arc` and `Location` the no_std lists had dropped and the code still uses. * The s3 test goes back to tokio's extension traits. It talks to a tokio `TcpStream` it starts itself, so the sweep that took the storage path off tokio should never have touched it. `event_ledger` is new here and was written against `std`. Its bookkeeping moves to `core` and `alloc`; only the two parts that genuinely need an operating system are gated, the backtrace capture and reading `WT_EVENT_LEDGER`, so without `std` the ledger is simply never enabled. `futures/std` goes in this crate's own `std` feature rather than on the dependency line, where a `--no-default-features` build would still have turned it on. cargo test --workspace --all-targets 933 passed ... --all-features 935 passed ... --features versioned-row-publication 1029 passed cargo clippy --workspace --all-targets [--all-features] clean cargo check --no-default-features clean
Owner
Author
|
Folded into #102, which now carries this work plus the tokio::fs removal and tunable page stride, rebased linear with no merge commit. WorkTable carries one PR for this chain. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
One PR, one repo. Three commits, linear, no merge commit.
1. The dependency floors
^0.1was already resolving both of these. Raising the floor says they are required, which is what a correctness fix means.ps-reclaim 0.1.4 closes a use-after-free. No lock was held across the participant scan and the extraction of garbage, so a retirement published between the two was judged against a decision taken before it existed, and a live reader's object could be reclaimed out from under it. The fix is a sequence cutoff captured under the garbage mutex before the scan. It has a deterministic regression test that fails on the old code.
Bounded drains were quadratic.
extract_if(..).take(k)compacts the unchecked tail on drop, so draining N in batches of k moved about N²/k records while holding the garbage mutex. At WorkTable'sadvance_up_to(256):The trigger is concrete here: a range scan pins for the whole traversal while writers retire, so backlog is retire-rate times hold-time. A ten-second scan at 8,000 retirements/sec built ~80,000 and spent ~10 ms inside that mutex.
arctic 0.1.11 changes nothing for this crate — WorkTable links
stdand takes ps-reclaim directly with default features, so unification gives itstdeither way:The floor matters for what sits downstream: pathscale/WorkTable-vec#4 is the consumer that actually needs the decoupling.
2. What a data page has to say about itself
beta.19 changed the on-disk format and every
.wt.dataon this machine was thrown away and rebuilt on 6 September 2026, because nothing could read the old shape. That is a regeneration event, and it does not have to happen again.The reason it did: a data page cannot be read without the index that points into it.
Rows are bump-allocated into
dataandlengthis a high-water mark. There are no delimiters, so nothing can tell where one row ends and the next begins.empty_links_listin theSpaceInfoPagerecords freed ranges and is explicitly lossy —bound_empty_links_listtruncates it when it outgrows the info page and logs "space leak, not corruption".This is not asking for the schema again.
SpaceInfoPagealready carriesrow_schema,primary_key_fieldsandsecondary_index_types, andensure_schemarefuses a mismatch by name. A reader already knows how to decode a row. What it cannot do is find one.tests/slotted_page_requirement.rsstates that as two assertions:a_store_reopens_without_being_rebuilta_data_page_says_where_its_rows_are#[ignore = "beta.20: a data page carries no row directory"]The ignored one is the requirement, written so that the day a page carries an
(offset, length)directory growing down from the end, deleting one attribute is the whole proof — rather than someone re-deriving the argument from a changelog entry.3. Paper two's evidence map
docs/paper-2-plan.md. The CIDR submission (beta.6, notification 2026-10-06) deferred the lock-discipline scaling comparison, crash consistency, formal checking of the protocols, the cost of monomorphization, and baselines beyond redb and LMDB. This plans the paper those deferrals point at and pins each candidate contribution to the code and measurements that already back it, so the writing starts from what has landed instead of from an outline.Checks
No version bump: nothing here changes WorkTable's API surface, and beta.19 is already published.