Skip to content

Require arctic 0.1.11 and ps-reclaim 0.1.4, and write down the beta.20 page requirement - #97

Closed
pathscale wants to merge 4 commits into
masterfrom
deps/arctic-0.1.11
Closed

Require arctic 0.1.11 and ps-reclaim 0.1.4, and write down the beta.20 page requirement#97
pathscale wants to merge 4 commits into
masterfrom
deps/arctic-0.1.11

Conversation

@pathscale

@pathscale pathscale commented Sep 7, 2026

Copy link
Copy Markdown
Owner

One PR, one repo. Three commits, linear, no merge commit.

1. The dependency floors

^0.1 was already resolving both of these. Raising the floor says they are required, which is what a correctness fix means.

ps-reclaim 0.1.4 closes a use-after-free. No lock was held across the participant scan and the extraction of garbage, so a retirement published between the two was judged against a decision taken before it existed, and a live reader's object could be reclaimed out from under it. The fix is a sequence cutoff captured under the garbage mutex before the scan. It has a deterministic regression test that fails on the old code.

Bounded drains were quadratic. extract_if(..).take(k) compacts the unchecked tail on drop, so draining N in batches of k moved about N²/k records while holding the garbage mutex. At WorkTable's advance_up_to(256):

backlog before after
32,000 3.00 ms 0.25 ms
128,000 23.29 ms 1.28 ms

The trigger is concrete here: a range scan pins for the whole traversal while writers retire, so backlog is retire-rate times hold-time. A ten-second scan at 8,000 retirements/sec built ~80,000 and spent ~10 ms inside that mutex.

arctic 0.1.11 changes nothing for this crate — WorkTable links std and takes ps-reclaim directly with default features, so unification gives it std either way:

arctic-wt v0.1.11 [smr-ps-reclaim]
ps-reclaim v0.1.4 [default,libc,spin,std]

The floor matters for what sits downstream: pathscale/WorkTable-vec#4 is the consumer that actually needs the decoupling.

2. What a data page has to say about itself

beta.19 changed the on-disk format and every .wt.data on this machine was thrown away and rebuilt on 6 September 2026, because nothing could read the old shape. That is a regeneration event, and it does not have to happen again.

The reason it did: a data page cannot be read without the index that points into it.

pub struct DataPage<const DATA_LENGTH: usize> {
    pub length: u32,
    pub data: [u8; DATA_LENGTH],
}

Rows are bump-allocated into data and length is a high-water mark. There are no delimiters, so nothing can tell where one row ends and the next begins. empty_links_list in the SpaceInfoPage records freed ranges and is explicitly lossy — bound_empty_links_list truncates it when it outgrows the info page and logs "space leak, not corruption".

This is not asking for the schema again. SpaceInfoPage already carries row_schema, primary_key_fields and secondary_index_types, and ensure_schema refuses a mismatch by name. A reader already knows how to decode a row. What it cannot do is find one.

tests/slotted_page_requirement.rs states that as two assertions:

test status
a_store_reopens_without_being_rebuilt passes — pins the behaviour we have
a_data_page_says_where_its_rows_are #[ignore = "beta.20: a data page carries no row directory"]

The ignored one is the requirement, written so that the day a page carries an (offset, length) directory growing down from the end, deleting one attribute is the whole proof — rather than someone re-deriving the argument from a changelog entry.

3. Paper two's evidence map

docs/paper-2-plan.md. The CIDR submission (beta.6, notification 2026-10-06) deferred the lock-discipline scaling comparison, crash consistency, formal checking of the protocols, the cost of monomorphization, and baselines beyond redb and LMDB. This plans the paper those deferrals point at and pins each candidate contribution to the code and measurements that already back it, so the writing starts from what has landed instead of from an outline.

Checks

cargo test                     927 passed, 0 failed, 5 ignored
cargo clippy --all-targets     clean

No version bump: nothing here changes WorkTable's API surface, and beta.19 is already published.

ps-reclaim 0.1.4 closes a use-after-free. A retirement published between the
participant scan and the extraction of garbage was judged against a decision
taken before it existed, so a live reader's object could be reclaimed under it.
The fix is a sequence cutoff captured under the garbage mutex before the scan.
It also stops bounded drains being quadratic: `extract_if(..).take(k)` compacts
the unchecked tail on drop, so a 128,000 backlog at `advance_up_to(256)` moved
23.29 ms of records under that mutex and now moves 1.28 ms.

Both were already being picked up by resolution, because `^0.1` allows them.
Raising the floors says they are required rather than merely permitted, which
is what a correctness fix means.

arctic 0.1.11 is the release where `smr-ps-reclaim` stopped implying `std`.
It changes nothing here - WorkTable links `std` and takes ps-reclaim directly
with default features, so feature unification gives it `std` either way - but
the floor is what lets a no_std consumer downstream rely on it.

    cargo test    927 passed, 0 failed
    cargo clippy --all-targets    clean
meh added 2 commits September 7, 2026 16:57
beta.19 changed the on-disk format and every `.wt.data` on this machine had to
be thrown away and rebuilt on 6 September 2026, because nothing could read the
old shape. That is a regeneration event, and the reason it happened is that a
data page cannot be read without the index that points into it.

This test states the requirement for beta.20 as an executable assertion rather
than a paragraph in a release note: a page that describes itself can be read by
a reader that has never seen the writer's index.
The CIDR submission deferred the lock-discipline scaling comparison, crash
consistency, protocol checking, the cost of monomorphization, and baselines
beyond redb and LMDB. This plans the paper those deferrals point at, and pins
each candidate contribution to the code and the measurements that back it,
so the writing starts from what has landed rather than from an outline.
@pathscale pathscale changed the title Require arctic 0.1.11 and ps-reclaim 0.1.4 Require arctic 0.1.11 and ps-reclaim 0.1.4, and write down the beta.20 page requirement Sep 7, 2026
`chunks_exact` with a constant chunk size is a lint on a newer clippy than the
one installed here, so the local run was clean and CI was not. `as_chunks`
also gives fixed-size arrays rather than slices, which is what the loop wanted.
pathscale pushed a commit that referenced this pull request Sep 9, 2026
Collapses what was PR #102 onto the arctic, event-ledger and page-list
work already on this branch, so WorkTable carries one pull request. #97,
#99 and #100 are folded in; #58 is not, being 224 commits behind master
and already conflicted, which is its own job.

`fsx` and every persistence signature go through `nagoya::io`, so the
production path names no runtime. A persisted table can set `page_size`,
which was refused before because the seeks computed offsets from a crate
constant while the generated table threaded the configured one.

Four places where the folded branches disagreed, each resolved by keeping
both rather than either:

* `arctic` and `ps-reclaim` keep #97's raised version floors and gain the
  `default-features = false` the no_std work needs.
* `allocated_bytes` keeps the page list as #100 left it and takes
  `core::mem` from the no_std work; the `.load()` in the older branch
  belonged to a shape #100 replaced.
* `batch.rs` and `task.rs` take the `core`/`alloc` imports, plus the `Arc`
  and `Location` the no_std lists had dropped and the code still uses.
* The s3 test goes back to tokio's extension traits. It talks to a tokio
  `TcpStream` it starts itself, so the sweep that took the storage path
  off tokio should never have touched it.

`event_ledger` is new here and was written against `std`. Its bookkeeping
moves to `core` and `alloc`; only the two parts that genuinely need an
operating system are gated, the backtrace capture and reading
`WT_EVENT_LEDGER`, so without `std` the ledger is simply never enabled.

`futures/std` goes in this crate's own `std` feature rather than on the
dependency line, where a `--no-default-features` build would still have
turned it on.

    cargo test --workspace --all-targets                  933 passed
      ... --all-features                                  935 passed
      ... --features versioned-row-publication            1029 passed
    cargo clippy --workspace --all-targets [--all-features] clean
    cargo check --no-default-features                     clean
@pathscale

Copy link
Copy Markdown
Owner Author

Folded into #102, which now carries this work plus the tokio::fs removal and tunable page stride, rebased linear with no merge commit. WorkTable carries one PR for this chain.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant