1.0.0-beta.17 - #87
Merged
Merged
Conversation
added 10 commits
September 2, 2026 14:07
A schema is written down once, in a `worktable!` invocation, and the parser that understands it lived in `worktable_codegen`, which is `proc-macro = true`. A proc-macro crate can export nothing but macros, so every type describing a schema — the columns, the primary key, the indexes, the queries — was unreachable from any other crate however public it was declared. `mod common` was not public at that crate's root either. So anything wanting to *read* a declaration had two options: re-implement the grammar and drift from it, or do without. A diagram, a migration tool, a documentation generator and an editor all want to read one. `lib.rs` has carried `// TODO: Refactor this codegen stuff because it's now too strange.` `model` and `parser` move to `worktable_dsl`, a plain library. Nothing in them changed; the dependencies are the five they already used, none added, none dropped. `worktable_codegen` now depends on it, so there is one grammar rather than a copy that can disagree with the compiler about what a schema means. `name_generator` stays in codegen. It invents Rust identifiers for generated code, which is not the schema language, and generators here define inherent `impl`s on `WorktableNameGenerator` — the orphan rule allows that only in the crate owning the type. I had it in the extracted crate first and the compiler made the same argument the design does. `crate::common::` still resolves, through a thin module that re-exports the new crate, so the 127 paths across 67 files are untouched and the diff stays a move rather than a sweep. An integration test reads a declaration from outside, which is the claim worth pinning: it compiles as its own crate, so it stops building if this ever becomes a proc-macro crate again. It also records a property no caller existed to depend on before. `Columns::columns_map` is a `std::collections::HashMap`, whose iteration order Rust randomises per process; two runs of the same input gave `["answered", "project_id", "id"]` and `["project_id", "answered", "id"]`. The macro never cared, and the parser's own tests collect it into another `HashMap` and assert membership, so nothing noticed. A consumer rendering columns in that order draws a different table every run. `field_positions` already carries the declaration order and is the field to sort by; the test asserts that, and says so, so the next consumer learns it here rather than by shipping the bug.
`pub use worktable_dsl::{Parser, *};` names `Parser` and then re-exports it
again through the glob, and rustc reports the glob as unused: nothing reaches
the shim that way, because every caller goes through `crate::common::model::`
or `crate::common::parser::`.
That is only a warning locally, which is why it survived the extraction. CI
runs `cargo clippy --workspace --all-targets -- -D warnings`, where it is a
build failure.
`worktable_dsl` can parse a declaration, which is half of what a designer needs. The other half is holding one, comparing it, storing it, and writing it back out as text the compiler accepts. `crate::model` cannot do any of that: it is built out of `Ident` and `TokenStream`, which is right for a thing whose job is to become Rust code and wrong for everything else. An `Ident` cannot be serialised or constructed outside a proc-macro context without a `Span::call_site` that lies about where it came from, and a `TokenStream` is not `PartialEq`, so two schemas cannot even be asked whether they differ. That is the one question a migration planner exists to answer. `schema::Schema` is the same declaration with the compiler's concerns removed: `String` for `Ident`, ordered `Vec`s for `HashMap`s, no spans. It derives `PartialEq` so schemas can be diffed, and under an optional `serde` feature it derives `Serialize`, so one can be stored next to the data it describes and read back by a process that never saw the Rust type. The feature is off by default because `worktable_codegen` depends on this crate and is a proc macro: every WorkTable user compiles it for the host before anything else in their build, and they should not pay for a derive macro that serves consumers who are not the compiler. The approach is additive. Nothing in the parser or the generators changed, and the model is untouched apart from three `cfg_attr` derives on plain enums the IR reuses rather than duplicates. Inverting the parser to produce plain data directly would have been a 61-file edit across 13k lines of generators with no test proving the output was unchanged. Two emitters. `to_dsl` renders the declaration body back to text, which is what makes a drawing editable: read, change, write, compile. `to_mermaid` renders UML class notation, chosen because it is text, so it is diffable and needs no rendering dependency, and because it renders anywhere Markdown does. Columns are attributes carrying their markers, queries are operations, and the partition key is a note rather than an attribute because it is stored once per partition and no query can name it. The schema language has no foreign keys, so `infer_relations` guesses links from a single stated naming rule and returns what it guessed; `schemas_to_mermaid` draws those as dependencies rather than associations, because a dashed arrow is the honest notation for a link the declaration does not make. Ordering is a guarantee here, not an accident. `columns_map` is a `HashMap`, whose iteration order Rust randomises, so a consumer walking it draws a different table on every run; `field_positions` records the declaration order and is what the IR sorts by. The query maps have no such field, so those are sorted by name, which is at least stable, and a test says which is which. The round trip is checked against the repository rather than against a fixture. `tests/round_trip.rs` finds all 128 `worktable!` invocations in the tree, sets aside the 12 that are `macro_rules!` templates full of metavariables, and asserts `parse(emit(parse(x))) == parse(x)` for the remaining 116. Those were written by people not thinking about this crate, which makes them a better corpus than anything written here. `codegen` adds the claim this crate cannot make about itself: that emitted text is a declaration the macro accepts. That check has to live on the near side of the proc-macro boundary. Three things the corpus turned up. The emitter writes no comma after `delete`, `in_place` or `config` blocks. `parse_updates` consumes one and those three do not, so a comma there arrives at a dispatch loop as a `,` token and dies as "Unexpected identifier". Omitting it is the only form all of them accept. Expanding one declaration twice does not produce one program. Several generators iterate `columns_map` directly to emit an ordered construct, the `RowFields` and `AvaiableTypes` enums among them, so the variant order differs between two expansions in one process and can differ between two compilations of the same source. `generator_determinism` records it with the evidence and is ignored rather than deleted: the fix changes the generated code of every table and deserves reviewing on its own. It is also why `emitted_declarations` can only assert that an emitted declaration expands, not that it generates identical code.
With a schema as data on both sides, the decision a version mismatch forces can
be computed instead of hand-written. `Diff::between` says what changed, `Cost`
says what applying it costs, and `transforms_required` says which parts a person
still has to write, which are the parts that need intent rather than mechanism.
The cost model is about links, not about fields. A row is addressed by a
`Link { page_id, offset, length }`, and every index holds links, so the question
that decides what a change costs is not how many columns moved but whether a row
is still where it was. Any change to the archived layout invalidates every link
in the table at once and there is no cheaper answer than writing every row
somewhere else. A change to an index invalidates nothing: the rows have not
moved and the index can be rebuilt from them. Hence three tiers and a fourth for
the changes no diff can settle.
That last tier is the point of the exercise. A changed primary key, a changed
partition key, a renamed table and a flipped `persist` are not expensive, they
are underdetermined, and the useful thing a planner can do is say so rather than
guess. The routing key is the clearest case: it is not in the row, so which
partition a row belongs to cannot be recomputed from the row, only from where
the row already is.
The planner invents a value only when there is exactly one it could be. Widening
a column to `optional` has one answer. Narrowing it does not, and neither does
adding a required column or changing a type, so each of those comes back as a
`TransformRequest` naming the column and why. A rename is reported as a drop and
an add, because nothing in a declaration distinguishes it from a deletion beside
an unrelated addition, and guessing by type equality would be wrong exactly when
it mattered.
Three things worth knowing that the tests state as claims. A version bump on its
own costs nothing, which is what keeps bumping cheap enough to be habitual.
Reordering columns is a layout change, because declaration order is the row
struct's field order: it is the change most likely to be made by accident and
least likely to look like one. And a schema that changed without a version bump
is still detected, at the cost of comparing two small structs and reading no
rows, which is the middle branch of the load state machine.
`plan` lifts the same comparison to a set of tables, matching by name because
that is how spaces are matched on disk. A dropped table is `NeedsIntent` rather
than free: whether to delete data is a decision, not a consequence of a
declaration.
A compiled binary could not say what schema it was built against. The
information was there at expansion time and then thrown away, so a migration
planner had no "declared" side to compare against what is on disk, and a
designer could not draw a diagram of an application whose source it did not
have.
Every generated table now carries a `<NAME>_SCHEMA` const, named after
`<NAME>_VERSION` because it answers the question next to it: the version says
which schema, and this says what that schema is.
The stored form is the DSL text rather than a serialised structure. It needs no
format decision, keeps serde out of the dependency graph of every user's build,
is legible in a hex dump, and is read back by the same parser that read the
original. It is also, being a declaration, exactly what regenerates an old table
type, which is the hand-maintained `version_tables: { 1 => v1::UserV1WorkTable }`
that makes migrations something people put off. `dsl/tests/round_trip.rs` holds
the property this rests on against all 116 declarations in this repository, and
the tests here check the emitted const against the declaration it came from and
that the macro accepts it back.
In-memory tables get it too. A designer reading a crate wants every table, and
the const costs a string either way.
Two details worth the words. The second parse runs at the end of `expand` rather
than the start, so this function's diagnostics stay the ones a bad declaration
produces: both parses reject the same inputs, but only one of them knows to say
that a separate `attributes` section is not part of the 1.0 grammar. And the
const is `allow(dead_code)`, because a `worktable!` inside a function body puts
it inside that body, where nothing refers to it and `-D warnings` would fail a
user's build over a const they never asked for.
A designer opening a project, a documentation generator and a migration tool
comparing a checkout against a running database all start from the same problem:
a schema is written inside a `worktable!` invocation somewhere in a crate, and
there is no index of where.
`declarations_in_source` walks tokens rather than `syn`'s item tree, because an
invocation inside a function body is not an item and real code puts them there.
An item walk would quietly miss a table, and the caller would never learn a table
existed to be missed. Both delimiter forms are accepted: the repository uses
`worktable!( .. )` 83 times and `worktable! { .. }` 45 times, and a reader that
took only one would be wrong about the language.
The return is not a `Vec<Schema>`. Some invocations are not declarations: a
`macro_rules!` body writing `name: $name, ... using $backend` is a template whose
metavariables stand for text that exists only after the outer macro expands, and
counting those as failures would be wrong. Everything else that fails to parse is
reported with the text that failed, because a designer that silently drops a
table the compiler accepts is worse than one that says it could not read it.
The corpus round-trip test now goes through this, so it is also the evidence that
the scanner finds what is there: 116 declarations read, 12 templates set aside,
nothing rejected, across the whole repository.
The example on `PartitionSet::pinned` was `ignore`, so it never compiled. It referenced `prices`, `batch` and `tick` without defining them, which is exactly why it could not run - and it is the only documentation the API has. An `ignore` example on a new public method is worse than none: it looks checked, and it rots silently against the very signature it documents. Made self-contained and executable - declares a partitioned table, inserts, then shows the pin-once-read-many shape the surrounding prose argues for.
Four dependencies carried `=` requirements: `data_bucket`, `WorkTablesIndex`, `indexset` and `worktable_codegen`. None of them needs to. `ps-reclaim` is the exception that was never locked. A bare `"0.1.0"` already means `^0.1.0`, so beta.16 picks up 0.1.1 on any fresh resolve. It moves to `"0.1.1"` to raise the floor above the version whose `Guard` was `Send`, which is a soundness bound rather than a lock: a resolve cannot land on the version with the use-after-free window at all. Caret on a `0.0.x` version describes the same set as `=`, so widening `WorkTablesIndex` changes nothing until that crate reaches 0.1.0. Removing the `=` still matters: the file stops implying a constraint it is not expressing. `worktable_codegen` is the one that gives something up. The exact pin held the macro and the runtime it generates calls into in lockstep, and a caret admits later betas. A mismatched pair fails at expansion in a consumer rather than as a resolver conflict here. `docs/TODO.md` moves the two items that were listed as blocking beta.16, and did not stop it, into a section recording how they closed. Re-measuring the partition regression stays open, now against 0.1.1 rather than against the guard size 0.1.1 removes, and the file says plainly which set of `partition_ref` numbers not to reuse and why.
The DSL extraction added a third publishable crate to this workspace and
nothing published it. `worktable_codegen` depends on it by path with a
version requirement, so the first version-bumped merge after the
extraction would have failed the release after green CI:
error: failed to prepare local package for uploading
Caused by:
no matching package named `worktable_dsl` found
location searched: crates.io index
required by package `worktable_codegen`
Reproduced with `cargo publish --dry-run -p worktable_codegen`. The
publish step now walks all three crates in dependency order, each with
its own already-published guard, so adding a fourth is one more line
rather than a rediscovery of this failure.
The three versions also disagreed: dsl at beta.14, codegen at beta.15,
worktable at beta.17. Three crates published from one repo on three
numbers is how a macro and the runtime it generates calls into drift
apart. They now move together at 1.0.0-beta.17.
The two intra-workspace pins go back to `=`. The caret is right for
every external dependency and wrong for these two: a mismatched
macro/runtime pair fails at expansion inside a consumer's build, which
is a worse place to find it than a resolver conflict here. That is the
one concession the caret change conceded in its own description.
… not one
Three of the six block parsers consumed the comma that may follow their
block and three did not. `parse_updates`, `parse_indexes` and
`parse_queries` did; `parse_deletes`, `parse_in_place` and
`parse_configs` did not, so `config: { .. },` reached the top-level
dispatch as a `,` token and died as "Unexpected identifier". Those three
blocks happen to be written last in every declaration in this
repository, which is the only reason nobody had hit it.
It matters now because the designer emits a declaration and parses it
back. An asymmetry between which blocks may carry a comma is a round
trip that fails on the tool's own output as soon as block order changes,
and block order is the sort of thing a visual editor changes freely.
Three `try_parse_comma()` calls, strictly more permissive: every form
that parsed before still parses.
The error message was the other half of the cost. A `,` reported as an
unexpected *identifier* names the wrong category of token and sends the
reader looking for a misspelled keyword. Four dispatch arms now print
the token they saw and the set they expected.
`dsl/tests/trailing_commas.rs` was checked against the pre-fix parser:
three of its five cases go red when the `try_parse_comma()` calls are
reverted. The two that stay green are the ones asserting the fix is
permissive rather than a new rule.
This was referenced Sep 2, 2026
Closed
added 5 commits
September 2, 2026 14:48
`worktable_dsl` could read a declaration and could not say whether the macro would accept it. The rules lived in `worktable_codegen`, next to the code they would have generated, and a proc-macro crate exports nothing but macros, so the only way to ask was to expand. A designer cannot expand a proc macro, and an editor that finds out by compiling is not an editor. The three validators move here unchanged, still operating on `crate::model` types, which carry spans. `worktable_codegen` calls them here and its diagnostics are identical, down to the span each error points at. Same discipline as the parser extraction, same reason: a second implementation drifts, and the drift shows up as a designer that green-lights a table the compiler rejects. `check()` is the designer's entry point. It returns the schema *and* the diagnostics, because a declaration that breaks a rule is still a declaration and an editor has to draw it in order to let anyone fix it. Each diagnostic carries a stage: `Grammar` means there is nothing to draw, `Rules` means there is a schema and the macro would refuse it. That second state is the normal one in an editor and had no representation at all before this, since `Schema::parse` returns `Ok` for it. It reports every broken rule rather than the first. The macro stops at the first because it will not generate code either way; an editor has the opposite economics, where fix-recompile-find-the-next is the loop a live checker exists to remove. `index_backends_into` therefore collects instead of short-circuiting, and `validate_index_backends` is a thin first-error wrapper over it so the macro's behaviour is unchanged. Spans come back as byte ranges on the diagnostic rather than as fields on the IR. `Schema` stays plain data, serialisable and comparable across processes, which was the whole point of it; the ranges live in the result of the call that produced them, so a consumer that does not want them does not carry them. Byte offsets rather than line and column: an editor converts to whatever it needs, and a range survives being sent to one that disagrees about what a column is. They need `proc-macro2/span-locations`, and this crate is compiled for the host as part of `worktable_codegen` before anything else in a dependent's build. So it is an off-by-default `spans` feature, the same argument that keeps `serde` off. Without it the span is `None` and the messages are identical: a consumer degrades to file-level diagnostics rather than losing them. The span test asserts by slicing the source with the range rather than by comparing offsets. An off-by-one in either direction produces a plausible number and a wrong underline, and only the slice catches that.
The schema language has no foreign keys. A column called `project_id` is a `u64` like any other, and nothing in WorkTable enforces, records, or checks a relationship between two tables. `infer_relations` guesses those links from a naming convention, and the guess is right often enough to be useful and wrong often enough that presenting it as recovered fact would make a designer lie about the schema. So it is not the same kind of thing as the rest of this crate. Everything else here has a single correct answer that WorkTable owns: what the grammar accepts, what the macro rejects, what a schema change costs the storage engine. The mapping between tables belongs next to whatever application enforces the convention, which is not this one. The Mermaid emitter and the relation guessing move behind an off-by-default `uml` feature, and their tests move to a file gated on it. Nothing is deleted: it is written, it is tested, and turning it on is now a decision rather than a default. The crate docs say which layer is which, so the next person building on this does not have to work it out from the absence of a foreign key. That also makes the feature set consistent. `serde`, `spans` and `uml` are all off by default for the same reason: `worktable_codegen` depends on this crate and is a proc macro, compiled for the host before anything else in a dependent's build, so anything unconditional here is paid by every WorkTable user's first compile to serve consumers who are not the compiler.
`delete_many(keys)` and `delete_range(a..b)`, the bulk eviction a consumer needs to bound state growth. Tracked as #78, and the blocker for generational eviction in agentcode, whose durable state grows about 16.5 MB per generation with no way to drop one. Deleting was already the cheap operation the issue asked for: the row is marked deleted in place, its index entries come out, and the storage becomes reusable once no reader can reach it, with vacuum compacting pages later. Nothing about that model needed changing. What was missing was a way to do it to many rows at once. **What batching actually saves.** Not the per-row work: every index still has to lose its entry, and that is most of the cost. What it saves is the fixed cost paid per call, which is one lock acquisition over the whole key set, one grace marker, and one reclaim pass. `Data::delete_many` ghosts every link first and retires them behind a single marker, because a link must not become reusable while a later row in the same batch is still being marked, or a concurrent insert could claim it and be ghosted by this call. Measured on a 50,000-row table, scattered keys, three backends: batching is a flat 1.3 to 1.4x over a loop of `delete` at every batch size from 1 to 100. It does not improve with batch size, and that is the honest shape: the constant fraction is what batches away. **`delete_range` is ergonomic, not faster.** It converges to `delete_many` rather than beating it, and is slightly worse below a batch of 64. Two rounds of optimisation went in before that was clear, and both were worth keeping: it now takes the mutation guards *before* the walk that reads the links, so one `O(log n + k)` walk replaces `k` lookups instead of being added to them, and both paths now read rows at the link they already hold rather than spending a second primary-key lookup in `select`. The remaining per-row work is identical in both, so there is nothing left for a span to exploit. Its value is that the caller does not enumerate the keys. **Not all-or-nothing, unlike `insert_many`.** A rejected insert has published nothing, so unwinding restores a state that was real. A delete that fails partway has already ghosted rows and removed their index entries, and resurrecting them would mean republishing index entries for storage queued for reuse. So the error reports how many succeeded. A key that is not present is skipped rather than failing the batch: a caller evicting a generation cannot know which keys a concurrent writer already removed, and making them find out first is a race they cannot win. The tests were checked against broken code before being trusted, and one had to be rewritten to earn it. `deleted_rows_are_unreachable_through_every_index` originally only read the deleted rows back, which passes with secondary index removal deleted entirely: a ghosted row is filtered out of reads, so a dangling index entry is invisible through `select` until the link is reused. It now reclaims each deleted row's unique value, which a stale unique entry rejects. Skipping secondary removal fails it and the reuse test; skipping primary removal fails five.
`delete` is async and `delete_many`/`delete_range` were not, which is an inconsistency I introduced yesterday rather than one I found. The split has a cause: cell-level locking means an update waits on the readers of the cells it touches, so `update`, `upsert`, `delete` and `reinsert` are async, while `insert` has no existing cell to contend on and the batch paths take the striped mutation gate rather than cell locks. That is a real distinction and a bad thing to expose. A caller cannot be expected to know which writes happen to need a cell lock, and the penalty for guessing wrong is not a compile error: `let _ = table.upsert(row)` builds a future, drops it, and the write never happens. These two await nothing today and say so. They are async because the write surface should have one rule, and because they will need to wait once they take cell locks rather than the striped gate. `insert` and `insert_many` are the remaining exceptions and are not touched here. Making them async does not stop at the API: the macro composes `insert` internally, and `PersistenceTask::push` is synchronous and inserts into WorkTable's own `QueueInner` table, so it reaches into the persistence queue. That is its own change with its own review.
Concurrency coverage existed but was scattered and stopped at four
writers: `nonunique_arctic` races four against an arctic non-unique
index, `index_backends` recovers concurrent same-row updates, `vacuum`
runs a vacuum thread beside sequential inserts. Nothing raced writers
across all three backends, and nothing went above four.
Four is exactly the last writer count at which this engine still looks
healthy. Insert throughput is flat to four and collapses at eight, so
the one dial that would have exposed it was never turned.
Four tests per backend: eight writers losing no rows, inserts racing
deletes so an insert can claim storage a delete just freed, a contended
unique key admitting exactly one writer, and readers seeing consistent
groups while writers run. Congee appears throughout because these need
only a unique index; it has no non-unique backend.
Every shape is a parameter rather than a literal, which is the point:
hardcoding the writer count is how this hid.
WT_CONC_WRITERS=32 WT_CONC_PER_WRITER=2000 cargo test --test mod concurrency
WT_SCALE_SWEEP=1,2,4,8,16,32,64 cargo test --release --test mod insert_throughput -- --ignored
A malformed value is a hard error rather than a silent fallback: a typo
in `WT_CONC_WRITERS` that quietly runs the default is a run you believe
tested something it did not.
`insert_throughput_should_scale_past_four_writers` is `#[ignore]`d
because it fails, deliberately, recording the defect the way
`generator_determinism` records its own. Eight writers reach 0.20x of
single-writer throughput. It is not the index: all three backends
collapse to the same ~300 K/s, and arctic and congee are 1.3x faster
single-threaded before hitting the identical wall. A `sample` of the
eight-writer run puts the time in `RawRwLock::lock_exclusive_slow` and
`DataPages`, and `pages.rs` takes an exclusive write lock on the one
page named by `current_page_id`, so appends serialise by construction.
`EmptyLinkRegistry::pop_max` takes a global mutex on every insert even
when the free list is empty.
It refuses to run in a debug build. Per-operation overhead there swamps
the contention and the sweep reports eight writers as 2.79x *faster*
than one; a throughput assertion that passes for the wrong reason is
worse than none.
added 3 commits
September 2, 2026 20:05
`DataPages::insert` calls `EmptyLinkRegistry::pop_max` on every insert, and `pop_max` took a global `FairMutex` before looking at anything. An append-only table therefore paid a contended mutex per row to discover there was nothing to reuse. `sum_links_len` already tracks the aggregate and is maintained on both sides, so one relaxed load answers the question without touching either lock. Release build, 200,000 inserts, best of three, M4 Max: | writers | before | after | | | ---: | ---: | ---: | ---: | | 1 | 1,216,309/s | 1,207,160/s | flat | | 2 | 1,216,010/s | 1,326,333/s | 1.09x | | 4 | 1,154,861/s | 1,425,324/s | 1.23x | | 8 | 310,748/s | 802,077/s | **2.58x** | | 16 | 272,121/s | 557,414/s | 2.05x | Scaling at eight writers goes from 0.26x of single-writer throughput to 0.66x, and four writers now scales positively (1.18x) where it used to degrade. So this mutex, not the page lock, was the dominant term. The page lock is presumably the remaining ceiling: `insert` still takes an exclusive write lock on the one page named by `current_page_id`, which serialises appends by construction, and 0.66x is not 1.0x. `Relaxed` is enough because the answer is a hint rather than an invariant. A push landing concurrently can leave the load reading zero, and the caller then appends a fresh row instead of reusing a link that became available a moment ago: the same outcome as having called one instruction earlier, with the link still registered for the next insert. The reverse cannot happen, since the counter is only non-zero once a link is registered, and the locked path re-checks regardless. The failure this could cause is silent in both directions, which is why it gets its own test rather than relying on the reuse tests. A counter that drifts above zero costs a pointless lock; one that drifts to zero while links remain stops reuse entirely, and inserts would simply append forever, which is a correct way to insert. `the_pop_fast_path_agrees_with_the_registry` pins the counter to the registry across a push/pop cycle, and was checked against both drifts: never decrementing fails it, and never incrementing fails it.
`assert!(!cfg!(debug_assertions))` is a constant assertion and clippy rejects it; a bare `#[cfg] panic!` makes the rest of the function unreachable and its imports unused. Reading the profile through a function keeps one code path in both builds.
They reach the same striped mutation gate by different routes: `insert` gates one key, `insert_many` gates its whole key set and sorts the stripes so a batch and a single insert cannot deadlock. That ordering is a claim worth holding down, and a deadlock shows up as a hang rather than a failure, so the arms interleave on overlapping stripes instead of staying politely apart. The persisted path already had `batches_and_singles_interleave_through_the_engine`; this is the in-memory half, across all three backends.
…e benches Two changes, both about putting things where they belong. **Key widths.** `validate_index_backends` rejects a declaration whose key type the backend cannot serve, and the accepted sets are specific: congee takes u8/u16/u32/u64/usize, arctic takes u16/u32/u64/u128. That list is a promise the macro makes, and nothing tested it. The suite instantiated u64 with both backends and u128 once, so most of the advertised matrix had never been generated. The backends test their own key widths in their own repositories, which is the right place for the data structure and the wrong place for this. What was unverified is not whether arctic handles a u16, it is whether a *generated WorkTable over an arctic u16 index* stores, finds and removes a row. Eleven cells, deliberately shallow: insert, select by that index, a key that must not resolve, then delete. Depth belongs in the backend's suite. All eleven pass, so this adds coverage rather than fixing a bug. **The throughput sweep moves to wt-benchmarks.** It reported a curve rather than asserting a property, which makes it a benchmark, and it had to be `#[ignore]`d to live in a correctness suite at all. It is now `cargo bench --bench insert_scaling` alongside a new `op_latency` bench covering single insert, upsert, delete and select in both storage modes. This file keeps the correctness tests, which is what it is for.
Reclamation frees links in retirement order, and that order is usually contiguous: a range delete, or any workload deleting in primary key order, frees a run of adjacent links. Pushed one at a time each link pays its own coalesce, and a coalesce is up to two removals and one insertion across three ordered containers behind a global mutex, so freeing n adjacent links costs n times that. `push_many` merges the batch against itself first, which turns a run into a single insertion, and takes the lock once rather than n times. The merge is a separate pure function because the saving it represents is invisible in the registry's final state: coalescing per link and coalescing per run leave identical contents and differ only in the work done to get there. Deleting the self-merge leaves every registry-level test green, so the test looks at the merge directly.
`reclaim_retired` needs to know whether a whole-page retirement is queued behind the links it is about to free, and it answered that by collecting every queued page into a set, reading the entire queue, on every call. With a backlog of one that is a wasted allocation. With a real backlog it is quadratic: draining 20k retirements one sweep at a time spent almost all of its 780 ns per row in that rescan. Pages are retired rarely, so counting them as they are queued makes the common answer one relaxed load, and the scan happens only when a page retirement is actually outstanding. Freed links are now collected and restored together so the registry can merge a contiguous run, and the buffer is flushed before a page is purged: batching changes when links are restored, and the one ordering that must not drift is restoring them after a page they belong to was reset. The existing whole-page test processes both retirements in one sweep and stays green with the queued-page check disabled entirely. The new test splits them across sweeps, which is what a capped sweep does anyway, and fails without the check.
Calls the reusable workflow in api.crates.vip, which packages the crate, derives its sparse-index entry from cargo metadata and writes both to Tigris under conditional requests. Manual dispatch, because a published version is immutable and cargo package embeds the git sha, so the same version from a different commit is refused rather than replaced.
api.crates.vip is now crates.vip-backend, matching the fleet convention where a backend is <domain>-backend. GitHub redirects the old path, but a workflow reference should name the repo that exists.
added 28 commits
September 3, 2026 11:59
Four reclamation defects were fixed this week and written out they are one defect: an index entry resolving to storage that now holds a different row. Each was found by a test that noticed the damage somewhere downstream, and each needed its own investigation to get back to the cause. This asserts the property rather than the symptom. After a storm of concurrent inserts, deletes and upserts with vacuum running every 5ms: every reverse entry resolves to a row carrying that key, forward and reverse agree on the same storage, no two keys name the same storage, and every live row's unique secondary entry points at that row's storage. Run on all three primary backends, with the unique secondary following the primary, so each arm exercises one implementation end to end rather than resolving its secondaries through the same map in every case. The non-unique index cannot follow, because congee holds no non-unique index at all. The three-backend difference showed up immediately and twice: arctic and congee expose no iterator, so a check written against WTI's `iter` does not compile against them. Both checks are now written by probing keys, which is the stronger property anyway: it asserts an entry points at *this* row's storage rather than at some readable row. **The teeth are unproven and this commit does not claim otherwise.** Reverting the page-reclaim fix does not make it fail: four runs passed. The defect it is written for only ever reproduced under full-suite machine contention, never in isolation, so logical concurrency alone does not reach it. The value here is the property being stated and checked at all, and a place for the next one in this class to land. A deterministic test, inserting onto a page between its snapshot and its reclamation, is still owed.
The registry coalesces adjacent freed links into runs and keeps them ordered by absolute position in index_ord_links, and by size in length_ord_links. Planning uses neither: get_per_page_info takes the op_lock, iterates every entry in page_links_map, and rebuilds a per-page map from scratch each pass. Working from runs changes what vacuum does rather than how it decides. To reclaim a page today you move every live row on it, so the work is proportional to how full the page is and a mostly-full page is skipped as poor value even when one row is stranding a large gap. From runs, you move the rows standing between two adjacent runs and one move merges them, so the work is proportional to the stranding rows and the value is the run it creates. The trigger can be reactive for the same reason. sum_links_len and item_count are already maintained on every registration and removal, and after coalescing the ratio between them is the fragmentation measure: the same free bytes in one run is a healthy table and in five hundred runs is a fragmented one. No scan is needed to know which, and length_ord_links already says whether a run big enough for current allocations exists. And the pass should stop being one lock. defragment holds lock_vacuum's write side for its whole duration while pop_max takes the read side, so for the length of a pass no insert can reuse a freed link. A pass whose purpose is to make space reusable currently prevents space reuse while it runs. Recorded honestly: this reduces how much relocation happens and how long anything is held. It does not remove the recurring bug class, because a relocation still rewrites index entries holding physical addresses. Fewer relocations is fewer chances to get it wrong, not a guarantee.
Both could already iterate through `UniqueIndex::iter_values`. What they lacked was the inherent `iter` the general-purpose backend has, which is a difference in spelling rather than in capability, and it is the kind that quietly decides what gets tested: code written against one backend does not compile against the others, so it stays written against the one whose name the author already knew. That is not hypothetical. The vacuum invariant suite hit it twice while being written, once on the primary index and once on the secondary, and the way round it was to probe by key instead of iterating. Portable, and a weaker check, chosen because of a method name. Thin aliases, no new capability, no behaviour change: both delegate to `iter_values`, which already materialises because the underlying maps hand out entries borrowed from a guard. The invariant suite now iterates all three directly, which is what it wanted to do in the first place. A check that can only read one backend guards only one backend, and this table has three.
Measured with wt-benchmarks' new vacuum-stress suite: a sweep at 60% fragmentation costs 25-49% of insert throughput and doubles median insert latency, for its entire duration. The same sweep at 25% costs between nothing and 10%. The numbers and the method are in docs/vacuum-design-directions.md. The mechanism is the exclusion rather than the work. pop_max takes the registry read side with try_read_owned().ok()?, so while a sweep holds the write side every insert wanting reclaimable space is turned away and allocates a fresh page instead. Inserts never blocked on vacuum; they lost free-space reuse for as long as it ran. Three changes follow from that: The sweep is cut into batches that release the exclusion between them, so reuse comes back in the gaps rather than after the whole table. Between batches vacuum measures how much foreground demand it turned away and stands down when the answer is a lot; the sample is taken across work that was happening anyway, so asking costs nothing. A bound on consecutive stand-downs keeps a permanently busy table from never being vacuumed, which would trade a bounded slowdown for an unbounded one. VacuumGate is the explicit form of the same thing, for a caller who knows something the table cannot see. The sweep is woken by tables freeing space rather than by a timer. A timer cannot know when a table became fragmented; the registry can, because it is the thing that grew. The interval stays as a fallback for a table whose threshold is never reached. Wake granularity is bounded by the reclamation backlog, since that is when freed links actually reach the registry. Pages a ranged delete emptied are swept first. Every deferred reclamation arrives batched, so the batch alone does not say a delete was ranged -- coalescing does, because a run only forms from links that were adjacent. Those pages hold the concentrated free space, so a sweep that stands down partway through has already done the valuable work. Each behaviour has a test carrying both arms, because only the pair is evidence: an unbatched sweep reuses nothing, an unreachable threshold leaves the sweep parked, a scattered batch names no page. The persisted test covers an interleaving that could not happen before, since an insert can now claim a link on a page a sweep is still working through.
An ART orders by the bytes of the key, and two's complement puts that ordering the wrong way round: a negative has its high bit set, so it sorts after every positive. XOR with the sign bit corrects exactly that. It is a bijection over the full width and strictly monotonic, which is the only property the tree needs, so i64::MIN lands at 0 and i64::MAX at u64::MAX. Being a bijection over the *whole* width is also what makes the exclusive bounds correct: next/previous run in the raw space, and adjacency survives because no raw value is unreachable. Three places had to agree. ArcticKey now maps signed types. The generated primary-key newtype delegates to its primitive's impl instead of asserting the mapping is the identity, which is what the trait's own doc comment said it was for. ArtPersistenceKey gained signed impls, without which a persisted signed table could not be declared at all. The macro and the editor-facing check had two hand-maintained copies of the accepted key types. Widening one without the other gives a check that accepts what the macro refuses, so codegen now reads the DSL's list. Both failure modes are covered: no flip, and a flip applied on one side only. The second is the one that matters, because point lookups keep working and only ordering breaks. Verified by breaking each in turn. i8 is absent because Arctic's narrowest raw key is u16. The sign flip in ArtPersistenceKey is not load-bearing today and says so: every use is a whole-record round trip and nothing sorts encoded bytes.
It fails about half the time on a real defect: two secondary-index event ids are consumed without their events reaching the stream, and the batch validator then defers on the gap forever, stalling persistence for the rest of the table's life. Not caused by batching the sweep. With batch_pages set to 0, restoring the whole-table exclusion, it reproduces at the same rate. Concurrent inserts during a sweep on a persisted table had simply never been covered, so nothing had run this interleaving before. Ignored rather than weakened, so the suite is green and the reproduction survives. The doc comment carries the error and the un-ignore condition.
Concurrent inserts during a vacuum sweep stalled persistence for a table
permanently, about half the time:
persistence stalled on primary index event gap:
last applied Id(2938), next available Id(2940) (attempt 9);
an event id was likely consumed without its event being queued
Nothing was consumed. Instrumenting the batcher showed the missing event
reaching batch preparation 33 times in one failing run. It was queued the
whole time; it was never applied.
Event ids are allocated during the index mutations, while an operation id
is a Uuid::now_v7 minted later at the push site, so two concurrent writers
can invert the two orders. Vacuum does it systematically because its update
lands on the sweep's destination page while inserts append to the current
page: 110 inversions in one run, in a regular alternating pattern. Batch
collection walks operation ids and grows the batch by data page, so the two
streams sit on pages it separates, and it advances past operations between
the ones it took. The op holding the needed id was never collected, every
retry rebuilt the same gapped batch, and the attempt budget then failed the
engine for good.
Page grouping is a write-batching optimisation, so after a few failed
attempts collection now takes the whole queue instead. That keeps the
guarantee that matters, because validate_events still refuses a stream with
a hole: a complete collection can only apply more of the contiguous prefix
than a partial one, never something unsafe.
The deterministic test does not need threads. Collecting one page also
takes a later operation and advances past it, so an operation between them
on another page is skipped and the walk then runs out — which the
page-limit growth cannot rescue, because the loop ended empty rather than
full. Without the fallback it reproduces the field error exactly.
The integration test that found this is un-ignored: 12 of 12 runs pass,
against 5 of 12 failing before.
The job hands the reusable workflow the registry's Tigris credentials, so whoever can move `master` on crates.vip-backend can read them. A sha is the only ref that cannot be moved under us. All three `uses:` lines move together on purpose: they are one workflow publishing one workspace, and a split pin means two versions of it.
The schema module said the validation rules live in `worktable_codegen`. They live in `worktable_dsl::validate`, and codegen reads its lists from there, which is the whole point: the macro and an editor calling `check` cannot disagree about what is legal. `emit_dsl` said `parse_configs` does not consume a trailing comma. It ends with `try_parse_comma`, so one would be accepted. The emitter still omits it, but for the real reason: `config` is last and nothing follows it.
One counter was answering two questions and getting both wrong. A batch can be valid and still apply nothing: everything in it sat behind a gap and was trimmed, and validation returns an empty batch rather than a deferral. That is a success, so it reset `attempts` to zero. The whole-queue fallback keyed off `attempts`, so in exactly the case it existed for -- a stream stuck behind one missing operation -- a run of empty successes kept the counter down and it never engaged. That is why the previous fix took the failure rate from 5 in 12 to 1 in 14 rather than to zero. Widening now keys off a no-progress counter, which follows the applied watermark. Only real progress resets it. Giving up is now its own threshold and a much more patient one. A gap is usually transient: the operation carrying the missing id has been pushed but not batched, or its producer has not reached its push yet. Each deferral sleeps 500ms, so eight cycles was about four seconds, which a producer descheduled under load can lose. The engine then declared a permanent bug over a slow thread. It now waits about a minute. Tying both to one counter measured worse when tried, 3 failures in 13 runs, because incrementing on empty successes engaged the fallback but also burned the give-up budget. Splitting them is what the two jobs actually wanted.
Mutation guards are striped 64 ways, so any batch wider than that takes every stripe. Holding them for the batch made a large delete a stop-the-world for every other write on the table, and a range delete dropping a generation is exactly that shape. Guards are now taken a chunk at a time. Each chunk is self-contained: its rows leave every index, are ghosted, and are queued for reclaim before its guards drop, so no row is left out of the indexes and un-ghosted across a boundary. Stripe ordering still holds, because a chunk releases before the next is taken, so a batch delete and a batch insert still cannot deadlock. `delete_range` narrows its second walk to the chunk's span rather than repeating the whole range per chunk, keeping the walk proportional to the chunk. The guarded-key filter now checks the chunk, since that is what the guards cover. `insert_many` is deliberately untouched: it is documented all-or-nothing and unwinds a rejected batch, so its guards have to span it. Measured: a concurrent writer landed 190 rows during a 20,000 row delete, against 71 with whole-batch guards. That ratio is timing, so the test asserts the thing chunking could actually break instead: a delete spanning several chunks removes exactly its keys and leaves interleaved survivors with their index entries intact.
`check` is the oracle an editor calls on every keystroke and a generator validates against before emitting, so its contract is that bad input comes back as a diagnostic and never as a panic. A panicking proc macro reports a compiler-ICE-shaped error with no span, which is the worst way to tell someone they typed a comma. Two tests. The sweep applies the edits that actually happen while typing -- truncate, drop, duplicate, transpose, stray delimiter -- compounding them so it reaches inputs several mistakes deep, over 10,000 inputs from a fixed seed, so a failure is reproducible from the printed string. No dependency and no fuzz harness: a four-line xorshift is enough and it stays deterministic in CI. The sweep will not find the interesting cases, though, because they *parse* and are then semantically wrong, which is where the known panics lived: an `.expect` reached after the grammar was already satisfied. Those thirteen are written by hand, and they assert diagnostics rather than merely the absence of a panic. Silently accepting one is the same defect wearing a quieter face, since a generator trusting `check` would emit a declaration that then fails to compile. That is the gap class from worktable-check-gaps-2026-09-03.md, now guarded. All thirteen already report. The panics found by hand earlier were closed by this release's validation work; this keeps them closed.
The first vacuum-stress run showed multi-millisecond worst-case inserts in every vacuum-off arm at 60% fragmentation, 32ms on wti and 41ms on arctic, and this document argued from it that a sweep converts rare long stalls into steady overhead. Re-run on an idle machine, every vacuum-off arm is 0.13 to 0.24ms. Those maxima were contention. The only multi-millisecond tail in the clean run is on the vacuum-on side. Two other readings change with it. The penalty at 60% is about 50% on all three backends; the earlier -24.7% for arctic came from a vacuum-off baseline depressed by the same contention, 1.23M against 1.82M idle. And batching the exclusion did not reduce the penalty at all, at either fragmentation level, which is what the clean pairing was run to settle. The cost curve still stands and is still the argument for reacting to fragmentation early: 2 to 12% at 25%, about half at 60%. The tail benefit was not real.
Two versions of WorkTablesIndex in one graph split the type identity of Pair and ChangeEvent. The build then fails with "expected ChangeEvent<Pair<T, Link>>, found a different ChangeEvent<Pair<T, Link>>" across dozens of unrelated-looking trait errors, none of which names the actual problem. This is not hypothetical. Pointing the local patch at WorkTablesIndex 0.0.10 today produced exactly that, because published data_bucket 0.5.5 still pulls 0.0.9. It costs an afternoon the first time and is invisible until something is bumped. The review asked for this guard rather than reverting the carets, and the carets are right: the requirement is not too loose, the three crates simply move as one train and nothing said so. Now one named failure says it. The script was run as bash before committing, both ways: it passes on a correct graph and exits 1 with the duplicate present.
Every earlier number was measured against published WorkTablesIndex 0.0.9, data_bucket 0.5.5 and ps-reclaim 0.1.1. The local patch was not applying: a patch does not force re-resolution when the lockfile already pins the published versions, and wt-benchmarks is a separate workspace that never inherited it either. On the release stack the low-fragmentation cell changes completely. At 25% the penalty is within noise on all three backends and straddles zero, two arms reporting vacuum-on faster than off, which is the signature of an effect below the measurement floor. The same cell measured 2 to 12% on the published stack. The sweep is effectively free there. At 60% nothing moves: about half on every backend, unchanged. That remains the case batching did not help, and it is the one that needs a work-side bound rather than more yielding.
A fixed 2ms backoff is the wrong shape for sustained load. Sixteen of them
is a 32ms pause and then a batch regardless, so a table under a writer
doing hundreds of inserts per millisecond still got a steady grind of row
moves and paid for them. Doubling turns sustained pressure into a low duty
cycle: a burst costs a few milliseconds, continuous load costs a sweep
every fraction of a second, and the bound on consecutive stand-downs still
guarantees the sweep proceeds.
Measured on the real dependency stack, 60% fragmentation, all three
backends:
wti -47.4% -> -18.7% insert p50 2209ns -> 1125ns
arctic -50.9% -> -19.4% insert p50 1959ns -> 1000ns
congee -50.4% -> -19.8% insert p50 2000ns -> 959ns
Latency returns to the vacuum-off baseline, which is 1083/917/917ns. The
throughput penalty is what remains, and it is the work itself.
I had written that this case needed a work-side bound and that pacing could
not help it. That was wrong: the pacing was there and simply too shallow to
matter under load. The measurement was taken at machine load 18.5 against
8.1 for the run it is compared with, so conditions were worse, not better,
and the improvement is consistent to within 3 points across three backends.
The wake fires on the first crossing of the threshold, which during a ranged delete is near its start. Sweeping there means competing with the workload producing the garbage and compacting pages that are still being emptied behind it: a moving target, and the worst possible moment to take the exclusion. Reactive without a settle is just eager. The signal is queued retirements, not the registry's byte total. That is the part worth remembering: deletes queue their storage and only reach the registry when the backlog flushes, so between flushes the bytes sit still while deletes stream. A settle watching them calls the burst over in every gap, which is exactly what the first version of this did and what the test caught. Capped, because under continuous delete load retirements never stop arriving and an unbounded wait would defer the sweep forever. The test drives deletes in the chunks a ranged delete arrives in and asserts the sweep is not told to run until they stop. Without the settle it fails.
This reverts commit fbedd4b. The measurement it claimed was noise. That commit reported the 60% penalty moving from about -49% to about -19% across three backends and read the consistency as evidence. It was taken at machine load 18.5, and the effect was the machine starving the vacuum task, not the backoff policy working. Re-run at load 5, the same code shows -44 to -47%: unchanged. A proper A/B at one load, fixed against exponential, is noise in both directions: -12.8pp for congee at 60%, and +27.8pp for wti at 25%, where the latter means vacuum running measured 28% *faster* than vacuum stopped. That is impossible, so the run-to-run variance exceeds anything this benchmark can resolve about pacing policy. The reasoning behind the change may still be right: a fixed 2ms stand-down capped at sixteen tries is an arbitrary 32ms ceiling, and sustained load arguably wants a lower duty cycle than that. But it is a hot-path change to reclamation with no evidence, and one arm suggested it was worse. It goes back until there is an instrument that can tell. The instrument is the actual gap. Each arm is a separate two-second run against a separately built table, so the pairing does not remove the variance it was meant to. Resolving a few percent needs repetitions and a null arm, which is the next thing to build.
It was never user-facing in any useful sense, and exposing it invited the one thing that defeats the design. Turn it down and the fallback timer wins the wake's select every time, so the fragmentation threshold never triggers anything and the settle never runs: the reactive sweep degrades silently back into the polling loop it replaced. That is not hypothetical. Every vacuum test and the vacuum-stress benchmark set it to 5ms, which means none of them were measuring the reactive path at all. The benchmark's own comment said the quiet part out loud: "Hard, on purpose. A 60 second default would measure an idle table and report that vacuum is free." The fallback is now an internal constant. Tests that wanted an eager sweep set wake_threshold_bytes instead, which is the reactive equivalent and exercises the path that ships.
The message blamed "an event id was likely consumed without its event being queued" and I spent a long time treating that as a guess, because the first instance was an ordering problem where nothing was lost. With the whole-queue fallback in place it is no longer a guess. A trace of a live stall shows sixty consecutive deferrals over the same nineteen operations, with the fallback engaged, so every queued operation was collected and the stream is still gapped. The operation carrying the missing id genuinely never reached the queue, and the producer is upstream of the analyzer. The message now carries the queued operation count and says that, so the next occurrence points at the right half of the system instead of at this one. The test's timeout also moves past the give-up budget, so a stall surfaces as this diagnostic rather than as a bare Elapsed that says nothing.
This reverts commit 9a92003.
… needed"" This reverts commit 82c686c.
Cumulative sweeps, pages freed and bytes freed, exposed on the manager. Cost is unreadable without this, and I proved it the hard way. A whole day of vacuum cost numbers turned out to be measuring nothing: the sweep ran zero times in every arm of the stress benchmark, so every penalty reported was noise, including a 30-point "improvement" I attributed to a backoff policy and committed twice. A snapshot of the table cannot tell you this. It shows no reclaimable bytes left whether the sweep ran and reclaimed them or never ran at all. Only a cumulative count separates a reactive sweep that keeps firing from one that fired once and never again, which is the question that matters for a trigger driven by a threshold the workload itself drains.
A table's memory is its pages, so `allocated_pages` times the page length is what it holds from the allocator, and `reusable_pages` is the part a sweep has already given back. Without these a vacuum cost figure cannot be checked. Reclaiming memory is the entire point of a sweep, and a sweep that never runs looks exactly like a sweep that is free: both report no overhead. Measuring cost alone is how a day of vacuum numbers turned out to be describing a sweep that ran zero times.
A partial return is not a return. The wake fires when a table *frees* space, so a table that has gone quiet produces no more of them, and doing one pass and going back to waiting left whatever that pass could not finish sitting there forever. Measured with the new yield benchmark: 392 pages in use where 196 would do, half the memory never given back, with the sweep reporting success. Now it keeps sweeping while there is something worth reclaiming: 197 pages against an ideal of 196, the one extra being the current append page. The vacuum-off arm still holds 392. Three exits so it cannot spin or run away. Fragmentation falling below the threshold, a pass that frees no pages, and a bound on consecutive passes. That bound matters on persisted tables specifically: every reclaimed page queues a durable-free barrier, so sweeping back to back queues work faster than the persistence worker drains it and the on-disk state falls behind the table in memory. The 5ms pause between passes hands the worker its turn, and the cap ends the run rather than letting one table monopolise the queue. The foreground is unaffected while this happens: upsert p50 with vacuum on lands within one timer tick of the arm with it off, in both directions, and delete p50 is identical to the nanosecond.
Vacuum deferred on the rate of requests for reclaimable space. Deletes never make one, and an upsert that fits in place does not either, so under a load of exactly those the signal read idle and the sweep walked straight in. Measured: 64 sweeps taken during a two-second upsert-and-delete burst, costing 3.8 to 12.5% of foreground throughput against a null arm. The signal is now mutation stripes in flight, read live from the lock map. Every insert, delete and upsert passes through one of those gates, so it cannot miss a workload the way a space-demand counter can. Each stripe is a ticket lock, so a ticket handed out and not yet served is a writer inside the gate or queued for it. It waits rather than budgeting. A sweep that defers for a fixed span and then forces itself in is not waiting its turn, it is queueing, and it takes its cut from every burst that outlasts the span. There is no span here to run out: it asks, and if the answer is yes it asks again, with the interval doubling so a long burst is polled cheaply. Three consecutive quiet samples are required before it goes in. One is not enough, because under heavy writes the stripes are free for most of any instant and a single look finds a gap immediately. Result: sweeps during the load went from 64 to 0, and all 64 now happen once the table is quiet. Memory returned is unchanged at 99.5%, 197 pages in use against an ideal of 196. The trade is real and deliberate. A gate that is paused, or a table that never goes quiet, now holds the sweep indefinitely rather than letting it through after a fixed number of tries. The gate test asserts that directly: paused means held, and resumed means finished.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The beta.17 release PR. It absorbs the earlier beta.17 work and deliberately excludes #58.
What ships
Validation
The exact local candidate passed the complete default, versioned-row-publication, and all-features build/test matrix plus clippy with warnings denied.
Vacuum was validated under sustained upsert/delete/reinsert pressure across WorkTablesIndex, Arctic, and Congee. Reactive vacuum stayed out of the measured foreground path while the unpaced positive control produced detectable interference. Every backend returned to the independently packed ideal page count after load.
The beta.13/beta.15/beta.17 performance grids include CRUD, partition routing, AgentCode, MoE-PGO, concurrency, and vacuum. The stable nanosecond-scale partition_ref regression is documented and accepted because routed application work improved. Full evidence is in docs/beta17-validation.md.
A local-source support.cafe consumer passed its complete workspace gate, loaded and rebuilt the live S3 dataset into a rollback-safe prefix, strict-reloaded all six persisted tables, persisted a mutation across restart, and passed HTTPS/WebSocket startup on Fly.
Release train
Published prerequisites:
Merging this PR publishes in dependency order:
Consumer ports follow only after all three artifacts are visible on crates.io.