Skip to content

Document the beta17 release closeout and vacuum precedent #92

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pathscale merged 1 commit into from
Sep 3, 2026
Merged

Document the beta17 release closeout and vacuum precedent #92

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 12 additions & 6 deletions docs/TODO.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
What is known to be unfinished, and enough context to act on it without the
conversation it came from. Ordered by whether it blocks a release.

Last reviewed 2026-09-04, against `release/beta17`.
Last reviewed 2026-09-04, against `master` after beta.17 publication.

## Closed, and how

Expand Down Expand Up @@ -72,7 +72,7 @@ previously ignored repro is a normal release-gating test and includes multiple
columns, indexes, update queries, and delete queries. This matters beyond
cosmetics because generated enum variant order can determine discriminants.

### The local beta.17 release-delta gate is green
### The beta.17 release-delta gate is green and published

The complete evidence and beta.13/beta.15/beta.17 performance grids are in
`docs/beta17-validation.md`. The exact local CI matrix passes in default,
Expand All @@ -88,11 +88,17 @@ a rollback-safe prefix, strict-loaded all six tables, performed an S3-backed
mutation and reloaded it after restart. ACME, HTTPS and WebSocket startup also
passed on Fly. Full evidence is in `docs/beta17-validation.md`.

## Blocking beta.17
WorkTable PR #87 merged after all six clean-checkout CI jobs passed, and
`worktable` 1.0.0-beta.17 plus its local dependency train are available on
crates.io. The benchmark changes and summaries merged through wt-benchmarks PR
#6. Post-publication resolution passed in AgencyZero PR #205; that consumer also
rebuilt and strict-opened the complete 18-table QA profile under beta.17.

There is no unresolved locally testable correctness or application-level
performance blocker. Branch/PR CI must still pass on the exact release commit
before publishing.
## Post-release administration

There is no unresolved beta.17 correctness or application-level performance
blocker. The release is published and its clean-checkout consumer smoke test
passes.

### Decide what happens to beta.16 on crates.io

Expand Down
63 changes: 48 additions & 15 deletions docs/beta17-validation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,35 @@ control exactly: 196 in-use pages and 100,000 live rows in all 18 of 18
backend/repetition cells. `off` retains 392 pages. This satisfies the 100%
reclamation requirement, not merely an approximate percentage.

## Reactive-maintenance precedent

The design has precedent in principle, but the exact gate appears unusual.
MySQL documents that InnoDB merges buffered secondary-index changes when the
server is nearly idle. RocksDB assigns user I/O higher priority than compaction
I/O and offers a rate limiter specifically to reserve bandwidth for online
queries. PostgreSQL autovacuum instead uses accumulated I/O cost and a timed
delay, and conflicting lock acquisition can interrupt a non-wraparound
autovacuum. InnoDB purge is driven by a periodic history list and can delay
foreground writes when purge lag grows too large.

Those systems establish the general policy that reclaim/merge work should
yield resources to foreground traffic. None of their official documentation
describes WorkTable's exact mechanism: one operation-wide mutation lease,
including bulk-operation chunk gaps, plus a completed-mutation epoch and a
check/recheck quiet buffer before the first vacuum batch. RocksDB in particular
does not simply block compaction while writes are active; when compaction falls
behind, its write controller can slow or stop the writes so maintenance catches
up.

Primary references:

- [InnoDB change-buffer merging](https://dev.mysql.com/doc/refman/8.4/en/innodb-change-buffer.html)
- [InnoDB purge scheduling and lag](https://dev.mysql.com/doc/refman/8.0/en/innodb-purge-configuration.html)
- [PostgreSQL cost-based vacuum delay](https://www.postgresql.org/docs/17/runtime-config-resource.html#RUNTIME-CONFIG-RESOURCE-VACUUM-COST)
- [PostgreSQL autovacuum lock interaction](https://www.postgresql.org/docs/15/routine-vacuuming.html)
- [RocksDB I/O priority and rate limiting](https://github.com/facebook/rocksdb/wiki/Rate-Limiter)
- [RocksDB write stalls when compaction falls behind](https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide)

## Beta.13 / beta.15 / beta.17 performance grid

Every version is a local WorkTable tree: beta.13 `48f250f`, beta.15 `e4dcfdf`,
Expand Down Expand Up @@ -256,20 +285,24 @@ as noise and not allowed to override the better application-level result.
local stack: 29 unit/invariant tests plus smoke execution of every Criterion
target, including all-backend KV/JSON, deletes, concurrency, PGO and YCSB.

## Remaining release TODO

1. Review and commit the WorkTable and `wt-benchmarks` changes, including the
WT DSL determinism fix, async benchmark correction, vacuum fix/stress and
validation documentation.
2. Write the beta.17 release notes from the coverage matrix above. The root
`CHANGELOG.md` still stops at 0.4.1 and does not describe the 1.0 beta line.
3. Run normal branch/PR CI on that committed candidate. Local CI is green, but
it is not a substitute for the repository's clean checkout and runner.
4. Decide the release-administration question in `docs/TODO.md`: whether to
yank ps-reclaim 0.1.0 and beta.16 after beta.17 supersedes them.
5. Publish the local dependency train in the required order, then perform the
post-publication consumer-resolution smoke test. Do not replace any of the
pre-release validation above with registry packages.
## Release closeout

The locally validated candidate was rebased and merged through
[WorkTable PR 87](https://github.com/pathscale/WorkTable/pull/87); all six CI
jobs passed on the exact release commit. The benchmark instruments and raw
summaries were rebased and merged through
[wt-benchmarks PR 6](https://github.com/pathscale/wt-benchmarks/pull/6).
WorkTable 1.0.0-beta.17 and its dependency train are published on crates.io.

Post-publication consumer resolution and persisted-data checks also pass.
support.cafe compiled and ran against the beta.17 train with its production S3
data, and [AgencyZero PR 205](https://github.com/pathscale/agencyzero/pull/205)
resolved beta.17 from a clean checkout, passed frontend and Rust CI, rebuilt
the complete 18-table QA profile, and strict-opened its 248 projects, 155 items,
and 2,212 messages.

No unresolved beta.17 correctness or application-level performance blocker
was found in the locally testable release surface.
was found. The remaining work is release administration: decide whether to
yank the superseded unsound `ps-reclaim` 0.1.0 and WorkTable beta.16 releases,
and eventually restore release-note continuity because the root `CHANGELOG.md`
still stops at 0.4.1.
Loading