feat(format): define hidden row lineage columns by BubbleCal · Pull Request #9253 · lance-format/lance

BubbleCal · 2026-09-15T21:02:49Z

First of a stack for §5.3 of the Stable Row ID GA design (#8931, tracked as #9250): let a fragment's row lineage sequences -- its row ids and its created-at and last-updated-at versions -- leave the manifest and live as hidden uint64 columns of a Lance data file. This PR defines the format; the stack continues with readers and the spill primitives, then compaction, the commit read-ahead, the update path, and compaction writing the columns into its own output files.

Builds on the prototype in #8953 (Will is co-author).

Stack

this PR feat(format): define hidden row lineage columns
feat(dataset): read and write spilled row lineage columns #9336 feat(dataset): read and write spilled row lineage columns
feat(dataset): spill row lineage at compaction #9337 feat(dataset): spill row lineage at compaction
feat(table): load spilled row lineage ahead of a commit #9338 feat(table): load spilled row lineage ahead of a commit
feat(dataset): spill row lineage when updating rows #9339 feat(dataset): spill row lineage when updating rows
feat(dataset): write compaction's spilled lineage into the fragment's data file #9347 feat(dataset): write compaction's spilled lineage into the fragment's data file
test(bench): measure row lineage placement against a checked workload #9340 test(bench): row lineage spill benchmark

Problem

Each sequence is stored inline in the fragment's manifest entry. An appended fragment's sequences are single runs and cost a few dozen bytes, but once compaction merges fragments whose rows came from many places the row id sequence degrades to 4-8 bytes per row and the version sequences to a run per row. The manifest then grows with the table's row count and every commit rewrites all of it (#8621).

Format

Every negative field id is reserved for system use and never names a schema field; readers skip any negative id in a data file's fields. -1 and -2 keep their meaning; -3, -4, -5 are the hidden _rowid, _row_created_at_version and _row_last_updated_at_version columns.
DataFragment gains an empty RowLineageColumn marker arm on each lineage oneof: column_row_ids = 12, column_last_updated_at_versions = 13, column_created_at_versions = 14. The marker carries no file reference: the column lives in one of the fragment's files, the single entry whose fields carry the reserved id, and its column_indices locates it like a user column. Zero or several carriers is corruption. The three sequences may share one file with each other or with the fragment's user data.
The columns have an executable schema: three non-nullable uint64 fields, each holding exactly physical_rows values in physical row order. A null or a length mismatch is corruption and is rejected, never defaulted.
The marker is valid only in a fragment whose data files are Lance v2 files: a legacy v1 DataFile has no column_indices to locate the column, and a fragment cannot mix v1 and v2 files. A writer on a v1 dataset leaves every sequence inline.
New feature flag FLAG_UNSTABLE_SPILLED_ROW_LINEAGE = 1 << 11 (value 2048), the next free bit after the two fragment-reuse flags; FLAG_UNKNOWN moves to 1 << 12. Every earlier build has its unknown boundary at or below bit 11 (v11.0.0 at 256, the v12/v13 pre-releases at 512, main before this PR at 2048), so each already refuses such a dataset. Like data overlay files, release builds understand the bit only with LANCE_ENABLE_UNSTABLE_SPILLED_ROW_LINEAGE=1; debug builds always do.
The external_* arms (field numbers 6, 8, 10) are retired: no writer ever emitted them and the column arms replace that design. Their numbers and names are reserved in the proto, and the External variants of RowIdMeta and RowDatasetVersionMeta go away with them. ExternalFile itself stays; the fragment reuse index uses it.

Placement rule, which the writers in later PRs follow: a value the commit assigns -- an appended fragment's row ids, an inserted row's created-at, every row's last-updated-at -- can change when a commit conflict is retried, so it stays inline where the retry can rewrite it. A value carried over from existing rows is fixed before the commit and may go to a data file.

Code

RowIdMeta::Column and RowDatasetVersionMeta::Column unit variants, with proto and JSON round trips ({"column": {}}, mirroring the empty proto message) and manifest interning. Fragment::row_lineage_file(field_id) finds the carrier among files and reports more than one as corruption. Because the carrier is an ordinary entry of files, cleanup, shallow-clone base_id rewriting, file listing and validation already cover it; validation skips the negative ids in a file's fields.
apply_feature_flags sets the flag when any fragment uses a column arm.
Nothing writes the columns yet. Reading one -- in the row id loader, the fragment reader, and the commit-time paths that resolve an update's lineage from existing fragments -- returns NotSupported instead of falling back to defaults.
assign_row_ids treats a spilled sequence as covering every physical row, as it always does.

Validation

cargo test -p lance-table
cargo test -p lance --lib -- rowid row_version stable_row optimize::tests dataset_transactions fragment::tests feature_flag update::tests merge_insert::tests dataset_io
cargo clippy --all --tests --benches -- -D warnings, cargo fmt --all
cargo check --manifest-path python/Cargo.toml, cargo check --manifest-path java/lance-jni/Cargo.toml

Refs #8931, #9250

🤖 Generated with Claude Code

github-actions · 2026-09-17T08:56:46Z

Important

Format specification vote

This PR modifies the Lance format specification, so it requires 3 binding +1 votes from PMC members (excluding the proposer), at least one of them on the latest commit, and a minimum 72-hour voting period, weekends excluded, before it can merge. Vote by approving this PR (+1) or requesting changes (−1, a veto). See the voting process.

Approvals carry over across pushes, so a rebase or a typo fix does not send everyone back to re-vote. Whoever approves the latest commit is vouching that nothing substantive has changed since the earlier approvals; if something has, ask for fresh votes.

Status: ✅ Vote passed — 3 PMC approvals, voting period elapsed


Approvals	@westonpace, @wjones127, @Xuanwo (3/3)
Latest commit approved by	@wjones127, @Xuanwo
Vetoes	none
Voting period	elapsed — ended Tue 2026-09-22 08:56 UTC (01:56 PDT)

_{Updated automatically by the format-spec vote gate, which re-checks every 15 minutes — just voted? Re-check now (press Run workflow; leave the input blank to re-check every open format PR). A PMC member may apply the format-waived label to waive the vote for a trivial edit (typo, wording, formatting).}

BubbleCal · 2026-09-17T10:54:08Z

Addressed the three gatekeeper points, here and in the stack:

Single owner for the carrier file: the column arms are now empty RowLineageColumn markers, the file is one of DataFragment.files found by its reserved field id, and zero or several carriers is corruption.
Executable schema: the three columns are defined with pa.schema as non-nullable uint64, exactly physical_rows values each; nulls or a length mismatch are rejected, never defaulted.
V1: the marker is valid only next to v2 data files, and a v1 dataset never spills.

The rest of the stack (#9336 through #9340, plus the new #9347, which writes compaction's spilled columns into the fragment's own data file) is rebased on this representation.

lance-gatekeeper

✅ Gate recommendation: approve.

3 fixed. The follow-up is borne out by the current head: DataFragment.files is the single carrier authority, the hidden columns have an executable non-nullable uint64 schema with exact row-count and corruption semantics, and the marker is V2-only while V1 writers remain inline. The feature flag fences unsupported readers and writers.

westonpace

Vote / review on the spec changes only

westonpace · 2026-09-17T13:08:15Z

+`-1` (not yet assigned; only ever exists in memory and must not be written) and the
+`-2` tombstone above, `-3`, `-4` and `-5` are the hidden `_rowid`,
+`_row_created_at_version` and `_row_last_updated_at_version` columns that hold a
+fragment's row lineage sequences when they are not stored inline; see


Suggested change

fragment's row lineage sequences when they are not stored inline; see

fragment's row lineage sequences when they are not stored in the manifest; see

"inline" seems a little ambiguous since it could be read as "inline in the manifest" (intended) or "inline with the data" (not intended)

westonpace · 2026-09-17T13:21:33Z

+version, so on a table whose sequences do not run-encode well -- the output of
+compacting a globally shuffled table, for instance -- the manifest grows with the total
+row count and every commit rewrites all of it.


Suggested change

version, so on a table whose sequences do not run-encode well -- the output of

compacting a globally shuffled table, for instance -- the manifest grows with the total

row count and every commit rewrites all of it.

version.

We just said it's only for small (200KB sequences). I think these last sentences describe the old state, and not something that exists. See my comment above for a different spot to put this motivation.

westonpace · 2026-09-17T13:22:52Z

+`DataFragment` defines inline, external, and column alternatives for row ID sequences
+and row version sequences.


Suggested change

`DataFragment` defines inline, external, and column alternatives for row ID sequences

and row version sequences.

`DataFragment` defines inline, external, and column alternatives for row ID sequences

and row version sequences. This allows small sequences to stay in the manifest (fewer IOPS) while larger sequences (resulting from frequent updates) move outside the manifest.

westonpace · 2026-09-17T13:24:44Z

+or `column_last_updated_at_versions`) is an empty `RowLineageColumn` marker: it carries
+no file reference, because the file is one of the fragment's `files` and is found by a
+reserved negative field id in that entry's `fields`. Exactly one file of the fragment
+carries each id; zero or more than one is corruption. The `column_indices` entry paired


Suggested change

carries each id; zero or more than one is corruption. The `column_indices` entry paired

carries each reserved field; zero or more than one is corruption. The `column_indices` entry paired

id is a little ambiguous because it's not clear if we are talking about field ids or row ids.

westonpace · 2026-09-17T13:25:56Z

+The marker is valid only in a fragment whose data files are Lance v2 files. A legacy
+v1 data file has no `column_indices` to locate the column by, and a fragment cannot
+mix v1 and v2 files, so a writer on a dataset that stores v1 files MUST leave every
+sequence inline; a marker in a fragment with a v1 data file is corruption.


We could maybe put a more general blanket statement just say somewhere that the stable row id feature is only enabled for v2 files.

westonpace · 2026-09-17T13:26:40Z

+    pa.field("_rowid", pa.uint64(), nullable=False),
+    pa.field("_row_created_at_version", pa.uint64(), nullable=False),
+    pa.field("_row_last_updated_at_version", pa.uint64(), nullable=False),


Does that mean these are reserved field names? Do we prevent users from writing them?

Created: #9455

westonpace · 2026-09-17T13:28:03Z

+Each column present holds exactly `physical_rows` values, one per physical row in
+physical row order, deleted rows included; the value at offset `i` is the row id or
+version of the row at offset `i`, the same thing the inline encoding's `i`-th entry
+would be. A null value, or a column whose length differs from `physical_rows`, is
+corruption and MUST be rejected rather than read as a default. A file may carry any
+subset of the three columns; a sequence whose arm is not the column marker is not read
+from any file, whatever the file holds.


You can keep this paragraph if you want but it seems kind of redundant.

westonpace · 2026-09-17T13:30:54Z

+The `external_*` arms predate this and are raw byte ranges holding the same encoded
+form as the inline arms. Current Lance writers do not emit them. Current Lance readers
+can load `external_row_ids`; the external version-sequence arms are valid encodings
+that current readers cannot load.


The last two sentences are a bit confusing. Are you saying that current readers can read/load them? Or current reads cannot read/load them?

westonpace

How does "find-by-id" work when we have row ids stored in the data files? Do we load them into memory and cache them? Do we sort them as we do this? If we have a billion scale table then what does the open dataset cost look like to load and sort a billion row ids?

BubbleCal · 2026-09-17T13:47:44Z

How does "find-by-id" work when we have row ids stored in the data files? Do we load them into memory and cache them? Do we sort them as we do this? If we have a billion scale table then what does the open dataset cost look like to load and sort a billion row ids?

Yes, we load them into memory and sort them, loading happens when we take rows by id not when we open the table. The long term solution is we will have a btree index over row ids, but that won't be included in this stack.

wjones127 · 2026-09-21T21:19:27Z

How does "find-by-id" work when we have row ids stored in the data files? Do we load them into memory and cache them? Do we sort them as we do this? If we have a billion scale table then what does the open dataset cost look like to load and sort a billion row ids?

Goal will be to make BTREE support the _rowid field.

wjones127

This is looking pretty good, but my one request is that we deprecate External files at the same time. That will make it clearer this is a replacement for that design.

Reserve every negative field id for system columns and give each of a fragment's three row lineage sequences -- row ids (field -3), created-at versions (-4) and last-updated-at versions (-5) -- an empty marker arm on its DataFragment oneof, so a sequence too large to keep inline can live as a hidden uint64 column of one of the fragment's data files, found by its reserved field id. Placement follows when a value is known: sequences the commit assigns stay inline where a conflict retry can rewrite them; sequences carried over from existing rows may go to a data file. The external_* arms of the same oneofs, which no writer ever emitted, are retired at the same time: their field numbers and names are reserved and the External variants of RowIdMeta and RowDatasetVersionMeta go away with them. The column arms replace that design. The new feature flag takes bit 11 and FLAG_UNKNOWN moves to bit 12; every earlier build has its boundary at or below bit 11, so each refuses such a dataset. This build understands it in debug builds or with LANCE_ENABLE_UNSTABLE_SPILLED_ROW_LINEAGE=1. Readers skip any negative id in a data file's fields so the columns may share the fragment's main data file. The marker is valid only next to v2 data files. Nothing writes the columns yet. Reading one is refused with NotSupported rather than falling back to defaults. Co-authored-by: Will Jones <willjones127@gmail.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

wjones127

Looks good now. Thanks!

Second of the §5.3 stack (#8931, #9250), on top of #9253. The previous PR defined the hidden row lineage columns; this one reads them and adds the primitives that write them. Nothing calls the writer yet; compaction does in the next PR. ## Stack 1. #9253 feat(format): define hidden row lineage columns 2. this PR feat(dataset): read and write spilled row lineage columns 3. #9337 feat(dataset): spill row lineage at compaction 4. #9338 feat(table): load spilled row lineage ahead of a commit 5. #9339 feat(dataset): spill row lineage when updating rows 6. #9347 feat(dataset): write compaction's spilled lineage into the fragment's data file 7. #9340 test(bench): row lineage spill benchmark ## Read - `load_row_id_sequence` gains a `Column` arm that reads the `_rowid` column back through the ordinary file reader, projected by field id, into the same per-fragment cache as inline sequences. - New `load_row_version_sequence(dataset, fragment, RowVersionKind)` loads either version sequence wherever it is stored; a spilled one is cached per fragment and file, an inline one decodes from the manifest bytes as before. - `FileFragment::open` loads the version sequences asynchronously alongside the row ids and hands them to the reader, instead of the reader builder decoding them synchronously and silently falling back to version 1 on any failure. The `NotSupported` guard from the previous PR goes away with it. - `Dataset::validate` checks the length of spilled version sequences too. ## Write primitives `place_row_lineage(dataset, &RowLineage)` encodes a fragment's three sequences and, for each one whose encoding exceeds the table's inline budget, writes it as a column of one new lineage file per fragment. It returns a `PlacedRowLineage`: the three arms to put on the fragment plus the lineage file, which `apply` adds to the fragment's `files` so the marker can be resolved. It is only correct for lineage a commit conflict cannot change, which is what its callers carry over from existing rows. Spilling is opt-in per table: `lance.row_lineage.spill=true`, with `lance.row_lineage.inline_max_bytes` overriding the 200 KiB default. A table that never sets it is unchanged, and a build that does not understand the feature flag never spills either, so it cannot write a dataset it then refuses to open. A legacy v1 dataset never spills regardless of its config, since the format only allows the columns in v2 files. The primitives are public so a writer outside this crate that assembles its own transactions can spill at write time. ## Validation - `cargo test -p lance --lib rowids::` (round trip of all three columns through one shared file, found by field id among the fragment's files; only the sequences over the budget spill; a table that has not opted in never spills; a v1 table never spills) - `cargo test -p lance --lib -- rowid stable_row fragment::tests dataset_transactions` - `cargo clippy --all --tests --benches -- -D warnings`, `cargo fmt --all` - `cargo check --manifest-path python/Cargo.toml`, `cargo check --manifest-path java/lance-jni/Cargo.toml` Refs #8931, #9250 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Will Jones <willjones127@gmail.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

Third of the §5.3 stack (#8931, #9250), on top of #9336. Compaction becomes the first writer of the hidden row lineage columns. ## Stack 1. #9253 feat(format): define hidden row lineage columns 2. #9336 feat(dataset): read and write spilled row lineage columns 3. this PR feat(dataset): spill row lineage at compaction 4. #9338 feat(table): load spilled row lineage ahead of a commit 5. #9339 feat(dataset): spill row lineage when updating rows 6. #9347 feat(dataset): write compaction's spilled lineage into the fragment's data file 7. #9340 test(bench): row lineage spill benchmark ## Change Compaction used to rechunk the row ids and the two version sequences in two separate passes and write each back inline. It now computes all three together, then places each one through `place_row_lineage`: inline when its encoding fits the table's budget, otherwise as a column of one lineage file per output fragment, listed among the fragment's files after its data file. The sixth PR of the stack moves those columns into the data file itself. Compaction's output is retry-stable -- every value is carried over from the input fragments -- so spilling it needs nothing from the commit. Only tables that set `lance.row_lineage.spill=true` are affected; a table that never opts in compacts exactly as before. ## Not in this PR Updating rows of a table whose compaction has spilled still returns `NotSupported` from the commit, which cannot read a data file; the next PR lifts that. With this PR alone, a user who opts in must not update the table until then, which is why the flag stays unstable and the config defaults off. ## Benchmarks `LANCE_ENABLE_UNSTABLE_SPILLED_ROW_LINEAGE=1 cargo bench --bench rowid_spill` (the benchmark lands as the last PR of the stack), 8 fragments x 1,000,000 rows, local NVMe (macOS), one run per arm, measured on the pre-split branch whose compaction and read code is what this stack carries. The two arms differ only in the table config: the inline arm never opts in (today's behavior), the spilled arm sets `lance.row_lineage.spill=true` with the default 200 KiB budget. Byte counts are deterministic; latencies are single samples (cold open averaged over 10, sequence loads over 3), so treat differences under about 2x as noise. Lower is better for every row. **deleted**: 30% of rows deleted, then compacted. The row id sequences still run-encode as range plus bitmap, so this is the workload where spilling is a bad trade on bytes. | Scenario / metric | Baseline (inline) | This PR (spilled) | Benefit | | --- | ---: | ---: | ---: | | manifest size | 5.73 MiB | < 0.01 MiB | >500x smaller | | compaction transaction file | 2.86 MiB | < 0.01 MiB | >500x smaller | | cold dataset open | 1.45 ms | 0.55 ms | 2.6x speedup | | append commit (mean of 5) | 3.65 ms | 1.46 ms | 2.5x speedup | | load one fragment's sequence (cold) | 0.15 ms | 4.88 ms | 33x slowdown | | row id index build (cold) | 0.11 ms | 16.34 ms | 150x slowdown | | take by row id, index built | 0.54 ms | 4.55 ms | 8x slowdown (first touch, see below) | | compaction | 221 ms | 258 ms | 1.2x slowdown | | data files on disk | 74.4 MiB | 89.2 MiB | 1.2x larger | **shuffled**: every row rewritten in random order, then compacted. No run structure survives, which is the workload the design is for. | Scenario / metric | Baseline (inline) | This PR (spilled) | Benefit | | --- | ---: | ---: | ---: | | manifest size | 30.52 MiB | < 0.01 MiB | >3000x smaller | | compaction transaction file | 61.04 MiB | 30.52 MiB | 2x smaller | | cold dataset open | 5.54 ms | 0.20 ms | 28x speedup | | append commit (mean of 5) | 15.73 ms | 1.34 ms | 12x speedup | | load one fragment's sequence (cold) | 2.35 ms | 2.30 ms | 1.0x | | row id index build (cold) | 142 ms | 165 ms | 1.2x slowdown | | take by row id, index built | 2.65 ms | 2.75 ms | 1.0x | | compaction | 305 ms | 296 ms | 1.0x | | data files on disk | 136.0 MiB | 158.1 MiB | 1.2x larger | The `deleted` take row: `FileFragment::open` loads the fragment's row id sequence whenever the table uses stable row ids, whether or not `_rowid` is projected, and the index build reads sequences uncached, so the first take after it pays one spilled-file read per fragment. That eager load predates this stack; making it conditional on the projection is a follow-up. ## Validation - `cargo test -p lance --lib rowids::` (compaction spills all three sequences into one lineage file among the fragment's files and reads them back through the scan and the loaders; cleanup keeps the live lineage file; a cold reopen serves the columns) - `cargo test -p lance --lib -- rowid stable_row optimize::tests cleanup::tests dataset_transactions` - `cargo clippy --all --tests --benches -- -D warnings`, `cargo fmt --all` Refs #8931, #9250 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Will Jones <willjones127@gmail.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

Fourth of the §5.3 stack (#8931, #9250), on top of #9337. Updates on a table with spilled lineage keep every row's lineage, the way they do on an inline table. ## Stack 1. #9253 feat(format): define hidden row lineage columns 2. #9336 feat(dataset): read and write spilled row lineage columns 3. #9337 feat(dataset): spill row lineage at compaction 4. this PR feat(table): load spilled row lineage ahead of a commit 5. #9339 feat(dataset): spill row lineage when updating rows 6. #9347 feat(dataset): write compaction's spilled lineage into the fragment's data file 7. #9340 test(bench): row lineage spill benchmark ## Problem Building a manifest is synchronous and has no object store. Two commit-time paths need existing lineage: resolving which rows an update rewrote, to carry each row's created-at version over (`resolve_update_version_metadata`), and overlaying a partial column rewrite's patched offsets onto the fragment's last-updated-at sequence (`refresh_row_latest_update_meta_for_partial_frag_rewrite_cols`). Once a fragment's sequences are spilled, neither can read them, and the previous PRs made them refuse with `NotSupported`. ## Change The commit path in `lance` reads every spilled sequence of the current manifest ahead of each build attempt (`load_spilled_row_lineage`, served from the same caches the readers use) and hands them over in `ManifestBuildConfig::spilled_row_lineage`. The two paths consult that map for a spilled fragment and still refuse if a sequence they need is missing, so a caller of `build_manifest` that skips the read-ahead gets an error rather than defaulted lineage. Loaded per attempt, so a rebase onto a newer manifest sees that manifest's fragments; only `Update` and `DataOverlay` operations pay for it. `UpdateBuilder`, `merge_insert` in both write modes, and externally assembled `Operation::Update`s therefore work on a spilled table unchanged. The lineage they produce for the rewritten rows is inline; a refreshed last-updated-at sequence goes back inline as well. An update-heavy table thus regrows its manifest between compactions and the next compaction spills it again. ## Validation - `cargo test -p lance-table row_version` (spilled source lineage resolves from the config; missing lineage still refuses) - `cargo test -p lance --lib rowids::` (update keeps the rewritten row's id and created-at; merge_insert keeps matched rows' lineage and stamps inserted ones; a partial column rewrite on a spilled fragment stamps only the patched rows) - `cargo test -p lance --lib -- rowid row_version stable_row update::tests merge_insert::tests dataset_transactions` - `cargo clippy --all --tests --benches -- -D warnings`, `cargo fmt --all` Refs #8931, #9250 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

Part of the §5.3 stack (#8931, #9250). #9253, #9336, #9337 and #9338 are merged; this PR is now based on `main`. Updates on a table that opts into spilling move the lineage of the rows they rewrite out of the manifest, so an update-heavy table no longer regrows its manifest between compactions. It also fixes schema-only commits dropping the file a spilled sequence lives in, which has been reachable since #9337. ## Stack 1. #9253 feat(format): define hidden row lineage columns (merged) 2. #9336 feat(dataset): read and write spilled row lineage columns (merged) 3. #9337 feat(dataset): spill row lineage at compaction (merged) 4. #9338 feat(table): load spilled row lineage ahead of a commit (merged) 5. this PR: feat(dataset): spill row lineage when updating rows 6. #9347 feat(dataset): write compaction's spilled lineage into the fragment's data file 7. #9340 test(bench): row lineage spill benchmark ## Change **Update places the lineage it carries over, only when it spills.** - `UpdateBuilder` resolves the table's inline budget before any IO. A malformed `lance.row_lineage.inline_max_bytes` now fails the update before anything is written. - Only on a table that can spill (opted in, v2 files, a build that understands the flag) does the scan project `_row_created_at_version` next to `_rowid`. `make_rowid_capture_stream` captures it run-length encoded, so a rewrite of N rows written at a few versions costs a few runs rather than 8 bytes per row. - After the write, each new fragment's row ids and created-at versions are split to its size. They leave the manifest only when one of them is over the budget, into a lineage-only file added to the fragment's `files`. Last-updated-at is never placed by the writer; it is the commit's. - A table that never opted in, or an update whose sequences fit inline, leaves created-at to the commit exactly as `main` does. **Commit contract** (`resolve_update_version_metadata`, `has_writer_placed_lineage`): - A new fragment whose row ids or created-at versions are `Column` is writer-placed. It keeps its created-at and gets last-updated-at stamped with the commit version. - Placed inline created-at must have one value per physical row, or the commit is refused. - Every other fragment is resolved from the existing fragments as before. A caller-supplied inline created-at is still recomputed, which keeps the documented contract for fragments committed from Python. - The commit skips the created-at lookup for writer-placed fragments, and the read-ahead from #9338 is skipped for an update whose new fragments are all writer-placed. **Spilled lineage survives schema-only commits.** `Operation::Project` (drop, rename, nullability), `DataReplacement` and the cast in `alter_columns` kept only files with a live schema field, so they dropped the only carrier of a spilled sequence and left a `Column` arm pointing at nothing. After cleanup, those row ids and versions were gone. All three now keep a file that carries one of the fragment's spilled ids. `build_manifest` also refuses to commit a fragment whose `Column` arm has no carrier, or a carrier in a v1 file, so a future path cannot commit that state silently. `merge_insert` is unchanged: its rewritten rows still get inline lineage resolved at commit time. ## Validation Run on an x86 build host (r8i.8xlarge): - `cargo fmt --all -- --check`, `cargo clippy --all --tests --benches -- -D warnings` - `cargo test -p lance-table`: 573 passed, including `project_keeps_file_carrying_spilled_row_lineage`, `data_replacement_keeps_lineage_carrier`, `build_manifest_rejects_spilled_arm_without_carrier` and the new `row_version` contract tests. - `cargo test -p lance --lib -- rowids:: row_version stable_row rowid optimize::tests dataset_transactions fragment::tests cleanup::tests feature_flag update::tests merge_insert::tests io::commit dataset_io schema_evolution versions:: write::tests utils::tests`: 1545 passed, including `schema_change_keeps_spilled_lineage` (update or compaction followed by rename, drop or cast), `update_leaves_inline_created_at_to_the_commit`, `update_rejects_malformed_inline_max_bytes_before_writing` and `place_rewritten_lineage_splits_lineage_by_output_fragment`. - `cargo check --manifest-path python/Cargo.toml`, `cargo check --manifest-path java/lance-jni/Cargo.toml` Refs #8931, #9250 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

… data file (#9347) Part of the §5.3 stack (#8931, #9250); based on `main` now that #9339 has merged. Compaction writes the lineage sequences it spills into each output fragment's own data file, as hidden columns next to the user columns, instead of a separate lineage file. It also fixes three bugs in reading, placing and reclaiming spilled lineage that the review of this stack found. ## Stack 1. #9253, #9336, #9337, #9338, #9339 (merged) 2. this PR: feat(dataset): write compaction's spilled lineage into the fragment's data file 3. #9340 test(bench): row lineage spill benchmark (lands after this PR) ## Change **Lineage in the data file.** - On a table that can spill, compaction computes its output fragments' lineage before writing. The row ids and versions carry over from the inputs. - A sequence type that is over the inline budget in every output fragment rides along as an extra non-nullable `uint64` column of the batches being written. The fragment's metadata then marks it as spilled into that same data file. - A type that is over budget in some outputs and not others (`RowLineagePlan::PerFragment`) writes no hidden columns. After the write, each fragment is placed on its own, so a cheap `Range` sequence never turns into a column read. - Tables that never opted in compact exactly as before: lineage is rechunked after the write from the written sizes. - `versions::write_fragments` sets the hidden fields aside by their reserved ids before the schema check and puts them back on the written schema. `Schema::validate` admits the reserved ids only on top-level fields with those names. - Binary-copy compaction cannot add columns and still writes a separate lineage file. **Fixes.** - **Written sizes, not planned sizes.** A byte limit (`max_bytes_per_file`) can close a file early and re-split the rows, so the plan's file sizes are not the written ones. Placement used to zip the plan with the written fragments. That made a stable-row-id compaction fail outright, or shift row ids by one when the fragment count happened to match. It now rechunks the planned lineage to each fragment's `physical_rows` and checks every inline sequence's length. - **Reading a spilled column.** `read_spilled_column` took the projected field from the file schema by column index. Once lineage columns follow nested or list user columns, the column index is no longer the top-level position, so the read errored or took the wrong field. The field is now built from the reserved id. Only the needed column's metadata is opened. - **Carriers whose user columns are gone.** Now that #9339 keeps carriers through drop, cast and replace, a data file can hold lineage and only dead user columns. `FileFragment::validate` no longer tries to open it as user data. The compaction planner picks such a fragment up, so the lineage moves into a fresh file and the dead bytes are reclaimed. Lineage-only files never match, so this cannot loop. **Memory.** Spilled sequences are moved into their column values rather than copied. Each batch gets its own buffer, so the encoder no longer cuts a page per batch from a shared slice. Pre-write planning only happens on tables that can spill. ## Validation Run on an x86 build host (r8i.8xlarge): - `cargo fmt --all -- --check`, `cargo clippy --all --tests --benches -- -D warnings` - `cargo test -p lance-table`: 573 passed. - `cargo test -p lance --lib -- rowids:: row_version stable_row rowid optimize::tests dataset_transactions fragment::tests cleanup::tests feature_flag update::tests merge_insert::tests io::commit dataset_io schema_evolution versions:: write::tests utils::tests`: 1576 passed. - New tests: - `compaction_spills_and_reads_back_row_lineage` cases: `flat`, `nested_v2_2`, `list_v2_0`, `multi_output`, `with_deletions`. - `compact_twice_reads_back_in_file_lineage`. - `place_row_lineage_in_files_follows_written_sizes`: extra file, shifted split, mismatched totals. - `plan_row_lineage_spill_decides_per_kind` and `per_fragment_plan_spills_only_the_fragments_over_budget`. - `append_row_lineage_columns_gives_each_batch_its_own_buffer`. - `dropping_every_column_of_a_lineage_carrier_keeps_lineage_until_compaction`: drop, cast, replace. - `compaction_reclaims_lineage_carriers_of_dead_user_columns`. - `cleanup_keeps_a_live_spilled_file`: in-file and lineage-file layouts. - `split_row_lineage_fields_keys_on_reserved_ids`. - `cargo check --manifest-path python/Cargo.toml`, `cargo check --manifest-path java/lance-jni/Cargo.toml` Refs #8931, #9250 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

…#9340) Last of the §5.3 stack (#8931, #9250). The benchmark for row lineage placement, inline in the manifest versus spilled to data file columns. Everything it measures is on `main`: this PR only adds `rust/lance/benches/rowid_spill.rs` and its `[[bench]]` entry in `rust/lance/Cargo.toml`. ## Stack 1. #9253, #9336, #9337, #9338, #9339, #9347 (merged) 2. this PR: test(bench): measure row lineage placement against a checked workload ## What it measures ```bash LANCE_ENABLE_UNSTABLE_SPILLED_ROW_LINEAGE=1 cargo bench -p lance --profile release-with-debug --bench rowid_spill ``` It is sized with `BENCH_FRAGMENTS`, `BENCH_ROWS_PER_FRAGMENT`, `BENCH_DELETE_PERCENT`, `BENCH_APPENDS`, `BENCH_SCENARIOS` and `BENCH_INLINE_MAX_BYTES`. Bad values are rejected up front. There are two workloads: - `deleted`: a share of rows is deleted, then compacted. The sequences still run-encode. - `shuffled`: every row is rewritten in random order, with its real created-at and last-updated-at versions, then compacted. No run structure survives. Within each workload, the inline arm never opts in, and the spilled arm sets `lance.row_lineage.spill=true` with the given budget. Per arm it reports: - steady-state manifest bytes, split by sequence; - the compaction manifest and transaction file; - cold open; - the commit of a small append, timed apart from the data write; - one cold sequence load; - the row id index build; - a take by row id with the fragment's sequence preloaded; - compaction time and the bytes compaction wrote. Timed rows report the median and the minimum of their samples. The bench checks itself outside the timers: - the spilled arm must actually have spilled, and the inline arm must not; - a full scan asserts every row's `_rowid`, created-at and last-updated-at against the workload; - every take probe must return the row it asked for. ## Smoke run This is one small run, not a performance claim. It ran on an x86 r8i.8xlarge (local NVMe) with 2 fragments x 200,000 rows, `BENCH_INLINE_MAX_BYTES=0` so everything spills, the `release-with-debug` profile, and each arm once. Lower is better for every row, and the ratio is inline / spilled. | Scenario / metric | Inline | Spilled | Ratio | | --- | ---: | ---: | ---: | | shuffled: steady-state manifest | 7.49 MB | < 0.01 MB | 10367x smaller | | shuffled: cold dataset open (median) | 4.32 ms | 0.12 ms | 36.5x faster | | shuffled: append commit (median) | 6.27 ms | 0.78 ms | 8.1x faster | | shuffled: row id index build, cold (median) | 36.11 ms | 35.53 ms | 1.0x | | shuffled: one sequence load, cold (median) | 0.18 ms | 1.97 ms | 0.09x (spilled is slower) | | shuffled: compaction output (data files) | 1.93 MB | 2.96 MB | 0.65x (spilled is larger) | | deleted: steady-state manifest | 0.14 MB | < 0.01 MB | 237x smaller | | deleted: row id index build, cold (median) | 2.53 ms | 5.48 ms | 0.46x (spilled is slower) | The trade-off is the one the design expects. Spilling takes the per-row lineage out of every manifest read and write, so open and commit stop scaling with the table. The cost moves to a column read the first time a sequence is needed, and 2 to 3 bytes per row in the compacted data files. On the `deleted` workload the sequences are small either way, which is why the default 200 KiB budget keeps them inline. ## Validation - `cargo fmt --all -- --check`, `cargo clippy --all --tests --benches -- -D warnings` on the head of this branch - The smoke run above completed with every self-check passing. Refs #8931, #9250 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Will Jones <willjones127@gmail.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

github-actions Bot added A-python Python bindings A-format On-disk format: protos and format spec docs format-change A change to the format spec, which requires a vote. Remove if minor (e.g. fixing typo). enhancement New feature or request labels Sep 15, 2026

BubbleCal force-pushed the yang/oss-2269-row-lineage-spill-column branch 3 times, most recently from e952b51 to f5faf25 Compare September 17, 2026 07:33

BubbleCal changed the title ~~feat(format): spill row lineage sequences to hidden data file columns~~ feat(format): define hidden row lineage columns Sep 17, 2026

BubbleCal force-pushed the yang/oss-2269-row-lineage-spill-column branch 2 times, most recently from b2491b6 to e1b6002 Compare September 17, 2026 08:47

BubbleCal added this pull request to stack #9343 September 17, 2026 08:56

BubbleCal marked this pull request as ready for review September 17, 2026 08:56

This comment was marked as outdated.

Sign in to view

lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Sep 17, 2026

BubbleCal force-pushed the yang/oss-2269-row-lineage-spill-column branch from e1b6002 to cda87e8 Compare September 17, 2026 09:28

lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Sep 17, 2026

This comment was marked as outdated.

Sign in to view

lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Sep 17, 2026

BubbleCal force-pushed the yang/oss-2269-row-lineage-spill-column branch from cda87e8 to 99bc517 Compare September 17, 2026 10:52

BubbleCal mentioned this pull request Sep 17, 2026

feat(dataset): write compaction's spilled lineage into the fragment's data file #9347

Merged

lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Sep 17, 2026

lance-gatekeeper Bot reviewed Sep 17, 2026

View reviewed changes

lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 17, 2026

westonpace approved these changes Sep 17, 2026

View reviewed changes

westonpace reviewed Sep 17, 2026

View reviewed changes

BubbleCal force-pushed the yang/oss-2269-row-lineage-spill-column branch from 99bc517 to 67716a2 Compare September 17, 2026 13:39

lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 17, 2026

BubbleCal force-pushed the yang/oss-2269-row-lineage-spill-column branch 2 times, most recently from b969952 to 79b878d Compare September 19, 2026 07:49

wjones127 requested changes Sep 21, 2026

View reviewed changes

Comment thread protos/table.proto

Comment thread protos/table.proto Outdated

Comment thread rust/lance-table/src/rowids/version.rs Outdated

Comment thread rust/lance-table/src/transaction/row_version.rs

wjones127 mentioned this pull request Sep 22, 2026

feat(transaction): implement the Transaction V2 action vocabulary #8644

Draft

BubbleCal force-pushed the yang/oss-2269-row-lineage-spill-column branch from 79b878d to 36df220 Compare September 22, 2026 04:46

BubbleCal requested review from westonpace and wjones127 September 22, 2026 08:54

wjones127 approved these changes Sep 22, 2026

View reviewed changes

Xuanwo approved these changes Sep 23, 2026

View reviewed changes

BubbleCal merged commit 06b77c7 into main Sep 23, 2026
39 of 40 checks passed

BubbleCal deleted the yang/oss-2269-row-lineage-spill-column branch September 23, 2026 10:46

BubbleCal mentioned this pull request Sep 23, 2026

feat(format): separate index keys from covering fields #9159

Merged

	fragment's row lineage sequences when they are not stored inline; see
	fragment's row lineage sequences when they are not stored in the manifest; see

		`DataFragment` defines inline, external, and column alternatives for row ID sequences
		and row version sequences.

	carries each id; zero or more than one is corruption. The `column_indices` entry paired
	carries each reserved field; zero or more than one is corruption. The `column_indices` entry paired

Conversation

BubbleCal commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Stack

Problem

Format

Code

Validation

Uh oh!

github-actions Bot commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Format specification vote

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

BubbleCal commented Sep 17, 2026

Uh oh!

lance-gatekeeper Bot left a comment

Choose a reason for hiding this comment

Uh oh!

westonpace left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

westonpace left a comment

Choose a reason for hiding this comment

Uh oh!

BubbleCal commented Sep 17, 2026

Uh oh!

wjones127 commented Sep 21, 2026

Uh oh!

wjones127 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wjones127 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

BubbleCal commented Sep 15, 2026 •

edited

Loading

github-actions Bot commented Sep 17, 2026 •

edited

Loading