feat: stabilize field IDs across schema evolution by Xuanwo · Pull Request #8658 · lance-format/lance

Xuanwo · 2026-08-20T09:44:58Z

Field IDs are physical bindings in Lance, but legacy allocation derives the next ID from fields that the current snapshot still references. After a field and its files disappear, a later field can reuse the same integer. A bare field ID is therefore not a durable identity across schema evolution.

This PR adds an explicit, writer-side stable field-ID contract:

New and existing datasets keep legacy behavior until a Rust caller runs migrate_to_stable_field_ids.
Migration writes both max_allocated_field_id and FLAG_STABLE_FIELD_IDS. Both values must be present for stable mode or absent for legacy mode; either mismatch is invalid. Stable field IDs do not set a reader flag.
In stable mode, each new or replacement field ID must be greater than the previous high-water mark. IDs may be sparse, which leaves room for future reservation.
Rename, reorder, metadata changes, and compatible overwrite preserve identity. Type replacement allocates a new identity.
Incoming Arrow field-ID metadata is descriptive input, not allocation authority for a new field in stable mode. The commit path assigns IDs and updates new file mappings.
Activation is one-way. Restore to a pre-activation version is rejected because an ID in that version may refer to a different field than it does at activation; retaining the current high-water mark cannot resolve that conflict.

The migration API is Rust-only in this PR. Python and Java enforce the contract when they operate on an activated dataset, but they do not expose activation yet.

The format documentation also defines how Blob logical fields participate in dataset field identity while writer-prepared and stored descriptor children remain file-local representation details.

Validation includes Rust stable/legacy migration, schema evolution, retry/rebase, restore, clone, detached commit, Blob identity, and Arrow canonicalization coverage, plus Java Merge, Project, and Overwrite boundary coverage.

github-actions · 2026-08-20T09:45:06Z

Important

This PR touches the Lance format specification.

Substantive changes to the format specification — the .proto definitions
and the spec docs under docs/src/format/ — require a PMC vote before merge.
Minor edits such as typo fixes, wording, or formatting are excluded; use your
judgment.

If this is a meaningful format change:

Start a vote following the Lance community voting process.
Format specification modifications need 3 binding +1 votes (excluding the
proposer), held on GitHub Discussions, with a minimum voting period of 1 week.
Once the vote passes, link the completed vote in this PR. It should not be
merged until the vote is linked.

westonpace · 2026-08-20T16:59:43Z

Do you think this will cause transactions to conflict with each other that did not before? I don't think it does today since we treat any schema change as conflicting with any other schema change.

Do we know why we reused field ids in the first place?

codecov · 2026-08-20T19:29:15Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

wjones127

The spec looks a lot better. Reviewed the implementation this time.

wjones127 · 2026-09-04T23:41:01Z

+The writer may temporarily add `kind`, `blob_id`, `blob_size`, and `position`. A Lance data file
+stores this descriptor shape:
+
+```python
+pa.schema([
+    pa.field(
+        "image",
+        pa.struct([
+            pa.field("kind", pa.uint8(), nullable=False),
+            pa.field("position", pa.uint64(), nullable=False),
+            pa.field("size", pa.uint64(), nullable=False),
+            pa.field("blob_id", pa.uint32(), nullable=False),
+            pa.field("blob_uri", pa.string(), nullable=False),
+        ]),
+    ),
+])
+```


question(non-blocking): when these fields are stored, how is that represented in both DataFile.fields and DataFile.column_indices? Below it says DataFile.fields = [0]. Are they really missing from the DataFile metadata? If so, how does a reader know where in the Lance file to find those columns?

The descriptor occupies one physical column: fields = [0] maps to column_indices = [k]. The reader decodes its children from column k using the Blob page layout, so they need no separate mapping entries.

wjones127 · 2026-09-04T23:51:30Z

+        let err = validate_operation(Some(&manifest), &reused).unwrap_err();
+        assert!(


suggestion: could we also assert which error variant this is emitting? I think it would be InvalidInput, right?

Yes, InvalidInput; the test now checks both the variant and the message.

wjones127 · 2026-09-04T23:57:02Z

+            let Some(restored_max_field_id) = manifest.max_allocated_field_id else {
+                return Err(Error::invalid_input(format!(
+                    "Cannot restore version {version}: stable field IDs were activated after that version"
+                )));
+            };


question: remind me, why can't we restore? What bad happens if we let users restore to a previous state?

A pre-activation version could bind ID 1 to an integer field while the activation version binds it to a string field. Retaining the current high-water mark prevents future ID reuse, but cannot resolve that existing conflict. Reading the old snapshot is still supported.

wjones127 · 2026-09-05T00:05:05Z

+        let mut current_id = i64::from(schema_max_id.max(max_existing_id)) + 1;
+        let unassigned_count = self.fields_pre_order().filter(|field| field.id < 0).count() as i64;
+        if unassigned_count > 0 && current_id + unassigned_count - 1 > i64::from(i32::MAX) {
+            return Err(Error::invalid_input(
+                "No further field ID can be allocated because IDs are exhausted",
+            ));
+        }


suggestion: could we keep in the i32 domain by using saturating arithmetic?

Suggested change

let mut current_id = i64::from(schema_max_id.max(max_existing_id)) + 1;

let unassigned_count = self.fields_pre_order().filter(|field| field.id < 0).count() as i64;

if unassigned_count > 0 && current_id + unassigned_count - 1 > i64::from(i32::MAX) {

return Err(Error::invalid_input(

"No further field ID can be allocated because IDs are exhausted",

));

}

let mut current_id = schema_max_id.max(max_existing_id).saturating_add(1);

let unassigned_count = self.fields_pre_order().filter(|field| field.id < 0).count() as i32;

if unassigned_count > 0 && (current_id - 1).saturating_add(unassigned_count) > i32::MAX {

return Err(Error::invalid_input(

"No further field ID can be allocated because IDs are exhausted",

));

}

The wider cursor can represent i32::MAX + 1 after assigning the last valid ID, so the next checked conversion reports exhaustion; saturating arithmetic would lose that distinction.

wjones127 · 2026-09-05T00:08:43Z

+/// Transient schema-metadata marker used by bindings for raw Arrow input.
+pub const TRANSACTION_SCHEMA_SOURCE_RAW_ARROW: &str = "lance:transaction_schema_source_raw_arrow";


question: what's the purpose of this? When is this necessary?

Arrow conversion may number [b, a] as [0, 1], even when the dataset has a = 0, b = 1. This is now resolved at the Arrow conversion boundary, so commits need no source marker or mode; Project still preserves explicit IDs for renames.

lance-gatekeeper

❌ Gate recommendation: request changes.

The allocator simplification preserves the current implementation contract, but the one remaining rollout requirement is still absent: explicit migration must require retiring every pre-v12.0.0-beta.4 writer before activation. Add that precondition to the Rust API and the schema/versioning operator docs so a one-way migration cannot be followed by an older writer that drops the high-water state.

lance-gatekeeper · 2026-09-21T15:12:08Z

+that sets only one is invalid. The reader flag for stable field IDs must remain unset because the
+feature does not change read behavior.
+
+A dataset changes to stable field IDs only through an explicit migration commit.


This migration remains unsafe while any pre-v12.0.0-beta.4 writer can still commit. Those versions predate the generic writer-feature check, so they can republish an activated manifest without max_allocated_field_id; a later add can then reuse a retired ID and violate the durable identity contract. Make writer retirement a required precondition here, in versioning.md, and in Dataset::migrate_to_stable_field_ids before the one-way commit. This succeeds the malformed predecessor and preserves the same unresolved rollout boundary.

lance-gatekeeper

❌ Gate recommendation: request changes.

The merge from main does not change the pull-request patch, so the remaining rollout blocker is unchanged: explicit activation must require retiring every writer older than v12.0.0-beta.4. Document that mandatory precondition in Dataset::migrate_to_stable_field_ids and the schema/versioning operator docs; otherwise a legacy writer can remove the high-water state after the one-way migration.

lance-gatekeeper · 2026-09-21T15:55:04Z

+that sets only one is invalid. The reader flag for stable field IDs must remain unset because the
+feature does not change read behavior.
+
+A dataset changes to stable field IDs only through an explicit migration commit.


This remains unsafe while any pre-v12.0.0-beta.4 writer can still commit. Those versions predate generic writer-feature enforcement, so they can republish an activated manifest without max_allocated_field_id; a later add can then reuse a retired ID. The current revision only merges main, so the previous finding remains unresolved. Make writer retirement a required precondition in this schema contract, versioning.md, and the Dataset::migrate_to_stable_field_ids API documentation before activation.

wjones127 · 2026-09-23T15:17:14Z

Decision: manifest validity as an action-design principle, and what it means for `DropField`

Decision

Adopt the principle in its composite form: applying a complete action set yields a valid manifest; validity is checked once at assembly, not per action. Per-action validity cannot hold (an AddFragment alone leaves a fragment with rows and no files; an AddField alone leaves a column no fragment backs). The principle needs an explicit list of manifest invariants to mean anything, and that list is the open item below.

Status

Proposed 2026-09-23 in the Transaction V2 stack discussion (#8644). Not yet written into the action module docs. The DropField representation stays as it is on #8646 (tombstone to -2 in surviving files) until the watermark question here is settled.

Options and criteria

Criteria: no dangling references a reader has to special-case; dropped field ids are never re-minted; one convention, not two.

Tombstone dropped ids to -2 in every surviving file (current V2 DropField). Satisfies the invariant "every id in a file's field list is in the schema or is a tombstone". Erases the derived Manifest::max_field_id watermark for the dropped id, so the id can be re-minted unless the watermark becomes explicit.
Prune only, matching legacy Project (the 2026-09-08 direction on this PR). Keeps the derived watermark intact because the dangling id stays in file field lists. Requires "file lists may name ids not in the schema" to remain a permitted state, which is a second convention beside -2.

The two options are not decided by the principle; they are decided by whether the watermark is derived or explicit. With an explicit watermark (this PR's direction), tombstoning is strictly better and the principle holds cleanly.

Rejected, with the reason and when it was tried

Per-action validity as the principle (2026-09-23): the vocabulary is deliberately fine-grained and intermediate states are invalid by design.

Open questions

The invariant list itself: which manifest states are valid after assembly. Candidates: every file field id is in the schema or is -2; every fragment has at least one file or zero physical rows; every index field is in the schema; no fragment id or field id below the watermark is re-minted.
Whether the action path runs the manifest validation the legacy path does. Not verified.
Sequencing: make the field-id watermark explicit here first, then keep V2 DropField as tombstoning; or land prune-only now and revisit. Recommendation is the former.

…eld (#9447) Closes #9424 ## Mechanism `fix_schema` in `rust/lance/src/io/commit.rs` remaps duplicate field ids, then applies the mapping to the schema: ```rust for (old_field_id, new_field_id) in &old_field_id_mapping { let field = manifest.schema.mut_field_by_id(*old_field_id).unwrap(); field.id = *new_field_id; } ``` Only `id` is written. `Field` carries an independent `pub parent_id` (`rust/lance-core/src/datatypes/field.rs`), and the protobuf conversion in `rust/lance-file/src/datatypes.rs` writes `parent_id: field.parent_id` verbatim rather than recomputing it from the tree. `Schema::mut_field_by_id` recurses into children, so nested fields are reachable targets of this loop. The consequence is durable. Once a struct field is renumbered, its children still name the old parent id, that value is persisted into the manifest, and the read path in `rust/lance-file/src/datatypes.rs` hard-errors on reopen: ``` Field 'item' (id=8) references parent id 2, which must appear earlier in the protobuf field list ``` The tolerant branch immediately above only covers duplicate parent ids that still exist, not a parent id that was renumbered away. The commit succeeds and the table can never be read back. `fix_schema` is called unconditionally on both live commit paths; the check at the top of the function is a fast-path short-circuit, not a guard. ## Fix Propagate the new id to the field's direct children alongside the id write. `Field::set_id` already establishes this convention (it sets `parent_id` and recurses), but it is not directly reusable here because it also assigns ids from a seed for fields with `id < 0`, which is not what this loop wants. The update is order-independent. A grandchild names its own parent's id, which that parent's own mapping entry handles, so no entry depends on another having run first. New ids start at `max_field_id() + 1`, so a new id can never collide with an old id still pending in the mapping. ## Why #8658 does not cover this PR #8658 (`feat: stabilize field IDs across schema evolution`) touches `fix_schema`, but only to make it `pub(crate)`, to early-return when stable field ids are active, and to widen the id arithmetic to i64. It does not re-parent children, so it neither fixes nor obsoletes this. It is also currently CONFLICTING with CHANGES_REQUESTED. ## On the validation alternative The issue notes that the trigger requires a writer committing duplicate field coverage across two data files in one fragment, and concedes that such a writer is itself buggy. It offers an alternative of rejecting duplicate coverage at validation instead. That alternative is reasonable and this change does not preclude it. But under either approach, a fixup that runs and then emits an unopenable manifest is indefensible: if duplicate coverage is later rejected outright, this loop becomes unreachable and harmless, and until then it should not corrupt the manifest it is meant to repair. ## Tests New `test_fix_schema_nested` covers `struct s { text, struct inner { ordinal } }` with duplicate coverage on both structs, asserting children and grandchildren follow their renumbered parents, plus a closing sweep that every non-root field names a parent id that still exists. Before, on unmodified source: ``` ---- io::commit::tests::test_fix_schema_nested stdout ---- thread 'io::commit::tests::test_fix_schema_nested' panicked at rust/lance/src/io/commit.rs:2584:9: assertion `left == right` failed: child of a renumbered struct kept a stale parent_id left: 1 right: 5 test result: FAILED. 131 passed; 1 failed; 0 ignored; 0 measured; 3733 filtered out; finished in 0.34s ``` After: ``` test io::commit::tests::test_fix_schema ... ok test io::commit::tests::test_fix_schema_nested ... ok test result: ok. 132 passed; 0 failed; 0 ignored; 0 measured; 3733 filtered out; finished in 0.31s ``` `cargo fmt --all --check` and `cargo clippy -p lance --tests -- -D warnings` are both clean on the pinned 1.97.0 toolchain. Note that `cargo clippy --all` fails on `lance-core` on current main independently of this change (three `chunks_exact` findings in `utils/bloomfilter/sbbf.rs` and `utils/stable_partition.rs`), but only under a newer clippy than the pin, so it should not affect CI.

feat: stabilize field IDs across schema evolution

08bebbe

github-actions Bot added the enhancement New feature or request label Aug 20, 2026

Xuanwo marked this pull request as ready for review August 20, 2026 09:47

github-actions Bot added A-python Python bindings A-java Java bindings + JNI A-format On-disk format: protos and format spec docs labels Aug 20, 2026

This comment was marked as outdated.

Sign in to view

lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Aug 20, 2026

fix: canonicalize stable field ids at commit boundaries

2964690

lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Aug 20, 2026

fix: enforce stable field ids in namespace rewrites

7e46c86

github-actions Bot added the A-namespace Namespace impls label Aug 20, 2026

This comment was marked as outdated.

Sign in to view

lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Aug 20, 2026

wjones127 self-requested a review August 20, 2026 16:07

fix: preserve stable identities across schema operations

7d59c81

Xuanwo added 3 commits August 21, 2026 01:50

fix: reject ambiguous stable field mappings

2b45dbe

fix: centralize binding field id remapping

8851484

fix(java): preserve legacy field id remapping

bb42780

lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Aug 20, 2026

This comment was marked as outdated.

Sign in to view

lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Aug 20, 2026

Xuanwo added 2 commits August 21, 2026 03:48

fix(java): validate project field identities

d199a9a

test(java): verify legacy project values

3b041d1

lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Aug 20, 2026

This comment was marked as outdated.

Sign in to view

lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Aug 20, 2026

wjones127 requested changes Sep 5, 2026

View reviewed changes

wjones127 mentioned this pull request Sep 20, 2026

feat(transaction): implement the remaining drafted actions #8646

Draft

shoemoney mentioned this pull request Sep 21, 2026

fix(commit): re-parent children when fix_schema renumbers a nested field #9447

Merged

fix: clarify stable field IDs and make schema input explicit

7927343

lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Sep 21, 2026

This comment was marked as outdated.

Sign in to view

lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Sep 21, 2026

Xuanwo added 2 commits September 21, 2026 21:24

refactor: resolve Arrow field IDs at conversion boundaries

eed6d73

Merge origin/main into xuanwo/stable-field-ids

32f617c

lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Sep 21, 2026

This comment was marked as outdated.

Sign in to view

lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Sep 21, 2026

fix: reconcile Arrow schema conversion after merging main

f73688a

lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Sep 21, 2026

This comment was marked as outdated.

Sign in to view

lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Sep 21, 2026

refactor: simplify stable field ID allocation and tests

f4afe81

lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Sep 21, 2026

This comment was marked as outdated.

Sign in to view

lance-gatekeeper Bot reviewed Sep 21, 2026

View reviewed changes

lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Sep 21, 2026

Merge branch 'main' into xuanwo/stable-field-ids

122f7f7

lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Sep 21, 2026

lance-gatekeeper Bot reviewed Sep 21, 2026

View reviewed changes

lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Sep 21, 2026

wjones127 mentioned this pull request Sep 22, 2026

feat(transaction): implement the Transaction V2 action vocabulary #8644

Draft

lance-gatekeeper Bot added K-changes Latest Gatekeeper recommendation requests changes. and removed K-changes Latest Gatekeeper recommendation requests changes. labels Sep 23, 2026

		let err = validate_operation(Some(&manifest), &reused).unwrap_err();
		assert!(

		/// Transient schema-metadata marker used by bindings for raw Arrow input.
		pub const TRANSACTION_SCHEMA_SOURCE_RAW_ARROW: &str = "lance:transaction_schema_source_raw_arrow";

Conversation

Xuanwo commented Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 20, 2026

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

westonpace commented Aug 20, 2026

Uh oh!

This comment was marked as outdated.

Uh oh!

codecov Bot commented Aug 20, 2026

Codecov Report

Uh oh!

This comment was marked as outdated.

Uh oh!

wjones127 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Xuanwo Sep 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

lance-gatekeeper Bot left a comment

Choose a reason for hiding this comment

Uh oh!

lance-gatekeeper Bot Sep 21, 2026

Choose a reason for hiding this comment

Uh oh!

lance-gatekeeper Bot left a comment

Choose a reason for hiding this comment

Uh oh!

lance-gatekeeper Bot Sep 21, 2026

Choose a reason for hiding this comment

Uh oh!

wjones127 commented Sep 23, 2026

Decision: manifest validity as an action-design principle, and what it means for DropField

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

Xuanwo commented Aug 20, 2026 •

edited

Loading

Xuanwo Sep 21, 2026 •

edited

Loading

Decision: manifest validity as an action-design principle, and what it means for `DropField`