Skip to content

feat: let a transaction carry a schema metadata update - #8553

Closed
wkalt wants to merge 1 commit into
lance-format:mainfrom
wkalt:ticket/ent-1961/atomic-metadata-commit
Closed

wkalt wants to merge 1 commit into
lance-format:mainfrom
wkalt:ticket/ent-1961/atomic-metadata-commit

Conversation

@wkalt

@wkalt wkalt commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

A transaction may now carry a schema metadata update alongside its operation, applied to the same manifest, and two transactions that both carry one conflict whatever their operations, so the loser is rejected before anything lands. This matches how two UpdateConfig transactions already treat schema metadata. An update on a single side is not a conflict: such an append rebases over an unrelated index build. The proto field is additive: an older reader applies such a transaction's data without its metadata.

@github-actions

Copy link
Copy Markdown
Contributor

Important

This PR touches the Lance format specification.

Substantive changes to the format specification — the .proto definitions
and the spec docs under docs/src/format/ — require a PMC vote before merge.
Minor edits such as typo fixes, wording, or formatting are excluded; use your
judgment.

If this is a meaningful format change:

  • Start a vote following the Lance community voting process.
    Format specification modifications need 3 binding +1 votes (excluding the
    proposer), held on GitHub Discussions, with a minimum voting period of 1 week.
  • Once the vote passes, link the completed vote in this PR. It should not be
    merged until the vote is linked.

@github-actions github-actions Bot added A-format On-disk format: protos and format spec docs enhancement New feature or request labels Aug 15, 2026
A transaction may now carry a schema metadata update alongside its
operation, applied to the same manifest, and two transactions that both
carry one conflict whatever their operations, so the loser is rejected
before anything lands. This matches how two UpdateConfig transactions
already treat schema metadata. An update on a single side is not a
conflict: such an append rebases over an unrelated index build. The proto
field is additive: an older reader applies such a transaction's data
without its metadata.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@wkalt
wkalt force-pushed the ticket/ent-1961/atomic-metadata-commit branch from fd2f590 to aa34612 Compare August 15, 2026 15:18
@github-actions github-actions Bot added the A-python Python bindings label Aug 15, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Gate recommendation: maintainer decision required.

This adds a schema-specific composition mechanism to the stable V1 transaction envelope while Transaction V2 is actively defining general ordered actions, including UpdateSchemaMetadata. A closely analogous top-level metadata proposal (#6255) was previously held for this choice because both wire contracts would need long-term support.

The maintainers need to choose between:

  • shipping the narrow V1 extension now, which delivers atomic data + schema metadata sooner but makes older writers ignore the modifier during conflict analysis and commits the project to a second composition model;
  • expressing this as Transaction V2 actions, which is broader and fail-closed for older writers but delays this use case until that implementation is ready.

Please choose based on whether the immediate schema-metadata use case justifies a permanent V1 contract, and record that decision through the required format vote.

@lance-gatekeeper lance-gatekeeper Bot added the K-decision Latest Gatekeeper review requires a maintainer decision. label Aug 15, 2026
@wkalt

wkalt commented Aug 15, 2026

Copy link
Copy Markdown
Contributor Author

Closing this — looking into another approach.

@wkalt wkalt closed this Aug 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-format On-disk format: protos and format spec docs A-python Python bindings enhancement New feature or request K-decision Latest Gatekeeper review requires a maintainer decision.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant