feat(transaction): implement the Transaction V2 action vocabulary by wjones127 · Pull Request #8644 · lance-format/lance

wjones127 · 2026-08-19T22:53:14Z

Stacked on #7954, which defines the Transaction V2 wire format. That PR defines the actions on the wire; this one implements them in Rust. It carries no format change of its own — every proto edit this stack needs now lands in #7954, so the format is a single vote.

Supersedes #8554, which had its head on a fork and so could not be part of a GitHub stack.

Today a transaction names one operation from a fixed list — Append, Delete, Merge, and so on — and each one is a post-image: it says what the dataset should look like afterwards. Two consequences follow. Work that spans several kinds of change needs several commits, so there is no way to add a fragment and then modify it atomically. And every new operation has to be taught how it conflicts with every existing one, which is an N-by-N table that grows quadratically.

This PR adds the delta-based alternative. A transaction can now carry a CompositeOperation: an ordered list of granular actions that commit together as one manifest change. Twelve actions are implemented — AddFragment, AddField, AddDataFile, AddBase, TombstoneFieldData, RemoveFragment, SetDeletionFile, AlterField, DropField, ConfigUpdate, ReserveFragmentIds, and ResetTable. Between them these cover the whole fragment, data-file, schema, and config surface; the drafted actions left for later are the index-segment, overlay, MemWAL, and assertion families.

Each action gets its own module holding its definition, how it is applied, which coordinates it writes, and its wire encoding, so one action can be reviewed — or added — without reading the other eleven.

Two things fall out of the delta model:

Steps can reference ids the same commit is about to create. An action that allocates a fragment, field, or base id names it with a local token; later actions in the same operation refer back to that token. Ids are minted when the actions are applied, against whichever manifest they land on, so replaying the same action set against a newer version simply produces different ids. Moving an action set to a newer version needs no rewriting at all.

Conflicts are decided by comparing write sets. Each action says which coordinates it writes — a field's data in a fragment, a fragment's deletion file, a field's definition, a base path's name, a config key. Two concurrent transactions can both commit when neither writes what the other writes. Adding an action means saying which coordinates it writes, not adding a row and a column to the conflict table. Footprints are computed at conflict time and never serialized, so a writer cannot pin down what a reader treats as a conflict.

Some writes cannot be enumerated, and those are tracked as such rather than approximated: removing a fragment writes everything inside it, replacing a config map writes keys it never names, and ResetTable writes the whole table, so it conflicts with any concurrent action set at all — including a pure append that writes no committed coordinate.

The two models interoperate. Append, Delete, UpdateBases, and DataReplacement can be translated into the actions they decompose into, which is both how an action set is checked against a concurrent named operation and the groundwork for squashing several operations into one commit. Each translation has a parity test that builds the manifest twice — once down the legacy path, once through the translation — and asserts the two agree.

Anything not yet expressible is rejected rather than approximated: an unimplemented action fails to parse, an untranslatable operation falls back to the conservative always-retry, and an action naming something that does not exist is an error.

Not included

Merge and Project are not translated. Both hand over a whole new schema instead of a description of what changed, so recovering the delta needs the read version's schema to diff against, which TryFrom<&Operation> does not have. The actions themselves are sufficient — Project is a set of DropFields and Merge a set of AddFields plus their data files — so this is a plumbing gap, not a vocabulary one.

Seven of the nineteen drafted actions are still unimplemented: AddOverlays, RefreshRowVersionMetadata, UpdateCompactedSsTables, AddIndexSegment, RemoveIndexSegment, AdjustIndexCoverage, and AssertUniqueKeys. Three of those need the index-segment model settled and three need the MemWAL and overlay subsystems; AssertUniqueKeys is not a delta at all but a precondition, so it needs a validation hook rather than an apply. Parsing any of them is an error rather than a skip, so an old reader cannot apply a partial transaction.

There are no Python or Java bindings. The Rust API is public but documented as a pre-vote draft, matching #7954's stability caveat, so it can change without a deprecation cycle.

Squashing several already-committed operations into one action set is left for later. The composite behavior it depends on is proven end-to-end here (rust/lance/tests/composite_transaction.rs), but rewiring one operation's committed references onto another's freshly minted ids is a separate problem.

wjones127 · 2026-09-20T19:51:36Z

Self-review pass (polish-pr), 2026-09-20

Decision

Two fail-open holes found by the self-review are fixed in this PR, each as a fixup at the commit that introduced it, each with a test proven to fail without the change.

The covered-index fence was dropped on the action path. Manifest::new_from_previous masks the previous manifest's flags down to STICKY_PAIRED_FLAGS, so FLAG_COVERED_INDEX_METADATA is cleared on every commit and has to be re-derived. The legacy path re-derived it in build_manifest; ApplyState::into_manifest called assemble_manifest directly and never did, so the first composite operation on a covered table republished its covering indices with the fence down. Fixed by moving the derivation into assemble_manifest, which now takes the index list the commit publishes — one place, both build paths. New test: test_a_composite_commit_keeps_the_covering_fence.

Note this is the shape feat(transaction): implement the index management actions #8645 had already arrived at independently; landing it here means feat(transaction): implement the index management actions #8645 no longer carries that hunk, and its commit message no longer claims the move.
Two conflict checkers answered Ok to a concurrent action set. Every check_*_txn was given an explicit CompositeOperation arm except two: Operation::Clone has no checker at all and returned Ok(()) from check_txn's dispatch, and check_add_bases_txn was the one checker matching with a _ => Ok(()) wildcard, which swallowed the variant. Both directions are asymmetric — the mirror direction failed closed via check_action_txn — and the PR's own assert_conflict helper asserts symmetry. Fixed by stating the rule at the Clone arm and by spelling the operations out in check_add_bases_txn like every sibling. New rstest cases case::update_bases and case::clone on test_a_legacy_operation_conflicts_with_an_action_txn; both fail without the change.

Status

Settled and implemented. cargo fmt --all clean, cargo clippy --all --tests --benches -- -D warnings clean, scoped pre-commit run --files clean. lance-table transaction tests 206 pass; lance dataset_transactions + io::commit 167 pass.

Options and criteria

For (1), the alternative was a standalone helper called from both paths. Rejected in favour of the parameter, because a helper leaves two call sites that a third build path can forget, while the parameter makes the index list part of what assembling a manifest means.

Rejected, with the reason and when it was tried

Hoisting the CompositeOperation arm above check_txn's per-operation dispatch to replace all thirteen pasted copies. Tried 2026-09-20 and rejected: the thirteen arms are required for exhaustiveness over Operation, so hoisting leaves thirteen unreachable branches — strictly worse than the duplication. The two gaps above were closed individually instead.

Open questions

Raised by the self-review, not addressed here:

Action::is_data_change and the thirteen per-action implementations have no caller outside their own unit test. Groundwork, or should it come out until something consults it?
Ref is a generic crate-public name that every external import site already aliases (Ref as ActionRef), and it forces pb::r#ref::Kind in generated Rust. Worth renaming before the vote pins it?
AddField/AddBase carry a bare local: u32 token where every other action uses Ref. Intentional (a mint can never be Committed), but unstated.
AddField { parent: Ref::Committed(p) } records nothing in its footprint, so a concurrent DropField(p) is declared disjoint and the loser fails inside apply with a non-retryable invalid_input instead of conflicting. Recording the committed parent as a required_field is cheap.
The reserved primary/clustering-key logic in action/config_update.rs is a near-verbatim second copy of the legacy UpdateConfig arm, and it already diverges: the action copy omits verify_primary_key()?, and it open-codes truthiness instead of using str_is_truthy.
Footprint::from(ours) is rebuilt on every check_action_txn call, i.e. retries x concurrent transactions, on the path that only runs under contention.

🤖 Generated with Claude Code

wjones127 · 2026-09-20T20:08:41Z

Legacy conflict characterization run against this PR (burn-down item 14)

Decision

#9220's characterization suite passes unchanged on this PR: 42 passed, 0 failed, 6 ignored, ~2s.

Status

Verified 2026-09-20. Nothing in the stack changes legacy conflict behaviour.

Options and criteria

The suite's branch (will/legacy-conflict-characterization, base 8cdffd30e) sits 56 commits ahead of this stack's base (93ba1d20c), so the suite does not compile on the stack as-is — FileFragment::write_overlay, which scenarios.rs uses, does not exist at the older base. That is base skew, not a behavioural difference.

Method: for each PR, a throwaway branch off the PR head, merge 8cdffd30e (clean, no conflicts), cherry-pick the suite's five commits, cargo test -p lance --lib conflict_matrix. Throwaway branches deleted afterwards; nothing here is pushed.

Results, all four layers:

PR	Result
#7954	42 passed, 0 failed, 6 ignored
#8644	42 passed, 0 failed, 6 ignored
#8645	42 passed, 0 failed, 6 ignored
#8646	42 passed, 0 failed, 6 ignored

#8644 is the layer that matters most here — it is the one that touches conflict_resolver.rs, adding the fail-closed CompositeOperation arm to every check_*_txn and spelling out check_add_bases_txn's wildcard. The suite covers exactly that file and is unaffected.

Open questions

The suite's 6 ignored cases are ignored on its own branch too; this run does not change them.
The stack will need a rebase onto current upstream/main before merge regardless; the clean merge above is evidence that rebase is not expected to be eventful.

🤖 Generated with Claude Code

wjones127 · 2026-09-21T18:27:56Z

+        assert!(
+            error.to_string().contains("already exists"),
+            "unexpected error: {error}"
+        );


question(blocking): why do we get an "already exists" error here? Each base has a unique local id, bucket path, and name.

Split into two checks with two messages in 09e8b97, each naming the conflicting base's id — a caller that trips one needs to know which, and a name clash and a location clash are fixed differently.

While in there I noticed the name comparison is on Option<String>, so two bases that both leave the name unset collide. That is inherited from Operation::UpdateBases, which compares the Options directly, so I kept the behaviour and pinned it with test_two_unnamed_bases_collide_on_the_empty_name rather than quietly diverging here. Happy to exempt unnamed bases instead, but that would be a change to both paths.

You're right, and my earlier reply was wrong on the precedent. Operation::UpdateBases guards its name comparison with && new_base.name.is_some() (conflict_resolver.rs:1883), so unnamed bases never collide there, and the format spec calls name an optional alias without stating any uniqueness rule. I introduced the restriction and then cited a precedent that says the opposite.

Fixed in 22deb00. The same bug was in the footprint — Coordinate::BaseName(None) made two concurrent unnamed adds conflict — so both paths now compare only names that are set, and the coordinate narrowed to BaseName(String) since an absent name coordinates nothing. Unnamed bases at the same location still collide, which is the rule the format does state.

Tests swapped accordingly: test_unnamed_bases_do_not_collide_with_each_other, plus footprint cases for unnamed bases at different and at identical locations. The different-locations case fails without the guard and passes with it.

Introduces the Rust side of the action-based transaction draft: `Ref`, `UserOperation`, `UserAction`, and the eight `Action` variants this build implements (AddFragment, AddDataFile, AddField, AddBase, TombstoneFieldData, RemoveFragment, SetDeletionFile, AlterField). Types only -- no wire conversion, apply, or conflict handling yet, so nothing reaches these from the commit path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Adds `Operation::UserOperation`, the Transaction V2 arm of the transaction oneof, and its protobuf conversions in both directions. Loading a V2 transaction now yields an action set instead of an outright rejection; an action the build does not implement is still rejected rather than skipped, so a concurrent V2 commit can never be silently treated as a no-op. The rest of the transaction machinery gains the variant but no behavior: build_manifest returns NotSupported, and conflict checks route through a single `check_action_txn` that conservatively demands a retry. Apply and conflict rules follow in later commits. Also extracts `From<&DeletionFile> for pb::DeletionFile`, previously inlined in the fragment conversion and now needed by SetDeletionFile too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Pulls the two operation-independent stages out of `build_manifest`: `normalize_fragments` (order, drop fully-tombstoned files, check overlay order) and `assemble_manifest` (construct the manifest and apply the tag, feature flags, timestamp, and fragment id watermark). The overwrite-only storage format override becomes an explicit parameter rather than a match on the operation inside the assembly step. No behavior change; the action-based apply path needs the same two stages. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Adds the action apply path: `build_manifest_from_actions` walks the action set in order against a working copy of the read-version state, and the four minting actions (AddFragment, AddDataFile, AddField, AddBase) allocate ids from the target's counters as they are reached. Each mint records its id against the action's local token, so a later action in the same operation resolves that token to the id this apply chose. The same action set replayed against a different version therefore lands on different ids without any action changing -- the property branch merge needs. An action set requires an existing dataset: it is a delta, so there is nothing for it to be a delta against at creation time. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Applies TombstoneFieldData, RemoveFragment, SetDeletionFile, and AlterField, completing the eight-action apply path. Tombstoning a field's data and retyping a field both leave any index over that field describing values the fragment no longer holds, so the affected fragments are dropped from the index bitmap rather than the index being discarded outright. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

A footprint is the set of coordinates an action set writes. Two concurrent sets can both commit when neither writes what the other writes -- one structural rule instead of a matrix over operation pairs, so adding an action means saying which coordinates it writes rather than extending an N-by-N table. Only committed coordinates appear. A minted fragment, field, or base has no id in the read version, so two writers minting at the same time never collide. Footprints are derived from the actions at conflict time and never serialized, so a writer cannot pin down what a reader treats as a conflict and the rule can be tightened without a format change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Two concurrent action sets can both commit when neither writes what the other writes -- one structural rule over coordinates instead of a matrix over operation pairs. That replaces the conservative always-retry for the same-form case. A named operation on the other side still conflicts unconditionally. A footprint is defined over actions, and a named operation carries a post-image rather than a delta, so comparing the two would mean first specifying all seventeen operations in terms of coordinates. That is not a specification this draft has, so the mixed pair fails closed. Rebasing an action set onto a newer version is a no-op: its minted ids are allocated when the actions are applied, against whichever manifest they land on, and its committed references name coordinates that do not move. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Replaying the same actions against the manifest a previous run produced re-resolves their local tokens against the newer counters, so the second run mints different fragment and field ids without any action changing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Each test is one commit doing work that would otherwise have taken several: a fragment added and then modified, a field added and then filled, both inside a single version. Also covers row id assignment for minted fragments and both sides of the footprint conflict rule through the real commit path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Removes a field from the schema, taking its descendants with it. The slots that backed the dropped fields in each data file are tombstoned rather than removed, so a file's remaining columns stay at the positions they were written at; a file left backing nothing live is pruned during normalization. For conflicts, a drop writes every coordinate belonging to the field -- its definition and its data in every fragment -- so it collides with a concurrent alter or data rewrite of the same field. A drop also has to collide with a concurrent write of *new* data for the field, which is not a coordinate: the fragment carrying it is minted, so no concurrent writer can name a cell inside it. The footprint records the fields such a write depends on separately, and a drop of one of them is a conflict. Without that, an append and a drop of a field the append is populating both commit, and the new fragment is left carrying data for a field the manifest no longer has. Field ids come from a monotonic counter, so a dropped id is never reused and a stale data file naming it cannot be mistaken for a later field. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The action code was split by phase -- apply, footprint, proto -- so understanding one action meant reading four files and adding one meant touching four match statements. Each action now owns a module holding its definition, `apply`, `footprint`, wire conversions, and tests. `Action` dispatches to them. The phase modules keep what is genuinely shared: `apply` holds the `ApplyState` the actions program against, `footprint` the coordinate space and the conflict comparison, `proto` the envelope and dispatch. No behavior change. The one difference is that `AddField` and `AddBase` now check for a duplicate local token before bumping the id counter, matching `AddFragment`; a duplicate token aborts the whole apply either way. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…ifest `build_manifest_from_actions` ran the actions and then assembled the manifest in one 80-line body. Move the assembly to `ApplyState::into_manifest`, so the entry point reads as the three steps it is: make the state, run the actions, turn it into a manifest. `ApplyState` now borrows the manifest it was built from rather than copying everything it needs out of it, which is also what lets `into_manifest` reach the read version without being handed it back. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`AlterField`, `DropField`, and `TombstoneFieldData` took raw committed field ids, so none of them could name a field minted earlier in the same operation. Squashing needs exactly that: collapsing "add column c" and "rename c" into one operation leaves an `AddField` whose field has no committed id, and a squash rewrites already-committed transactions, so it cannot re-plan the data to avoid the reference. All three now take a `Ref`, matching the fragment actions. Footprints follow the rule already used there: a local reference records no coordinate, since a field this operation mints is invisible to a concurrent writer. `AlterField` loses its `Default` impl -- `Ref` has no meaningful zero, and a default field reference would silently mean field 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Rebasing onto the merged transaction module split brings three API changes the action code predates: `RowIdMeta::Inline` wraps `InlineRowIds` rather than a byte vector, `DataFile::new`/`new_unstarted` take a `ConcreteFileVersion` instead of a major/minor pair, and `Operation::Project` carries `preserves_nullability`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Follows the wire change: `AddFragment` carries a `Ref` rather than a bare local token, so a fragment can be added at an id an earlier commit set aside with `ReserveFragmentIds`. That is what a writer needs when it bakes row addresses into an index it writes in the same commit as the data -- the addresses have to be known before the commit that would otherwise mint them. Apply checks the claim instead of making one: the id must be below the counter (so it was reserved rather than guessed) and must not already be in use. It does not move the counter, because the reservation already did. A claimed id is also a write. A reservation is meant to be one writer's alone, but nothing in the format enforces that, so `AddFragment` records `FragmentExistence` for a committed id -- two operations handed the same range now conflict rather than silently overwriting each other. A local token still records nothing, so ordinary appends still commute. `minted_fragments` becomes `new_fragments`, since a fragment at a reserved id is added by the operation without being minted by it.

The row id counterpart of `ReserveFragmentIds`. Reserving a fragment id fixes the high half of a row address, which is all a dataset without stable row ids needs; with them, an index records row ids instead, off a counter nothing could reserve from. A writer that has to know both before it commits needed both. The reservation is taken after the operation's own fragments are numbered, so the reserved range is always the `count` ids ending at the committed `next_row_id` -- whichever position the action holds in the set. Also rejects fresh row ids a writer supplied without reserving them. `assign_row_ids` honors a sequence a fragment arrives with and leaves the counter where it found it, which is right for a compaction, whose rows keep ids already below the watermark. Applied to unreserved ids it silently sets up a collision with the next append, so apply now checks that a new fragment's supplied ids sit below the watermark.

`build_manifest_with_read_version` short-circuited to `build_manifest_from_actions` before `read_version_state` was used, and that function took no such parameter, so `apply_mem_wal_index_coverage` never ran for an action-based commit. The helper is the invalidation half as well as the crediting half: it drops an index's catch-up entry when the new segments cannot be shown to cover the read version. Skipping it carried the MemWAL index metadata through untouched, so `details.index_catchup` kept asserting an index was caught up to a compaction generation it no longer covered -- and the WAL pod retires SSTables against that. It affected every action-path commit, not only the schema-evolution ones. Thread `read_version_state` through and derive coverage at the same point the legacy path does: snapshot the logical segments before the actions run, then apply the rules after the index list has been pruned and before the manifest is assembled. `None` stays legitimate for dataset creation and detached commits. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Transaction V2 is a pre-vote draft with no compatibility contract, and a library that predates it rejects the commit outright -- so a dataset that has one in its history cannot be read by an older reader. Turning that on is the caller's decision, not the library version's. `CommitBuilder::with_experimental_composite_operations` is that decision. Without it, `execute` rejects a `CompositeOperation` with `NotSupported` naming the method. It lives on the builder rather than in `WriteParams` because it gates the commit, not how data is written; `execute_batch` accepts only appends and routes through `execute`, so it is covered. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`mod composite` exercised the actions that reference each other's minted ids and nothing else, so `AddBase`, `RemoveFragment`, `SetDeletionFile`, `AlterField`, `ResetTable` and `ConfigUpdate` reached a real commit only through `build_manifest` in unit tests. One commit each closes that. `SetDeletionFile` also gets a concurrency test. It is the one action carrying an absolute post-image -- the whole deletion file, computed at the read version and written in blindly -- so it is correct only because two sets touching the same fragment's deletions collide on `Coordinate::FragmentDeletions`. Without the conflict the loser's deletions would be dropped rather than recomputed, and nothing pinned that until now. Finally, version independence: the same action set, committed after losing 0, 1 or 3 races, lands the same change every time while its ids come out different. That is the property `Ref::Local` exists for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The public types carried prose but nothing a reader could copy. Add a compiling example on `CompositeOperation` showing the thing the vocabulary exists for -- a second step naming the fragment the first one minted, which as named operations would be two commits with an empty fragment visible in between -- and one on `Ref` distinguishing a committed id from a token. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Follows the proto layer: "draft" is not a status the project recognizes, and the governance vote on experimental specification features passed on 2026-08-27. Restates the two commitments the label carries -- breaking changes without a separate vote, and removal if the stabilization vote does not pass -- in the module's Stability section and on the CommitBuilder opt-in that a caller reads before using it. The "drafted vocabulary" wording, which distinguished the specified action list from the implemented subset, becomes "specified"; that distinction is unrelated to the feature's status. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Carries the warning from the proto layer onto the two Rust surfaces a caller actually meets: the action module's Stability section and the CommitBuilder opt-in that gates the commit, which now has its own Compatibility section. The NotSupported message a caller hits without opting in names the consequence and the issue too, since that error is the first and maybe only place they read about it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

AddBase enforced two uniqueness rules with one check and one message, so a caller could not tell which it tripped -- and the two are fixed differently. Split them, name the conflicting base's id in each, and pin that two unnamed bases count as the same name, which AddBase inherits from Operation::UpdateBases comparing the Options directly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The field is not an Arrow DataType: it is LogicalType, Lance's own string encoding of one, and what Field::logical_type holds. It was typed as a bare String, which left that ambiguous at the API boundary. Name the type, and convert at the wire edges where the string form is what the schema persists. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

build_manifest_from_actions took Option<&Manifest> only to reject None on the first line. Move the rejection to the one caller that actually holds the Option, so the signature states the precondition instead of re-checking it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

"Cannot enable stable row ids on existing dataset" stopped being true with #8521, which added migrate_to_stable_row_ids. Use the legacy path's wording, which names it, and record why the action path does not take the migration bypass that guard carries. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

AddBase compared base names as Options, so a second base that left the name unset was rejected as a duplicate of the first, and two concurrent unnamed adds conflicted in the footprint. Nothing asks for that: the format calls the name an optional alias and puts no uniqueness rule on its absence, and Operation::UpdateBases guards its own name comparison with is_some() for the same reason. Compare only names that are set, and narrow Coordinate::BaseName to String now that an absent name coordinates nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

… operation Follows the proto change that removed CompositeOperation.uuid and read_version. The to-wire path existed only to copy those two fields off the enclosing Transaction, and the read path never looked at them, so the conversion is now a plain one. The round-trip test asserted the copies matched. It now asserts the identity on the envelope and the step descriptions on the operation, which is what the two levels actually carry. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The header still said V2 had "no write path", which was true of the wire-format PR but stops being true here, and contradicted the very next paragraph explaining that writing V2 is opt-in. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The caller opt-in said the calling code wanted V2. It could not say whether the deployment permitted it, because a library caller sets it itself. Add LANCE_ENABLE_UNSTABLE_TRANSACTION_V2 as a second, separate gate, following the LANCE_ENABLE_UNSTABLE_DATA_OVERLAY_FILES pattern: on in debug builds so tests need no setup, required in release. Committing V2 makes a dataset version unopenable by Lance before v12.0.0, so while the feature is experimental it should take a deliberate deployment decision, not just a library call. The two gates report separately, naming the one that is missing, and the check takes both answers as parameters so each is testable without toggling the build profile or the environment -- the same reason feature_flags::supported_flags_when is split out. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

wjones127 · 2026-09-22T00:10:11Z

Decision

While Transaction V2 is experimental, writing it requires two independent gates: the existing per-caller CommitBuilder::with_experimental_composite_operations, and a new LANCE_ENABLE_UNSTABLE_TRANSACTION_V2 environment variable (on in debug builds, required in release). No manifest feature flag is added yet; a reader flag is deferred to stabilization.

Status

Done in 2f8d830. The two gates report separately, naming the one that is missing, and the check takes both answers as parameters so each is testable without toggling the build profile or the environment — the same split as feature_flags::supported_flags_when.

Options and criteria

The problem: a table version committed as a CompositeOperation cannot be opened by Lance before v12.0.0 (#9454), and the fix is not being backported. The criterion was what a pre-v12 client actually experiences, checked against the released v11.0.0 tree rather than against intuition.

Environment variable gating writes. Chosen. Follows LANCE_ENABLE_UNSTABLE_DATA_OVERLAY_FILES and the in-flight feat(format): define hidden row lineage columns #9253. It does not make the feature compatible; it makes it inert unless a deployment opts in, which satisfies voting.md prerequisite 2 for everyone who has not, and makes adopting the hazard a deliberate operational decision rather than a library call.
Reader feature flag at bit >= 1<<8. Deferred to stabilization, not rejected. v9/v10/v11 do check reader flags on open — can_read_dataset at v11.0.0:rust/lance/src/dataset.rs:767, ahead of the inline-transaction decode at :804 — and their ceiling is FLAG_UNKNOWN = 256, so any bit at or above 1<<8 makes them refuse with "This dataset cannot be read by this version of Lance. Please upgrade Lance", instead of the current opaque decode failure. FLAG_MIXED_DATA_FILE_VERSIONS already uses that boundary as a deliberate fence. This is worth doing, but it only matters once V2 can be written without an explicit opt-in, so it belongs with the stabilization PR.
Suppress the inline transaction section for V2 commits. Rejected for now; see below.

Rejected, with the reason and when it was tried

Suppressing the inline transaction section (2026-09-21). Technically sound and verified: the pre-v12 open failure comes entirely from the inline section, v11 guards it with if let Some(transaction_offset) = manifest.transaction_section, and both functional consumers fall back to _transactions/. A v11 client would open and scan a V2 table normally. Rejected because it is the wrong trade while the feature is gated off: it only fixes pre-v12 reads, leaving those clients unable to read history or commit concurrently (Transaction::try_from still errors on an unknown operation), and it is largely mutually exclusive with option 2 — a reader flag makes the dataset refuse to open anyway. It also costs an extra object-store GET per read_transaction(). Revisit only if we decide read-only access for old clients beats a clean refusal.
Adding a reader flag now (2026-09-21). Nothing to protect: with writes gated off, no release build can produce a manifest that would set it.

Open questions

If suppressing the inline section is ever revisited, the gate belongs at the three inline_transaction decision sites in rust/lance/src/io/commit.rs (:361, :1123, :1481), not in lance-table's writer. lance-namespace-impls/src/dir/manifest.rs:1836 commits inline-only without setting FLAG_DISABLE_TRANSACTION_FILE, so a blanket suppression in the writer would silently drop the transaction there if it ever grew a V2 operation.
Reads are not gated by the environment variable, only writes. Reading remains fail-closed on an unrecognized action. Worth revisiting if a V2 table can reach a build that should not interpret it.
The reader flag at stabilization needs a bit chosen against whatever FLAG_UNKNOWN is at that point, and the choice only works if it is at or above the ceiling of the oldest release we intend to fence out.

…writes A footprint said only what an action set writes, and there are two ways to invalidate a field: drop it, or cast it. Only the first was guarded. `DropField` goes through `remove_field`, which feeds `removed_fields`, and that is matched against the `required_fields` an append records for the committed fields its new fragment carries data for. `AlterField` writes the plain `FieldDefinition` coordinate and nothing required it, so a set writing data for a field committed cleanly on top of a concurrent cast of that same field -- leaving the manifest describing those values as a type they were not encoded in. It failed the same way for a rewrite of a committed fragment, which writes `FieldData` while the cast writes `FieldDefinition`: different coordinates, no overlap, no conflict. `Footprint` gains a second set. `requires` holds coordinates a set read and needs to still hold what it read, and `add_field_data` fills it with the definitions of the fields it writes data for. Unlike a write, two sets may require the same coordinate -- two readers of one column do not collide. The check is asymmetric, and only for requirements: `conflicts_with` now takes the set that already committed. What that set required held when it committed and it serializes first, so nothing arriving later can break it retroactively. That is not a technicality here -- it is the difference between the two orders. A cast arriving after an append is safe, because applying it calls `rebind_field_everywhere`, which rebinds the field in every fragment the manifest has by then, including the one the append just added. An append arriving after a cast is not, because its data was already encoded against the old type. Only the second is rejected. Requiring the whole definition also rejects a concurrent rename or nullability change that would not have invalidated the data. That matches the granularity of the coordinate, since `AlterField` writes one `FieldDefinition` whichever facet it sets; splitting the coordinate by facet is a separate question. `test_conflicts` asserts both directions and cannot express this, so the directional pairs get their own `test_directional_conflicts`, which states each order rather than assuming they match. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

An empty footprint deserves a reason. A reserved fragment id becomes a coordinate the moment a fragment claims it, so two writers spending one reservation are caught by AddFragment; a reserved row id never does, so that misuse is not caught at all and the reservation has to be spent once, by the writer that made it. Also spell out that the range ends at the committed manifest's watermark, which is not where the read version left off if something landed in between. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Four pairs of concurrent action sets were passing the footprint check and producing a wrong table, or failing at apply where a conflict was the right answer. All four are field-scoped, and they resolve into one rule: anything that depends on a field requires its definition, and the two actions that redefine a field -- AlterField and DropField -- write it. Add-column vs compaction. Data written for a minted field into a committed fragment recorded nothing, because the field id is invisible to a concurrent writer. But the cells die with the fragment: a compaction landing second removes the fragment, and the replacement carries no file for the new field, so those rows read back as null. The fragment is now required in both orders (`required_fragments`, symmetric), which is what legacy Merge-vs-Rewrite also rejects. Field names. AddField wrote nothing and a rename wrote only the field's definition, so two writers adding a column `c` both landed and nothing at apply time noticed. Names are now a table-wide coordinate, and apply rejects a duplicate among siblings for the committed case the footprint cannot see. Table-wide rather than per-parent because a footprint has no schema to find a field's siblings with; the cost is a spurious conflict between two writers adding `c` under different structs. Child under a dropped parent, metadata on a dropped field. Both failed at apply with "does not exist". Both now require the parent's or field's definition, so a drop landing first is a conflict, and a drop landing second takes the child or the metadata with the field, as intended. Unenforced keys. The primary and clustering keys are declared through field metadata but held once per table, so two writers declaring one on different fields wrote disjoint ConfigEntry coordinates and the second failed in `reject_reserved_key_change`. They now claim a table-wide `UnenforcedKey` coordinate as well. The rule also answers the question left open on `required_fields` and `removed_fields`: DropField::apply tombstones the field out of every fragment the manifest holds at apply time, including one a concurrent append or rewrite just added, so the drop-second order is safe and the symmetric channel was over-strict. Both fields and `Coordinate::field` are gone; the drop-first order is caught by `requires` against the definition DropField writes. The previously symmetric drop-vs-append and drop-vs-rewrite cases move to `test_directional_conflicts` with both orders stated. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

wjones127 · 2026-09-22T23:46:40Z

Decision: closing the V2-vs-V2 footprint gaps

Decision

The 14 items from the V2×V2 conflict-rule audit (2026-09-22) are addressed across the stack, with one rule tying the field-scoped fixes together and one new write mode for overlays:

Anything that depends on a field requires its definition; AlterField and DropField write it. This makes the drop-then-write order a conflict via requires ∩ writes, and leaves the write-then-drop order allowed because DropField::apply tombstones the field out of every fragment present at apply time, including one a concurrent set just added. required_fields, removed_fields, and Coordinate::field() are removed.
A partial write commutes with a partial write, not with a full write. Footprint::partial_writes holds overlays. Two overlays over the same cells both land (newer wins); an overlay and a column rewrite of the same FieldData conflict in either order.
New coordinates: FieldName(String) (table-wide), UnenforcedKey(Primary | Clustering).
requires is also checked against the committed set's removed fragments and replaced maps, through the same removes predicate the write check uses.
Data written for a minted field into a committed fragment requires that fragment (required_fragments, symmetric).
An index segment requires the definitions of its keyed and covering fields; AdjustIndexCoverage requires the fragments it widens onto.
Key assertions are violated by any write into a key column, whole or partial; assertions are matched per key-column set rather than pairwise.
Apply rejects a duplicate sibling name for AddField and AlterField { name }.

Status

Implemented on #8644 (2e436690a, 9998adf24), #8645 (77c2d8885), #8646 (df53bcf62). Verified with lance-table unit tests, the 30 lance-crate end-to-end composite tests, and clippy on both crates; a skeptical read-only review of the three diffs verified the apply-side claims (DropField tombstones the field in every fragment present at apply time; a cast landing second prunes a committed segment's coverage). Nothing here changes the wire format.

One contract was tightened as a result of that review: an AssertUniqueKeys speaks for every row its set inserts, over its key columns. Per-key-column matching is only sound under that reading, so the struct doc now states it and rules out two merge-inserts over different keys each asserting only their own rows.

Options and criteria

Criteria: no silent data loss under either commit order; a hard apply error is acceptable only where the footprint has no way to see the collision; over-strictness is acceptable when it costs a spurious conflict on a rare pair and buys a one-rule model.

Field names as a coordinate, table-wide vs. per-parent. Chose table-wide: the footprint has no schema to place a field among its siblings. Cost is a spurious conflict between two writers adding the same name under different structs. Apply-time sibling check covers the already-committed case.
Partial writes as a Footprint field vs. a Coordinate flag vs. making overlays plain writes. Chose a separate set with a two-line rule in conflicts_with: it keeps Coordinate a plain identity and preserves concurrent overlays, which a plain write would forbid.
Keep required_fields/removed_fields vs. collapse into requires/writes. Collapsed, once DropField::apply was checked to handle a concurrently added fragment.

Rejected, with the reason and when it was tried

Making AddOverlays a full FieldData write (2026-09-22): would reject two concurrent overlays on the same column, which was a deliberate allowance.
Adding a field list to AdjustIndexCoverage so it can require field data (2026-09-22): needs a proto change on feat(transaction): action-based transaction wire format (Transaction V2) #7954 and a full restack for a residual the doc now states; the safe shape is to widen coverage in the same operation that mints the fragments.
A symmetric requires check (2026-09-22): would reject a cast or drop landing after an append or rewrite, both of which apply correctly.

Open questions

AdjustIndexCoverage over committed fragments still cannot see a concurrent rewrite of an indexed column in those fragments. Caller responsibility for now.
RefreshRowVersionMetadata is fragment-wide, so two column updates on different fields of one fragment collide. Over-strict; acceptable until the sequence records per-column versions.
protos/transaction/actions.proto still says AlterField conflicts are keyed on (field id, facet); the footprint keys on the whole FieldDefinition. Doc fix on feat(transaction): action-based transaction wire format (Transaction V2) #7954 deferred to avoid a restack for prose.
A segment with real fields but covered_fragments: None that lands before a cast is never pruned by the cast (prune_rebound_fields_from_indices skips a missing bitmap). Only system indices carry unstated coverage today and they have no fields, so unreachable through real writers; the action accepts the combination.
DropField::apply rewrites surviving data files' field lists to tombstones, which erases the derived max_field_id watermark for the dropped id. Pre-existing on the stack, not introduced here, and reportedly at odds with the 2026-09-08 decision on feat: stabilize field IDs across schema evolution #8658 to have DropField prune only. Needs a check of which decision stands.

…et against requires-only sets Move the per-module `footprint(actions)` fixture into `test_support` so every conflict test builds its two sides the same way, add the removing-the-same-fragment-twice pair, and check that a reset preempts a set that only requires coordinates or records nothing at all. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Replace the five parallel coordinate sets in `Footprint` with a single `claims: HashMap<Coordinate, Mode>` and a `regions` map for the parts of the manifest a set writes wholesale. Every coordinate pair goes through `pair_conflicts`, a table with one row and column per `Mode`, so a new way of touching a coordinate is a new variant and a new row, not a new field and a new `is_disjoint` clause. The one asymmetry -- a requirement is only checked against what committed after the set read -- lives in that table instead of in the order of the `conflicts_with` clauses. `exclusive` and `required_fragments` stay outside the table: the first conflicts with a set that records nothing, and the second is symmetric. Recorders are named for the mode they record: `write`, `write_field_data`, `write_field_definition`, `write_map_update` alongside `require`. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

wjones127 · 2026-09-23T16:17:24Z

Footprint comparison refactor (folded into this stack, 2026-09-23)

Decision
Footprint now compares through one table. Every coordinate a set touches is a claims: HashMap<Coordinate, Mode> entry, Mode ordered Requires < WritesPart < Writes, and every pair goes through pair_conflicts(committing, committed). Fragments removed and maps replaced are regions, matched by containment and by equality. The asymmetry (a requirement is only checked against what committed after the set read) lives in the table, not in clause order.

Recorders are named for the mode they record: write, write_part, require, plus write_field_data, write_field_definition, write_map_update (renamed from add_*).

Status
Landed across the stack, no behaviour change (pair tests unchanged):

this PR: two-mode table (Requires, Writes), regions, renames, shared footprint test fixture
feat(transaction): implement the index management actions #8645: index claims kept outside the table, require_field_data on the refactored shape
feat(transaction): implement the remaining drafted actions #8646: adds WritesPart as a third row and column, write_part for overlays, key assertions read claims

Options and criteria
Criterion was that adding a way of touching a coordinate should be a variant and a row, not a new HashSet field and a new is_disjoint clause. Two things stay outside the table on purpose: exclusive (collides with a set that records nothing) and required_fragments (symmetric, and must not collide with writes to other coordinates in the same fragment).

Rejected

Folding required_fragments into the table as a Requires on the fragment region: wrong direction (it must be symmetric) and would not reject a removal landing second. Tried and falsified by a skeptical pass before the refactor.
Modelling exclusive as a region covering everything: a set with no claims has nothing for the region to contain.

Open questions
None new; the DropField-vs-#8658 tombstoning question is tracked on #8658.

github-actions Bot added the enhancement New feature or request label Aug 19, 2026

This was referenced Aug 19, 2026

feat(transaction): implement the Transaction V2 action vocabulary #8554

Closed

feat: stabilize field IDs across schema evolution #8658

Open

wjones127 force-pushed the will/transaction-v2-actions branch from 247689d to fd6ecc6 Compare September 3, 2026 00:12

Xuanwo mentioned this pull request Sep 7, 2026

Support safe concurrent publication of derived columns in the same fragment #9043

Open

wjones127 force-pushed the will/transaction-v2-actions branch from fd6ecc6 to ac9c4dd Compare September 8, 2026 19:51

wjones127 mentioned this pull request Sep 20, 2026

feat(transaction): implement the remaining drafted actions #8646

Draft

wjones127 force-pushed the will/transaction-v2-actions branch from ac9c4dd to 2076ce5 Compare September 20, 2026 19:49

github-actions Bot added the A-java Java bindings + JNI label Sep 20, 2026

This was referenced Sep 20, 2026

feat(transaction): implement the index management actions #8645

Draft

feat(transaction): action-based transaction wire format (Transaction V2) #7954

Draft

wjones127 force-pushed the will/transaction-v2-actions branch 4 times, most recently from 0d8addd to f1b36fe Compare September 21, 2026 20:11

wjones127 commented Sep 21, 2026

View reviewed changes

wjones127 and others added 11 commits September 21, 2026 15:26

wjones127 and others added 18 commits September 21, 2026 15:26

wjones127 force-pushed the will/transaction-v2-actions branch from 22deb00 to 83c943b Compare September 21, 2026 22:33

github-actions Bot added A-format On-disk format: protos and format spec docs format-change A change to the format spec, which requires a vote. Remove if minor (e.g. fixing typo). labels Sep 22, 2026

wjones127 and others added 3 commits September 22, 2026 15:29

wjones127 and others added 2 commits September 23, 2026 08:57

Navigation Menu

feat(transaction): implement the Transaction V2 action vocabulary - #8644

feat(transaction): implement the Transaction V2 action vocabulary#8644
wjones127 wants to merge 43 commits into
will/oss-1530-draft-action-based-transactionfrom
will/transaction-v2-actions

wjones127 commented Aug 19, 2026

Uh oh!

wjones127 commented Sep 20, 2026

Uh oh!

wjones127 commented Sep 20, 2026

Uh oh!

wjones127 Sep 21, 2026

Uh oh!

wjones127 Sep 21, 2026

Uh oh!

wjones127 Sep 21, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wjones127 commented Sep 22, 2026

Uh oh!

wjones127 commented Sep 22, 2026

Uh oh!

wjones127 commented Sep 23, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

wjones127 commented Aug 19, 2026

Not included

Uh oh!

wjones127 commented Sep 20, 2026

Self-review pass (polish-pr), 2026-09-20

Decision

Status

Options and criteria

Rejected, with the reason and when it was tried

Open questions

Uh oh!

wjones127 commented Sep 20, 2026

Legacy conflict characterization run against this PR (burn-down item 14)

Decision

Status

Options and criteria

Open questions

Uh oh!

wjones127 Sep 21, 2026

Choose a reason for hiding this comment

Uh oh!

wjones127 Sep 21, 2026

Choose a reason for hiding this comment

Uh oh!

wjones127 Sep 21, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wjones127 commented Sep 22, 2026

Decision

Status

Options and criteria

Rejected, with the reason and when it was tried

Open questions

Uh oh!

wjones127 commented Sep 22, 2026

Decision: closing the V2-vs-V2 footprint gaps

Uh oh!

wjones127 commented Sep 23, 2026

Footprint comparison refactor (folded into this stack, 2026-09-23)

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant