feat(transaction): implement the remaining drafted actions

[wjones127]

Stacked on the index-actions PR. Implements the four actions the Transaction V2 draft still listed as unwritten — AddOverlays, RefreshRowVersionMetadata, UpdateCompactedSsTables, and AssertUniqueKeys — plus the lowering of the three legacy operations that map onto them: DataOverlay, UpdateMemWalState, and the vertical form of Update. With this the implemented vocabulary covers the whole draft. No format change: the wire shapes land in #7954.

Supersedes #8632, whose head was on a fork and so could not be part of a GitHub stack.

Three of the four actions are ordinary deltas. The fourth, AssertUniqueKeys, is the first thing in the vocabulary that is a precondition rather than a change: it carries the filter of key hashes a merge insert is about to insert, so that two concurrent inserts can be shown not to collide. Nothing in the manifest records which keys exist — the key columns are an unenforced primary key — so the filter has to travel with the operation, because it cannot be recovered from any post-image.

Example

Appending rows and asserting their keys are new, in one commit. If a concurrent commit inserted rows without saying which keys they carry, this one is rejected, because there is nothing to compare against:

CompositeOperation::new(vec![UserAction::new("insert new customers", vec![
    Action::AddFragment(AddFragment { local: 0, physical_rows: 500, .. }),
    Action::AddDataFile(AddDataFile { fragment: Ref::Local(0), file, .. }),
    Action::AssertUniqueKeys(AssertUniqueKeys {
        key_fields: vec![Ref::Committed(0)],
        filter: inserted_key_hashes,
    }),
])])

Conflict rules

Two of the new actions needed something the footprint could not previously express.

An overlay writes no coordinate at all. Overlays are appended, never replaced, and when two cover the same cell the newer committed_version wins — so two concurrent overlays must not collide with each other. What an overlay does need is for its fragment to still be there, which is a dependency rather than a write, and is now tracked as one.

A key assertion is not a coordinate either. The keys are user-supplied, so two writers can insert the very same key while writing it into fragments of their own — their coordinates stay disjoint however badly the keys collide. Uniqueness is a claim about values, not about structure. The footprint gained a row-insertion marker and the assertions themselves. Two sets are compatible when both say which keys they insert, over the same columns, and the filters provably do not intersect; an unqualified insert, different key columns, or filters built with incomparable parameters all leave the assertion unverifiable, which counts as a conflict. The marker is set by any AddFragment that is a data change, which over-approximates: the rows a merge insert updates arrive in a new fragment too and carry no new key, so two writers who only ever touched existing keys can still conflict.

Translating Update

Update translates in its vertical form — rows leaving the fragments they were in and arriving in new ones, which is a Delete and an Append in one step, plus the key assertion and the SSTable progress it carries. Its RewriteRows and RewriteColumns forms are rejected, for the same reason Merge and Project are: what they mean depends on what the read version holds, and the translation only sees the operation. Rejection falls back to the conservative always-retry, so nothing is approximated.

Behaviour differences from the legacy path

The legacy UpdateMemWalState arm of the manifest build never carries the read version's fragments into the manifest it produces, so committing it against a non-empty table empties it. The action path leaves the data alone, which is what the operation means. The parity test asserts both, so the difference is recorded rather than hidden; the legacy bug is untouched here.

A key assertion is compared symmetrically, so a plain append concurrent with a merge insert conflicts in both directions. The legacy check only runs from the retrying transaction's side, so an append that lands after a merge insert is currently allowed through and can introduce a duplicate key.

An update and a column rewrite of the same fragment no longer conflict. Which rows are gone and what a column holds are separate facts about a fragment, so writing a deletion file and rebinding a field's data both land; the legacy operation pairing had to reject this.

Not included

Merge and Project remain untranslatable from the operation alone, as do the two rewrite forms of Update, for the reason above.

[wjones127]

Decision record: Transaction V2 lowering

Reposted from docs/design/transaction-v2-lowering.md, which was dropped from this
stack (commits dc79fe230 and 5eb83ef99) — a decision file does not belong in the
repo, so the record lives here instead.

Two things have moved since it was written, and they override anything below that
contradicts them:

• lower() is rejected outright. The stack keeps no lowering layer and
  translate.rs is never introduced in any of the four PRs. CompositeOperation
  is hand-built by the caller and committed through CommitBuilder::execute,
  behind CommitBuilder::with_experimental_composite_operations.
• Cross-form conflicts fail closed. check_action_txn rejects a V1-vs-V2 pair
  it cannot reason about rather than guessing; there is no backoff routing for a
  checker-raised conflict.

The rest is retained verbatim as the record of what was tried and why, so it does
not get re-proposed.

---

Transaction V2: lowering Operation into actions

Decision record for the lower(operation, read) -> Vec<UserAction> work on the
Transaction V2 stack (#7954 → #8644 → #8645 → #8646). Not published by mkdocs
(docs_dir: src); this is development state, not format documentation.

Read this before proposing an approach in this area. Do not re-propose anything
on a rejected list.

---

1. DropField::apply matches legacy Project

Decision. DropField::apply leaves positive field ids in surviving data
files, prunes only files with no live schema field, and does not touch overlays
— i.e. it reproduces legacy Operation::Project exactly.

Status. Settled 2026-09-08 (Will). Not yet implemented —
DropField::apply still tombstones dead slots at dc79fe230
(action/drop_field.rs:43-60), and its test pins that (:182-185). This
section records the intended behaviour, not the current behaviour.

Options and criteria.

• (a) Change DropField to match legacy. Chosen. Legacy datasets depend on
  the derived watermark described below, and preserving read behaviour of an
  existing format is the higher-order constraint.
• (b) Change legacy Project to tombstone, matching DropField. Rejected —
  see below.

Rejected.

• (b) Legacy Project tombstones dead slots to TOMBSTONED_FIELD (-2).
  Tried and reversed 2026-09-08. Manifest::max_field_id() is derived, not
  persisted (rust/lance-table/src/format/manifest.rs:435-452) and
  deliberately counts field ids in data files already dropped from the schema.
  The stale positive id in a surviving file is therefore the watermark that
  prevents field-id reuse. Tombstoning it to -2 erases the watermark and
  introduces id reuse. Proof in tree:
  rust/lance/src/dataset/schema_evolution.rs:3902-3912 asserts max_field_id
  returns to 0 after drop_columns("x") and the next add_columns("y")
  re-mints that same id.

Open questions.

• PR feat: stabilize field IDs across schema evolution #8658 (feat: stabilize field IDs across schema evolution) adds
  max_allocated_field_id + FLAG_STABLE_FIELD_IDS, activated one-way by
  migrate_to_stable_field_ids. Once a dataset is migrated the watermark is
  persisted and option (b) becomes safe for migrated datasets only. Whether
  DropField should branch on the flag is not decided.

---

2. A cast needs new vocabulary

Decision. Widen the action vocabulary so a cast is expressible.

Status. Settled that we widen (Will, 2026-09-08). Shape not chosen.

Why it is not expressible today. alter_columns' cast path allocates a
new field id (rust/lance/src/dataset/schema_evolution.rs:736-790,
specifically field_dest.set_id(field_src.parent_id, &mut next_field_id)), so a
cast is a field replacement at a fixed schema position, not an in-place
change. Neither existing spelling works:

• AlterField { logical_type } keeps the right position but the wrong identity
  (it preserves the id; the cast does not).
• DropField + AddField gets the right identity but the wrong position, because
  AddField::apply appends to the end of the schema
  (rust/lance-table/src/transaction/action/add_field.rs:48-60).

Note this contradicts the design position currently stated in
protos/transaction/actions.proto's AlterField block comment, which says a
cast is TombstoneFieldData + AddDataFile under a preserved id. That comment
must be corrected alongside whichever shape is chosen.

Options and criteria. All three are unverified — none has been prototyped.

• (a) An identity facet on AlterField, e.g. optional uint32 new_local = 5,
  so one action carries "same position, new id".
• (b) A position anchor on AddField, so DropField + AddField can land in
  place.
• (c) A dedicated ReplaceField action.

Open questions.

• Which shape. Criterion offered: which one keeps AlterField's
  facet-commutativity story (independent facet changes to the same field
  commute) intact.

---

3. Where lowering happens, and what applies the result

This is the load-bearing decision. Status: proposed, not settled.

Decision (proposed). Option C:

1. Lower our own operation at the commit boundary, where the read-version
   manifest is available and uncontaminated, and apply from the lowered
   actions rather than from the legacy build_manifest arm — so conflict
   detection and manifest construction share one source of truth.
2. Lower a concurrent committed legacy operation against its own
   read_version manifest, fetching it if necessary. IO in the conflict path is
   accepted (Will, 2026-09-08: "It's fine if it requires IO").
3. Where a faithful lowering does not exist yet, claim Footprint::exclusive,
   which is behaviour-preserving relative to today's conservative rejection.

Options and criteria.

• (A) Mixed cross-form footprint comparison — keep legacy operations applying
  post-images, and compare them against action sets by lowering them for a
  footprint only. This is what the stack does today (footprint_of,
  rust/lance/src/io/commit/conflict_resolver.rs:2455) and what "Definition of
  done item 5" of the handoff doc asks to extend. Rejected, see below.
• (B) Per-table mutual exclusivity — a writer flag makes a table either V1 or
  V2, never both, so cross-form comparison is unreachable. Set aside, see
  below.
• (C) Lower at the commit boundary; apply from actions. Proposed.

Rejected.

• (A) as a general strategy. Rejected 2026-09-08. A lowering used only for a
  footprint describes a delta application that never happens; the legacy arm
  still installs a post-image. Two proven consequences (see §6):
  test_a_legacy_update_does_not_revert_a_concurrent_tombstone and
  test_a_legacy_delete_does_not_drop_a_concurrent_overlay. The general rule:
  a legacy operation may keep an unconditional conflict; what it must never do
  is claim a delta footprint while applying a post-image.

• "Never lower a concurrent operation; keep the legacy matrix for those."
  Proposed and withdrawn 2026-09-08. It rested on a claim that
  manifest-dependent lowerings cannot compose across several concurrent
  transactions. That claim was an artifact of diffing against the wrong
  manifest. Worked example: v3 is {i:0}; A commits Merge{schema:{0,1}} at
  v4 having read v3; B commits Merge{schema:{0,1,2}} at v5 having read v4.
  Lowered against our (v5) manifest, A fabricates removed_fields={2} for a
  field it never saw and B yields an empty schema delta despite having added
  field 2. Lowered against v3 and v4 respectively, both are correct. Composition
  is a property of using each transaction's own read version, not a property of
  schema diffs.

• An earlier argument that a writer flag "fences the wrong axis." Rejected by
  Will 2026-09-08: CompositeOperation exists only on this branch, not on main
  or any release, so no released library mixes the two forms and mutual
  exclusivity remains available. The error was conflating a flag as a version
  fence with a flag as a table mode.

Why (B) was set aside rather than rejected. It is still the fallback if (C)'s
cost proves too high. Against it: the stack deliberately built and tested
cross-form non-conflict (conflict_resolver.rs:4840-4900), so exclusivity
deletes a capability added on purpose; and it does not by itself deliver the
stated goal of deleting the legacy conflict-resolution code, since V1 tables
keep it either way.

What (C) requires that does not exist yet.

Structural.

• check_txn is synchronous (conflict_resolver.rs:299), so it cannot
  fetch a manifest. Either it becomes async or TransactionRebase::try_new
  (already async) pre-captures the manifests it will need.

• original_dataset is captured before the read-version checkout
  (rust/lance/src/io/commit.rs:1404 vs :1411-1419), so it is not reliably at
  read_version. initial_fragments_for_rebase already works around this by
  re-checking out (conflict_resolver.rs:2394-2402).

• The read_version restamp. Verified 2026-09-10 with a scratch probe
  (three scenarios, Delete with a concurrent Project, with a concurrent
  Delete + Project, and a no-concurrency control). Real, and broader than
  first described in all three respects:

  1. Not confined to retries. commit.rs:1456-1473 runs the rebase on
     every non-strict_overwrite attempt, and load_and_sort_new_transactions
     advances dataset first. So the restamp fires on attempt 0 whenever
     anything at all committed in between — the probe triggered it with a single
     concurrent Project that check_delete_txn:397 declares non-conflicting.
  2. The recorded read_version is degenerate. It is set to
     dataset.manifest.version, and target_version = dataset.manifest.version + 1
     four lines later (commit.rs:1494), so for any committed
     Delete/Update/DataOverlay on this path it is always exactly
     landed_version - 1. It carries no information about the planning base,
     and the original base is recorded nowhere — Transaction has one version
     field. The restamped object is what gets persisted (commit.rs:1477,
     :1483, :1559) and what a later writer reads back.
  3. The inconsistency runs both ways. Delete/Update/DataOverlay
     advance the stamp without fully advancing the payload (:2087-2090,
     :2097-2101, :2123-2128, :2183-2186 — the only four restamp sites).
     finish_create_index (:2200-2295) and finish_rewrite (:2366-2374)
     do the inverse: they rebuild the payload against the latest dataset
     while leaving read_version original.

  Probe output for the full hybrid: recorded read_version = 5, the post-image
  deletion_file written at v5 (merging the concurrent delete's rows), the
  post-image files still matching v3. One record, two bases.

  Mitigation — agreed shape, stronger reason than the one first offered. The
  earlier phrasing ("lower conservatively across both candidate bases") is not
  implementable: there are not two candidates. R == V - 1 always, so the
  "two" manifests are one, and the true base O is unrecoverable. What actually
  saves it is that the restamped set and the diff-requiring set are
  disjoint:

  	needs a base manifest to lower?	restamped?
  Delete, Update, DataOverlay	no — updated_fragments, deleted_fragment_ids, new_fragments, fields_modified, groups[].fragment_id are all absolute	yes
  Merge, Project	yes — the lowering is a schema diff	no (finish fall-through :1927, :1930)
  everything else	no	no

  So a conservative footprint for the hybrid Delete computes from
  updated_fragments ∪ deleted_fragment_ids with no manifest lookup — exact,
  not merely a superset, and it does not collapse to exclusive.

  Three constraints this imposes:

  1. It licenses the footprint only, not the apply. A lowering that diffs
     updated_fragments[*].files against manifest R will emit AddDataFile
     for a concurrently-pruned file — reproducing the resurrection bug in §7.
     The rule must be: lower Delete/Update to deletion-file and
     fragment-removal actions and ignore files entirely — except
     Update { update_mode: RewriteColumns }, whose updated_fragments
     legitimately carry a new data file a diff must pick up. That exception
     needs its own decision.
  2. Merge and Project are safe because they are not restamped, which is
     an accident of finish's fall-through rather than a stated invariant. A
     future finish_merge/finish_project must not restamp. Record this as a
     constraint on the code, not a property to rediscover.
  3. The recorded read_version of a Delete/Update/DataOverlay must
     never be used as a diff base — not "used carefully".

  Unverified. On attempt ≥ 2 the loop re-enters TransactionRebase::try_new
  with the already-restamped read_version, so initial_fragments_for_rebase
  (:2394-2402) checks out a newer base while the post-image stays stale, and
  the fragment.files != updated.files guard (:2455-2461) could fire
  spuriously. Code-reading only; not exercised.

Apply-side parity gaps (faithfulness required; see §4).

Footprint gaps (see §5).

Open questions.

• Whether requirement 1–4 of the handoff doc (replace the TryFrom, lower
  Merge and Project, make it public) ships before or together with routing
  legacy operations through the action apply. They are separable, and the second
  carries all the compatibility risk.
• Whether "Definition of done item 5" is dropped, or becomes sound under (C).
  Under (A) it is unsafe; under (C) it is redundant.

---

4. Apply-side parity gaps

Status. Found, not fixed. All three block routing a legacy operation through
the action apply. Sources: skeptical review 2026-09-08 (code reading only; not
executed).

1. Stable-row-id restamp. merge_fragment_physically_rewritten returns
   true on a file-count change alone
   (rust/lance-table/src/transaction/validate.rs:186-188), and add_columns
   pushes exactly one DataFile per fragment
   (rust/lance/src/dataset/updater.rs:252-260). So every add_columns on
   a stable-row-id table makes legacy rebuild last_updated_at_version_meta as
   a uniform new_version sequence (manifest_build.rs:896-898). On the action
   side AddDataFile::apply only pushes the file
   (action/add_data_file.rs:48-51) and assign_row_ids_to_new_fragments
   touches only minted fragments (action/apply.rs:540-574), of which an
   add-columns lowering has none. Existing legacy test pinning the behaviour:
   manifest_build.rs:2268-2339.

   Corrected 2026-09-10. An earlier revision of this section claimed no
   action can say "restamp every row of this committed fragment," making this a
   vocabulary gap. That was wrong. RefreshRowVersionMetadata
   (action/refresh_row_version_metadata.rs:13-44) says exactly that, on
   committed fragments, by calling
   refresh_row_latest_update_meta_for_full_frag_rewrite_cols — the same helper
   the legacy Merge arm calls. It is already in for_each_action!. So this is a
   lowering gap, not a vocabulary gap: the add_columns recipe must emit
   RefreshRowVersionMetadata for every fragment that gains a data file. The
   cast lowering must emit it too, since merge_fragment_physically_rewritten
   fires on a file-count change (validate.rs:186-188) and a cast always adds
   a file.

2. migrate_to_stable_row_ids cannot be lowered at all. It is an
   Operation::Merge (rust/lance/src/dataset.rs:3227-3250) committed with
   use_stable_row_ids = true on a dataset that has none
   (rust/lance/src/dataset/write/commit.rs:411-414), and
   build_manifest_from_actions rejects exactly that combination
   (action/apply.rs:56-60). ApplyState never reads
   config.migration_next_row_id, and no action sets row_id_meta on an
   existing fragment. Needs vocabulary plus a migration escape hatch.

3. Field-id order for nested additions. Legacy assigns ids by pre-order DFS
   over the merged schema (lance-core/src/datatypes/schema.rs:687-694), so a
   new child of an early struct gets a lower id than a later top-level column.
   mint_field assigns in action order (action/apply.rs:452-460). Fixable by
   emitting AddFields in DFS order. Also AddField::apply discards
   def.children (action/add_field.rs:44), so a struct must be fanned out
   parent-first.

Verified as not divergent (do not re-investigate): retain_relevant_indices
ordering relative to the prune; max_fragment_id; next_row_id; base_paths
and blob ops; Fragment::files ordering (both paths call
normalize_fragments); config / table_metadata; dictionary values (absent on
both paths); and the field-id seed — set_field_id(Some(max_field_id())) and
next_field_id: max_field_id() + 1 reduce to the same value, no off-by-one.

Parity-harness holes that must be closed before parity claims are
trustworthy. assert_parity_with_indices
(rust/lance-table/src/transaction/action/translate.rs:441-478) does not compare
reader_feature_flags, writer_feature_flags, config, table_metadata,
data_storage_format, fragment_offsets, or schema.metadata — the last is
structurally invisible because Schema's hand-written PartialEq compares only
fields (lance-core/src/datatypes/schema.rs:863-867). It also always passes
read_version_state: None on both sides, so anything downstream of that is
invisible to it.

---

5. Footprint gaps

Status. Identified, not designed. These are the cases where lowering makes
something not conflict that should — which by construction is a gap in an
action's preconditions or in the footprint vocabulary, not a reason to abandon
lowering.

1. Row-granular relocation. Proven by
   test_data_overlay_conflicts_with_a_row_relocating_action_set (§6).
   AddOverlays::footprint claims only require_fragment
   (action/add_overlays.rs:52-58); the real precondition is that the rows it
   covers were not relocated. Coordinate
   (action/footprint.rs:32-51) has nothing between fragment and row
   granularity.

   Shape constraints. It cannot be a new Coordinate variant, because
   conflicts_with tests writes.is_disjoint(..) over a HashSet<Coordinate>
   and row sets need intersection. It wants to be a side-channel field —
   HashMap<u64, RoaringBitmap> — checked pairwise, the shape
   required_fragments already has. It must be asymmetric: two concurrent
   overlays over the same cells deliberately both land
   (action/add_overlays.rs:52-58), and a plain delete over an overlaid cell is
   deliberately fine (conflict_resolver.rs:399-403). Only relocation loses
   data. Legacy encodes exactly that distinction as
   moves_rows = !new_fragments.is_empty() && matches!(update_mode, RewriteRows | None)
   (conflict_resolver.rs:1514-1515).

   Open question. How an action set says "relocated" rather than "deleted".
   SetDeletionFile + AddFragment{data_change: true} in one composite is
   suggestive but coarse — a composite that deletes from fragment 0 and appends
   unrelated rows would over-fire. Making relocation explicit in the vocabulary
   would be precise.

2. Nullability and schema-contract assertions. preserves_nullability
   (rust/lance-table/src/transaction/operation.rs:128, :186) drives
   may_alter_nullability (conflict_resolver.rs:48-59), which forces a
   conflict against any value supplier so a concurrent append cannot land a
   fragment with no data for a newly-required field. No action carries it and
   Coordinate has no slot for it. Same for
   updates_schema_or_field_metadata (:79).

   These two guards are safe today only because footprint_of returns
   None for Merge/Project/UpdateConfig, so check_action_txn rejects
   conservatively (:366-373). Lowering removes that net by construction, so
   the footprint rules must land in the same change as the lowering. With a
   correct read-version manifest the nullability tightening does lower — to
   AlterField { nullable: Some(false) } — but FieldDefinition(1) and
   FieldData{frag, 1} are disjoint coordinates, so a rule is still needed.

3. Liveness regressions to avoid. Fabricated removed_fields from diffing
   against the wrong manifest (fixed by §3's read-version rule). Compaction's
   AddFragments must be data_change: false, or inserts_rows makes every
   compaction conflict with every merge insert. And the legacy matrix's
   asymmetry, which a footprint comparison cannot express at all —
   conflicts_with is symmetric by construction, and assert_conflict
   (:4799-4806) loops [(a,b),(b,a)] against one expectation, so the harness
   cannot even state an asymmetric pair.

   Audited 2026-09-10 — see the appendix (§9) for the full matrix.
   34 ordered-pair relations disagree with their reverse, not the one this
   section originally named. They fall in three groups: 11 where Overwrite is
   permissive as ours and fatal as theirs, 10 the same for Restore, and 13
   genuine semantic asymmetries. 20 of the 34 are untested in either
   direction, and the untested set sits exactly where a symmetric redesign
   would change behaviour silently: the whole Restore row (it never appears as
   ours in the test module), Clone and UpdateBases (the only two operations
   that bypass check_action_txn, at :347 and the _ => Ok(()) catch-all at
   :1880), CreateIndex vs UpdateMemWalState for a non-MemWAL index (exact
   inverses), and Merge vs Project (which differ in severity — inc one
   way, retry the other — with nothing recording which is intended).

---

6. Evidence

Three tests were written to convert code-reading findings into reproductions.
All three fail on 27528dd20, each for its own reason:

Test (rust/lance/src/io/commit/conflict_resolver.rs)	Asserts	Observed
test_data_overlay_conflicts_with_a_row_relocating_action_set	overlay covering a relocated row conflicts	Ok(()) — no conflict. The disjoint-coverage case passes, so the test discriminates.
test_a_legacy_update_does_not_revert_a_concurrent_tombstone	field 1 stays tombstoned	[0, 1] vs [0, -2] — the field's data is resurrected
test_a_legacy_delete_does_not_drop_a_concurrent_overlay	the overlay survives	0 vs 1 — the overlay is dropped

All three are committed #[ignore]d, each pointing back at the section here
that explains it — this doc is the tracker; no GitHub issues were filed
(Will, 2026-09-08). Run them with:

cargo test -p lance --lib conflict_resolver::tests -- --ignored

Un-#[ignore] a test in the same change that closes its gap. If one starts
passing for any other reason, the reason is a regression somewhere else.

The third is a pre-existing bug independent of Transaction V2:
Operation::Delete's apply replaces the fragment entry wholesale and, unlike
Operation::Update's arm (manifest_build.rs:624, updated.overlays = f.overlays.clone()), does not carry overlays forward, while
check_delete_txn explicitly permits a concurrent DataOverlay. Ordering
dependent — overlay-after-delete is fine.

---

7. Separate bugs found, to be tracked on their own

• apply_mem_wal_index_coverage never runs on the action path. Confirmed:
  build_manifest_with_read_version short-circuits at
  manifest_build.rs:413-421 before read_version_state is used, and
  build_manifest_from_actions has no such parameter. That helper is the
  invalidation half as well as the crediting half (manifest_build.rs:279-294
  drops a catch-up entry when coverage cannot be shown), so a stale
  index_catchup survives a coverage-narrowing commit and the WAL pod retires
  SSTables against it (manifest_build.rs:157-158). Affects every
  action-path commit, not just Merge/Project. Live in the stack now.
• A legacy Delete un-prunes a concurrent Project. Found 2026-09-10 by
  the restamp probe (§3), same family as the overlay bug below: Delete's apply
  replaces the fragment entry wholesale from a post-image built before the
  Project landed, so a data file the Project pruned is reinstalled. The
  probe's v5 manifest has two data files for a schema with one field.
  check_delete_txn:397 permits Project for the same reason it permits
  DataOverlay. Consequence for §1: max_field_id goes 1 → 0 (the
  Project) → 1 (our Delete), so the derived watermark §1 makes load-bearing is
  non-monotonic under a Delete/Project race. Not yet characterised on
  the read path, and not yet reproduced as a committed test.
• test_non_null_claim_barrier (conflict_resolver.rs:4125-4202) cannot fail
  the way it claims to. It asserts
  assert_eq!(matches!(result, Err(RetryableCommitConflict{..})), claims), which
  for claims = false passes on Ok and on IncompatibleTransaction. Its
  Project fixture is an empty Schema::default() while the replacement's
  DataFile carries fields: vec![0], so (DataReplacement, Project) actually
  takes the data_replacement_field_removed_err branch (:1291-1301) — the
  widest asymmetry in the matrix runs inside this test and nothing checks it.
  Worth tightening independently of the redesign.
• test_an_update_does_not_conflict_with_a_column_rewrite_of_the_same_fragment
  (conflict_resolver.rs:4901-4915) asserts a non-conflict the legacy apply
  cannot honour — see §6, row 2. Its vertical_update_txn fixture also has
  new_fragments: vec![], so it does not model the merge insert its doc comment
  claims; the footprint is identical either way, so the test would still pass
  while the semantics became unsafe.

---

8. Standing open questions

1. Cast widening shape (§2).
2. Whether DropField branches on FLAG_STABLE_FIELD_IDS once feat: stabilize field IDs across schema evolution #8658 lands (§1).
3. How relocation is spelled in the action vocabulary (§5.1).
4. Sequencing: lowering-only PR vs lowering-plus-action-apply (§3).
5. Whether Update { update_mode: RewriteColumns } lowers by diffing files
   (the one case where the hybrid post-image genuinely bites the apply path) or
   by some other route (§3, mitigation constraint 1).
6. Which side of each of the 13 genuine asymmetries (§9) the symmetric footprint
   should adopt — in particular Merge/Project, where the two directions
   differ in severity rather than verdict, and UpdateBases/CompositeOperation,
   where a base-name collision is fatal legacy-to-legacy and silently fine
   legacy-to-action.
7. Whether the three §6 tests are committed #[ignore]d against tracking
   issues, or held until their fixes land. Settled 2026-09-08: committed
   #[ignore]d, referencing this doc rather than a GitHub issue. Findings stay
   on the branch until the design is agreed.

---

9. Appendix: the legacy pairwise conflict matrix

Status. Audited 2026-09-10 against dc79fe230. Reference material for §5.3
and open question 6, not a decision. Rows are ours (whose check_*_txn
runs); columns are theirs (the already-committed concurrent transaction).

ok no conflict · retry retryable · inc incompatible (unretryable) · 2nd
may pass here and conflict in a finish_* second phase · fp decided by
footprint comparison · cond ... conditional.

AP Append · DL Delete · OW Overwrite · CI CreateIndex · RW Rewrite ·
DR DataReplacement · DO DataOverlay · MG Merge · RS Restore ·
RF ReserveFragments · UP Update · PJ Project · UC UpdateConfig ·
MW UpdateMemWalState · CL Clone · UB UpdateBases ·
CO CompositeOperation.

9.1 Before dispatch

check_txn runs once per concurrent transaction, unfiltered
(commit.rs:1469-1471), skipped only for strict_overwrite. Two symmetric
guards fire before the operation-pair match and short-circuit to a retryable
conflict — the only symmetric rules in the file:

• G1 (:303-307) may_alter_nullability(one) && supplies_values(other).
  may_alter_nullability (:46-59) = Project/Merge with
  preserves_nullability: false; supplies_values (:65-73) = Append,
  Update, DataReplacement, DataOverlay.
• G2 (:311-315) Merge opposite updates_schema_or_field_metadata
  (:79-95) = UpdateConfig with non-empty schema_metadata_updates or
  field_metadata_updates.

Every check_*_txn has an explicit CompositeOperation arm routing to
check_action_txn — except two: Clone as ours never reaches a check_*
at all (:347), and check_add_bases_txn swallows it in _ => Ok(())
(:1880). Those are the only two such arms; the other fifteen matches are
exhaustive.

check_action_txn (:361-379) is symmetric but conservative-by-default: a
None from footprint_of (:2455-2464) on either side is a retryable
conflict. footprint_of yields Some for Append, Delete, UpdateBases,
CreateIndex, DataOverlay, UpdateMemWalState, DataReplacement, and
Update (only when reject_untranslatable_update, translate.rs:215-250,
passes), and None for Overwrite, Rewrite, Merge, Restore,
ReserveFragments, Project, UpdateConfig, Clone.

9.2 Table A — theirs in {AP, DL, OW, CI, RW, DR}

ours \ theirs	AP	DL	OW	CI	RW	DR
AP :1234	ok :1252	ok :1255	inc :1247	ok :1254	ok :1253	ok :1263
DL :381	ok :390	cond overlap → 2nd :424	inc :469	ok :392	cond frag overlap :402	cond target frag in ours :413
OW :1180	ok :1215	ok :1218	cond: inc if upsert_key_conflict, else retry :1186	ok :1219	ok :1220	ok :1221
CI :735	ok :751	ok :832	inc :963	cond name / frag-reuse / mem-wal / append-drop / identity :758	cond → 2nd :858	cond replaced field in index fields :927
RW :979	ok :997	cond frag overlap :1005	inc :1161	cond → 2nd :1075	cond frag overlap or both-sides FRI :1038	cond frag overlap :1059
DR :1268	ok :1279	cond: inc if target in deleted_fragment_ids, else ok :1310	inc :1423	cond index depends on replaced field :1365	cond frag overlap :1389	cond frag and field overlap :1403
DO :1453	ok :1462	cond overlaid frag in deleted_fragment_ids :1475	inc :1545	ok :1466	cond rewrite touches overlaid frag :1525	ok :1469
MG :1556	retry :1580 (G1 if on:false)	retry :1582	inc :1589	cond iff their new_indices has MEM_WAL_INDEX_NAME :1568	retry :1583	retry :1585
RS :1598	ok :1609	ok :1610	ok :1611	ok :1612	ok :1613	ok :1614
RF :1630	ok :1642	ok	inc :1639	ok	ok	ok
UP :490	cond iff our inserted_rows_filter.is_some() :625	cond overlap → 2nd :654	inc :701	ok :551	cond frag overlap :634	cond frag overlap :644
PJ :1661	cond G1, else ok :1673	ok :1675	inc :1688	ok :1677	ok :1680	cond G1, else ok :1678
UC :1696	ok :1742	ok	cond: inc if schema/field metadata or upsert_key_conflict :1710	ok	ok	ok
MW :1763	inc :1820	inc	inc	cond: MemWAL → sstable comparison, else ok :1800	ok :1816	inc
CL :347	ok	ok	ok	ok	ok	ok
UB :1844	ok :1880	ok	ok	ok	ok	ok
CO :361	fp	fp	retry (no footprint)	fp	retry	fp

9.3 Table B — theirs in {DO, MG, RS, RF, UP, PJ}

ours \ theirs	DO	MG	RS	RF	UP	PJ
AP	ok :1264	cond G1: retry if on:false, else ok :1260	inc :1248	ok :1257	ok :1256	cond G1, else ok :1258
DL	ok :397	retry :466	inc :469	ok :391	cond overlap → 2nd :424	ok :394
OW	ok :1222	ok :1223	ok :1224	ok :1226	ok :1227	ok :1228
CI	ok :754	cond iff our new_indices has MemWAL :846	inc :963	ok :856	ok, mutates new_indices via prune_updated_fields_from_indices :834	ok :857
RW	cond overlay on a rewritten frag :1024	retry :1072	inc :1161	ok	cond frag overlap	ok
DR	ok :1284	retry :1305	inc	ok	cond: inc if target removed; retry if moved_rows/field_rewritten; else ok :1328	cond G1; else inc if a replaced field is absent from their schema :1289; else ok
DO	ok :1470	retry :1538	inc :1545	ok	cond: retry if overlaid frag removed; else moves_rows → mark → 2nd; else ok :1498	cond G1, else ok
MG	retry :1586	retry :1584	inc :1590	ok :1576	retry :1581	inc :1591
RS	ok :1615	ok :1616	ok :1617	ok :1618	ok :1620	ok :1621
RF	ok	ok	inc :1639	ok	ok	ok
UP	cond: RewriteColumns → retry on frag+field overlap; RewriteRows/None with new_fragments → retry on moved-rows meeting coverage (or if no affected_rows); else ok :558	retry :698	inc :701	ok	cond key-filter pre-check :506, then overlap → 2nd :654	cond G1, else ok
PJ	cond G1, else ok :1679	retry :1684	inc :1689	ok :1682	cond G1, else ok :1674	retry :1684
UC	ok	cond G2: retry if we set schema/field metadata, else ok	ok :1742	ok	ok	ok
MW	inc	inc	inc :1823	ok :1816	cond sstable shard/generation :1789	inc
CL	ok	ok	ok	ok	ok	ok
UB	ok	ok	ok	ok	ok	ok
CO	fp	retry	retry	retry	cond: retry if untranslatable, else fp	retry

9.4 Table C — theirs in {UC, MW, CL, UB, CO}

ours \ theirs	UC	MW	CL	UB	CO
AP	ok :1261	inc :1249	ok :1262	ok :1259	fp :1242
DL	ok :395	inc :471	ok :393	ok :400	fp :387
OW	cond: inc iff upsert_key_conflict :1200	inc :1212	ok	ok	retry (no footprint)
CI	ok :926	cond: our MemWAL index → collect sstables, ok → 2nd; else inc :947	ok	ok	fp
RW	ok	ok :1002	ok	ok	retry
DR	ok	inc :1423	ok	ok	fp
DO	ok	inc :1545	ok	ok	fp
MG	cond G2 :1577	inc :1592	ok :1576	ok :1579	retry
RS	ok :1623	inc :1624	ok :1622	ok :1619	retry
RF	ok	ok :1656	ok	ok	retry
UP	ok :551	cond sstable shard/generation :704	ok	ok	fp / retry if untranslatable
PJ	ok :1676	inc :1690	ok :1681	ok :1683	retry
UC	cond: inc iff upsert_key_conflict or modifies_same_metadata :1726	ok :1755	ok	ok	retry
MW	ok :1816	cond sstable shard/generation :1777	inc :1828	ok :1819	fp
CL	ok	ok :347	ok	ok	ok :347
UB	ok	ok	ok	cond: inc on id / name / path collision :1848	ok :1880 (catch-all)
CO	retry	fp	retry	fp	fp

check_compacted_sstables_conflict (:1884-1911), behind the MW/UP/CI sstable
cells: ok if no shared shard_id; inc if
committed.generation >= to_commit.generation; retry otherwise.

9.5 The 13 genuine semantic asymmetries

Excluding the 21 that are artifacts of Overwrite's and Restore's permissive
Ok(()) arms (:1215-1229, :1609-1623) against everyone else's incompatible
arm. Note RS/OW is symmetric (ok both ways, :1224 and :1611) —
arguably the odd cell rather than the asymmetric ones.

1. AP/MG — (AP,MG) ok :1260; (MG,AP) retry :1581. Only when
   preserves_nullability: true. Merge's own doc comment
   (operation.rs:123-128) says the conflict is needed "in either commit
   order"; the code enforces one direction only.
2. MG/PJ — inc :1591 one way, retry :1684 the other. Same pair,
   different severity: fatal one way, rebasable the other.
3. DL/DO — (DL,DO) ok :397; (DO,DL) retry :1479. The permissive
   direction is known wrong (§6 row 3).
4. DL/DR — retry on any frag overlap one way :413; inc only if the
   fragment was removed, ok if merely tombstoned, the other :1310.
5. UP/DR — retry on plain frag overlap :644; vs inc-if-removed /
   retry-if-moved_rows-or-field_rewritten / else ok :1328.
6. DR/PJ — inc :1289 vs ok :1678. The widest gap in the table
   when preserves_nullability: true. Executed but not asserted by
   test_non_null_claim_barrier — see §7.
7. UP/DO under RewriteColumns — retry on frag+field overlap :558; ok the
   other way because moves_rows is false :1513. The two comments state
   opposite mechanisms for the same situation (:566-575 vs :1510-1511);
   one of them is wrong.
8. UP/AP — retry iff our inserted_rows_filter.is_some() :625; ok :1256.
   The filter lives only on the update, so the append cannot make the argument.
9. CI/MW — for a plain column index the directions are exact inverses:
   inc :947 vs ok :1800.
10. CI/RW — (RW,CI)'s (None, Some(_)) arm :1103 retries on
    fragment_bitmap straddle; the mirror falls through to Ok(()) :896
    after only an NGram-specific check :865.
11. CL/MW — ok :347 vs inc :1828.
12. CL/CO — ok :347 vs retry. Clone is the only operation that
    bypasses the action-footprint route entirely.
13. UB/CO — ok via the catch-all :1879 vs a real footprint comparison
    against AddBase's BaseName/BaseLocation coordinates
    (action/add_base.rs:50). Two base additions colliding on a name are
    inc legacy-to-legacy and ok legacy-to-action.

9.6 Pairs whose verdict depends on a second phase

Not reproducible by a footprint comparison alone:

• (DL,DL), (DL,UP), (UP,DL), (UP,UP) — check_* sets
  *needs_rewrite |= updated.deletion_file != fragment.deletion_file (:447,
  :677); finish_delete_update (:1943-2103) then reads each flagged
  deletion file (:1979), intersects it with affected_rows, and raises a
  retryable conflict on overlap (:1996) — or an internal error if
  affected_rows is None (:2091).
• (DO,UP) with moves_rows — check_* sets *needs_row_check = true
  (:1516); finish_data_overlay (:2117-2187) computes
  moved_rows = current - initial and conflicts if it meets the overlay's
  coverage (:2160), or if the overlaid fragment vanished (:2135). Pinned by
  test_data_overlay_finish_conflicts_with_row_moving_update (:4203).

Rewrites the transaction but never conflicts — reproducible by a footprint, but
not by an unmodified one: (CI,RW) with both sides carrying an FRI
(:890 → :2200); (CI,MW) with our MemWAL index (:956 → :2245);
(RW,CI) in the (Some, Some) arm (:1093 → :2304); and (CI,UP), which
mutates new_indices in place at :838. Unverified: those finish_* paths
.unwrap() the loaded details (:2206, :2225, :2315) — whether any is
reachable with None was not established.

9.7 Test coverage of the asymmetric pairs

assert_conflict (:4801-4809) loops [(a,b),(b,a)] against a single
expectation, so it structurally cannot express an asymmetric pair; it is used
only by the four action-footprint tests at :4810-4841, none of which touch
one. The directional harnesses are test_conflicts (:3189-3695, 15 rows × 9
columns — AP, CI, DL, MG, OW, RW, RF, UP, UC only), test_data_overlay_conflicts
(:3696-3877), test_conflicts_data_replacement (:5518-5836), and the MemWAL
sstable tests (:5837-6125).

9 of the 34 are covered in both directions, 5 in one direction only, 20 not at
all. Untested: the entire Restore row (10 pairs — Operation::Restore
appears exactly once as an operand in the whole test module, as theirs at
:3829, and never as ours); OW/DR, OW/PJ, MG/PJ (Project is neither a
row nor a column of test_conflicts); DL/DR and UP/DR (covered in the
DataReplacement direction only); UP/AP (the test_conflicts Update row has
inserted_rows_filter: None at :3446, so :625 is never taken); CI/MW for a
plain column index; CL/MW and CL/CO; and UB/CO
(test_add_bases_no_conflict_with_data_operations :5296-5345 uses only
Append, Delete, Update as theirs).

UP/DO is covered in both directions but the two expectations are asserted in
separate tests, so nothing flags that they disagree.

9.8 Provenance

check_txn :299-350, check_action_txn :361-379, check_delete_txn
:381-486, check_append_txn :1234-1266, check_merge_txn :1556-1596,
check_restore_txn :1598-1628, check_project_txn :1661-1694,
check_add_bases_txn :1844-1882, all four finish_*, footprint_of
:2455-2464 and initial_fragments_for_rebase :2382-2410 were read line by
line by two readers. The remaining cells — check_update_txn,
check_create_index_txn, check_rewrite_txn, check_overwrite_txn,
check_data_replacement_txn, check_data_overlay_txn,
check_update_config_txn, check_update_mem_wal_state_txn, and the
test-coverage line numbers — come from a single read and are UNVERIFIED by a
second reader. Spot-check before treating a cell as normative.

[wjones127]

Self-review pass (polish-pr), 2026-09-20

Decision

ApplyState::update_compacted_sstables was documented as "creating the MemWAL index when the dataset does not have one yet". update_mem_wal_index_compacted_sstables does the opposite — a missing MemWAL index is an invalid_input error — which is what UpdateCompactedSsTables's own doc and test_recording_progress_without_a_mem_wal_index_is_rejected already assert. The doc stated the inverse of the contract and is corrected. Also hoisted an in-function use in the same module's tests, per rust/AGENTS.md.

Status

Settled and implemented. fmt clean, clippy --all --tests --benches -- -D warnings clean, scoped pre-commit run --files clean. lance-table transaction tests 276 pass; lance dataset_transactions + io::commit 176 pass.

Open questions

The one worth a decision before this merges:

• KeyAssertion.key_fields compares Ref::Local tokens across transactions. Footprint::assert_unique_keys stores the unresolved refs, and key_assertion_violated_by compares them between two different transactions with ours.key_fields != theirs.key_fields. Ref::Local(n) is a per-transaction token, so two writers that both name Ref::Local(0) for unrelated new fields compare equal and fall through to the filter intersection — which can then clear an assertion over a column the other writer never touched. That is the unsafe direction. Every other footprint helper strips local refs first (committed_field, committed_fragments). The proto round-trip test feeds Ref::Local(3) into key_fields, so the case is representable. The same shape appears in feat(transaction): implement the index management actions #8645's IndexIdentity; both should go the same way.

Smaller ones:

• inserts_rows is documented as an over-approximation set by any data-change AddFragment, and is silent on whether a data-change AddOverlays can restate a key column's value and introduce a key without any new fragment. If it can, the flag is an under-approximation, which is the unsafe direction for an assertion.
• AssertUniqueKeys is never driven through CommitBuilder — all conflict behaviour is unit-tested against Footprint::conflicts_with directly, so nothing proves the commit loop consults the key assertions at all. commit_from_stale plus a RetryableCommitConflict assertion is ~15 lines and is the cheapest high-value test missing here.
• Footprint::key_assertion_violated_by only ever sees FilterType::ExactSet in tests. The Ok((true, _)) (ExactSet vs Bloom) and Err(_) (mismatched Bloom parameters) arms are untested, and mismatched-size Bloom filters are the common production case.
• The retry re-stamp property that ApplyState::new_version() documents is tested nowhere: every test commits with no intervening writer, so new_version() == read_version + 1 throughout and a plan-time-stamping implementation would pass all of them. This is what decides overlay precedence.
• AssertUniqueKeys carries its key columns twice — the authoritative key_fields, and filter.field_ids inherited from the shared KeyExistenceFilter, which the writer clears on encode but the decoder keeps.
• test_a_commit_carrying_a_key_assertion_changes_nothing_itself asserts only a fragment count, which the AddFragment in the same commit already guarantees; the assertion action contributes nothing observable.
• The Footprint::from(&CompositeOperation::new(vec![UserAction::new("step", actions)])) closure is now redeclared in nine action test modules; action/test_support.rs exists for exactly this.

🤖 Generated with Claude Code

[wjones127]

Legacy conflict characterization run against this PR (burn-down item 14)

Decision

#9220's characterization suite passes unchanged on this PR: 42 passed, 0 failed, 6 ignored, ~2s.

Status

Verified 2026-09-20. Nothing in the stack changes legacy conflict behaviour.

Options and criteria

The suite's branch (will/legacy-conflict-characterization, base 8cdffd30e) sits 56 commits ahead of this stack's base (93ba1d20c), so the suite does not compile on the stack as-is — FileFragment::write_overlay, which scenarios.rs uses, does not exist at the older base. That is base skew, not a behavioural difference.

Method: for each PR, a throwaway branch off the PR head, merge 8cdffd30e (clean, no conflicts), cherry-pick the suite's five commits, cargo test -p lance --lib conflict_matrix. Throwaway branches deleted afterwards; nothing here is pushed.

Results, all four layers:

PR	Result
#7954	42 passed, 0 failed, 6 ignored
#8644	42 passed, 0 failed, 6 ignored
#8645	42 passed, 0 failed, 6 ignored
#8646	42 passed, 0 failed, 6 ignored

#8644 is the layer that matters most here — it is the one that touches conflict_resolver.rs, adding the fail-closed CompositeOperation arm to every check_*_txn and spelling out check_add_bases_txn's wildcard. The suite covers exactly that file and is unaffected.

Open questions

• The suite's 6 ignored cases are ignored on its own branch too; this run does not change them.
• The stack will need a rebase onto current upstream/main before merge regardless; the clean merge above is evidence that rebase is not expected to be eventful.

🤖 Generated with Claude Code

[wjones127]

Restacked onto the new #8645 head and added df53bcf62: overlays are partial writes of their fields' data (commuting with each other, conflicting with a column rewrite in either order) and require their fields' definitions; key assertions are violated by any write into a key column and matched per key-column set, with the contract that an assertion speaks for every row its set inserts. Decision record is in #8644 (comment).