UserOperation: Action-based Transactions for real diffs

[wjones127]

Create a new operation to improve transactions so that:

1. It's easy for users to read the transaction history
2. The operation is a true diff of the manifest

This is based on earlier proposal #3734. I've simplified the proposal, removing CompositeOperation and setting the advanced use cases aside.

Design

There will be two levels

User Operation

User operations are the user-facing ones. They will be the high-level operations that users recognize, such as INSERT, CREATE INDEX, etc.

message UserOperation {
  // Human-readable description, e.g. "INSERT INTO t VALUES (1);"
  string description = 1;
  // Unique identifier for this user operation
  UUID uuid = 2;
  // The version this operation was planned against
  uint64 read_version = 3;
  // The ordered list of manifest changes
  repeated Action actions = 4;
}

Actions

Action is a granular change to the manifest. Traditional operations will be made up of multiple of these.

message Action {
  oneof action {
    AddFragments add_fragments = 1;
    RemoveFragments remove_fragments = 2;
    UpdateFragment update_fragment = 3;
    AddFields add_fields = 4;
    DropFields drop_fields = 5;
    UpdateSchemaMetadata update_schema_metadata = 6;
    AddIndex add_index = 7;
    RemoveIndex remove_index = 8;
    UpdateConfig update_config = 9;
    UpdateTableMetadata update_table_metadata = 10;
    SetDataFormat set_data_format = 11;
    SetFeatureFlags set_feature_flags = 12;
  }
}

Details

// Append new fragments to the table.
// Covers: Append, the "add new rows" part of Overwrite, new fragments from compaction.
message AddFragments {
  repeated DataFragment fragments = 1;
}

// Remove fragments entirely by ID.
// Covers: the "remove old" part of Overwrite, removing compacted fragments.
message RemoveFragments {
  repeated uint64 fragment_ids = 1;
}

// Modify a single existing fragment. One message per fragment touched.
// Covers: Delete (new deletion file), Update (new data files + new deletion file),
//         Merge/AddColumns (new data files for new fields), DropColumns (tombstone fields).
message UpdateFragment {
  // The fragment being modified.
  uint64 fragment_id = 1;

  // --- Data file changes ---

  // New data files to add to this fragment (e.g. new column data, rewritten columns).
  repeated DataFile added_data_files = 2;
  // Data file paths to remove from this fragment (e.g. replaced by rewritten files).
  // Matched by DataFile.path.
  repeated string removed_data_file_paths = 3;

  // --- Deletion file changes ---

  // If set, replaces the fragment's current deletion file.
  // To clear an existing deletion file (e.g. after materializing deletions), set
  // clear_deletion_file = true instead.
  optional DeletionFile new_deletion_file = 4;
  // If true, remove the current deletion file (deletions have been materialized).
  bool clear_deletion_file = 5;

  // --- Row ID / lineage changes ---

  // Updated row ID sequence, if changed.
  oneof new_row_id_sequence {
    bytes inline_row_ids = 6;
    ExternalFile external_row_ids = 7;
  }

  // Updated row lineage tracking, if changed.
  oneof new_last_updated_at_version_sequence {
    bytes inline_last_updated_at_versions = 8;
    ExternalFile external_last_updated_at_versions = 9;
  }

  oneof new_created_at_version_sequence {
    bytes inline_created_at_versions = 10;
    ExternalFile external_created_at_versions = 11;
  }

  // Updated physical_rows, if it changed (e.g. after partial compaction within a fragment).
  optional uint64 physical_rows = 12;

  // Field IDs to tombstone (-2) in existing data files within this fragment.
  // Used for drop_column: marks fields as dead in their original data files without
  // removing the files (other fields in those files may still be live).
  repeated int32 tombstoned_field_ids = 13;
}

// ---- Schema Actions ----

// Add new fields to the schema.
// Covers: AddColumns / Merge, adding fields during Overwrite.
message AddFields {
  repeated lance.file.Field fields = 1;
  // Optional new entries for the schema metadata map.
  map<string, bytes> added_schema_metadata = 2;
}

// Drop fields from the schema by field ID.
// Covers: DropColumns.
// Note: The corresponding data is tombstoned via UpdateFragment.tombstoned_field_ids.
message DropFields {
  repeated int32 field_ids = 1;
  // Schema metadata keys to remove.
  repeated string removed_schema_metadata_keys = 2;
}

// Update schema-level metadata without changing fields.
message UpdateSchemaMetadata {
  map<string, bytes> set_values = 1;
  repeated string remove_keys = 2;
}

// ---- Index Actions ----

// Add or replace an index.
message AddIndex {
  // The full index metadata entry (same structure already used in the index section).
  IndexMetadata index = 1;
}

// Remove an index by its UUID.
message RemoveIndex {
  UUID index_uuid = 1;
}

// ---- Config / Metadata Actions ----

// Update table configuration (lance.* keys, etc.).
message UpdateConfig {
  map<string, string> set_values = 1;
  repeated string remove_keys = 2;
}

// Update table metadata (arbitrary user key-value pairs).
message UpdateTableMetadata {
  map<string, string> set_values = 1;
  repeated string remove_keys = 2;
}

// Set the data storage format.
message SetDataFormat {
  Manifest.DataStorageFormat format = 1;
}

// Set reader/writer feature flags. Values are OR'd onto existing flags.
message SetFeatureFlags {
  // Bits to set (OR) on reader_feature_flags.
  uint64 enable_reader_flags = 1;
  // Bits to set (OR) on writer_feature_flags.
  uint64 enable_writer_flags = 2;
}

Benefits

• Easier cleanup of cancelled transactions: can see what the set of new files in a transaction are more easily.
• Easier diffs between table versions
• Can do more complicated operations like append and update an index in a single transaction.

[westonpace]

Very cool. Should description be repeatable? Or maybe something like...

message UserOperation {
  // Unique identifier for this user operation
  UUID uuid = 2;
  // The version this operation was planned against
  uint64 read_version = 3;
  // The ordered list of user actions
  repeated UserAction actions = 4;
}

message UserAction {
  // Human-readable description, e.g. "INSERT INTO t VALUES (1);"
  string description = 1;
  // The ordered list of manifest changes
  repeated Action actions = 4;
}

I think a singular description might satisfy the "append and update in a single transaction" case but might not fully satisfy the branch-and-merge case. This would purely be for analyzing the transaction log. To apply the operation it should be fine to just flatten the list of actions.

[charleshuang119]

Love this design. The move from monolithic oneof Operation to composable repeated Action is a big improvement. Being able to expressAddFragments + AddIndex in one commit directly solves the performance cliff problem from #5952.

One gap I noticed: there's no tag-related action in the Action enum. In #5952 @beinan highlighted the need for atomic Data + Index + Tag commits, which is exactly our use case at Uber (we run on OCI-compatible objectstorage). Today, tags live as separate JSON files under _refs/tags/ via a plain PU, completely decoupled from the manifest commit.

Would it make sense to add:

message AddTag {
    string tag_name = 1;
}
message RemoveTag {
    string tag_name = 1;
}

as Action variants? That way, a UserOperation could bundle AddFragments + AddIndex + AddTag and the tag becomes visible atomically with the data and index. No second PUT, no split-brain window. (The existing Transaction proto already has a tag field, and Manifest has one too, but neither is connected to _refs/tags/. This new Action model seems like the right place to unify them.)

Also +1 to @westonpace's UserAction middle layer. For the branch-and-merge workflow in #5952, having per-step descriptions(e.g. "Worker A: append batch" / "Worker B: build index") would make the transaction log much easier to debug.

[wjones127]

I've created a milestone to track prototyping this: https://github.com/lance-format/lance/milestone/11

We'll have a vote once it's ready to merge.

[wjones127]

I'm reviving this now. The prototyped protobuf files are here: #7954

I plan on drafting implementations of Rust library changes and conflict resolution next. I'll be integrating it with an e2e spike in LanceDB, and once I'm happy with that I'll have a final change up for a vote.

[wjones127]

Status update: implementation is up, merging as an experimental feature

The prototype is now a four-PR stack, and I'd like to merge it under the experimental specification features process that passed in #8305, rather than hold it all behind a single stabilization vote.

• feat(transaction): action-based transaction wire format (Transaction V2) #7954 — wire format (protos/transaction/actions.proto), read-side fail-closed
• feat(transaction): implement the Transaction V2 action vocabulary #8644 — apply, footprints, conflict resolution, the CommitBuilder opt-in
• feat(transaction): implement the index management actions #8645 — index actions
• feat(transaction): implement the remaining drafted actions #8646 — the remaining actions

What changed from the proposal above

The design in the opening post has drifted enough that it should not be read as current.

The two-level shape landed as @westonpace suggested it. CompositeOperation holds UserActions (description + actions), which hold Actions. Descriptions survive squashing; action lists are flattened on apply.

Actions are singular and finer-grained than the sketch. AddFragments / RemoveFragments / UpdateFragment became AddFragment, AddDataFile, RemoveFragment, SetDeletionFile, TombstoneFieldData, AddOverlays, RefreshRowVersionMetadata. Coarse actions made concurrent edits to one fragment conflict when they need not.

New id references. A minting action (AddFragment, AddField, AddBase) cannot know its id until commit, and gets a different one when replayed onto another target, so it names the id with Ref::Local(n), resolved against the target's counters at apply. A writer that must bake an id into a file it writes in the same commit takes Committed ids from a range reserved by ReserveFragmentIds / ReserveRowIds instead.

Index actions operate on segments, not indices. The format has no first-class index apart from its segments, so AddIndexSegment / RemoveIndexSegment / AdjustIndexCoverage mirror that; a logical index is the set of segments sharing a name.

Assertions. AssertUniqueKeys carries a precondition no post-image can reconstruct — the key set a merge-insert inserted.

Cross-form conflicts fail closed. A CompositeOperation concurrent with any named operation is rejected as a conflict rather than compared. Parity with the legacy conflict matrix is explicitly not attempted: an audit found ~20 asymmetric operation pairs untested in either direction, so there was no specification to be faithful to.

Scope. The user-facing surface is hand construction through CommitBuilder, gated behind with_experimental_composite_operations. No high-level operation emits V2, and there is no Operation -> Vec<Action> lowering — that was prototyped and removed.

@charleshuang119 — on atomic Data + Index + Tag: the vocabulary still has no tag action, and I think that should be a separate issue rather than folded in here. Data + index in one commit works today. Tags are the harder half: they live in _refs/tags/ and are not manifest-coordinated at all, so an AddTag action would first need the tag store tied into the commit, which is its own design change with its own compatibility story. The action model does look like the right eventual home for it.

Experimental declaration

Prerequisites:

1. Marked experimental in protobuf and documentation. actions.proto, transaction.proto, the Rust module docs, and a new section in the transaction specification.
2. Forward compatible, with one caveat worth stating plainly. A composite operation produces an ordinary manifest, so scanning a table written this way needs no knowledge of the feature. But a transaction is usually inlined into the manifest, and a reader that fails when an inline transaction does not decode cannot open the table at all. Rust has tolerated that since v12.0.0 (fix: don't fail dataset open on an undecodable inline transaction #7740). Readers older than that do not, which is a pre-existing bug affecting several already-shipped operations, now filed as bug: readers before v12.0.0 fail to open a dataset whose inline transaction they cannot decode #9454 — it is not specific to this feature, but this feature would add to its blast radius, so it is worth fixing first. Separately, a reader that decodes the transaction on purpose — history inspection, or a concurrent commit's conflict check — rejects it by design, so that a concurrent V2 commit aborts an in-flight commit rather than being silently skipped.
3. Backward compatible. Writers that do not use it are unchanged.
4. Dropping requires no data rewrite. Only transaction files carry the new variant; manifests are ordinary.

Commitments 2–4 accepted: the feature is removed if the stabilization vote does not pass, breaking changes may be made without a separate vote, and the PMC may require backwards-incompatible changes during stabilization.

On commitment 5 ("a stable version may not contain experimental features"): this feature adds no feature flag and no format version bump — it lives in the transaction file — so it does not gate a stable file format version.