What should be in scope for Lance?

[wjones127]

Lance today spans a wide range. At one end is functionality that's clearly core to a file and table format: encodings (lance-encoding), the file reader/writer (lance-file), the manifest and fragment model (lance-table), object store I/O (lance-io), and index structures.

At the other end is functionality that looks a lot like a query engine:

• Scanner — filter and projection pushdown, limits, vector/FTS/hybrid search planning (dataset/scanner.rs, ~16k lines)
• SQL parsing and planning (dataset/sql.rs, lance-datafusion/src/sql.rs)
• Merge-into, update, and delete as full operations with their own logical and physical plans (write/merge_insert, write/update.rs)

I don't want to argue for a particular end state in this thread. What I do want to put on the table is that scope has a maintenance cost, and I think we're paying it:

• We have a very large volume of PRs, and they are a mix of both. To review them, one has to know a lot about query engines and about the fundamental design of the format.
• We ship a full query engine—DataFusion—in our binaries and Rust dependency tree. This is a heavy dependency for a query engine to take on just to support a table format.
• The two largest source files in the repo are query-layer files (scanner.rs and merge_insert.rs, ~16k and ~15k lines with tests). That's also where a lot of the churn and breakage concentrates. I can't help but feel these need a major refactor anyways.
• Meanwhile several low-level crates — lance-encoding, lance-file, lance-io, lance-select, lance-arrow — have no DataFusion dependency at all. The gradient already half exists.

There's a "what is Lance for" question underneath this too. There are already a lot of good query engines: DuckDB, DataFusion, Spark, Trino, LanceDB. I'm skeptical that Lance should try to be another one, when the thing only Lance can provide is the format and the primitives to read it well. delta-kernel-rs is one example of the alternative scope. It contains engine-agnostic building blocks, leaving planning and execution to the engines.

But I know the high-level APIs within Lance, including pylance, get a lot of use. "Narrow the scope" might mean different things depending on which APIs are your main entrypoints to Lance. Whatever we do has to start from real use cases rather than from an architecture diagram. So I'd like to hear:

• What do you use from the high-level surface today — Scanner, SQL, merge insert, update/delete? What would it look like to use a downstream engine (DuckDB, Spark, etc.) for that instead?
• If you integrate Lance with another engine, where do you wish you could plug in? What do you currently have to reach into internals to get?
• Where would you draw the line between core and high-level?

Nothing is being removed and no decision has been made. This is the start of the conversation.

[wjones127]

In discussing the different layers, we found it was really helpful to provide engine-agnostic implementations of operations. Good example today: CompactionTask - engine agnostic, easy to use. Could be used in multi-threaded or distributed contexts just as easily. Bad example: Scanner - tied to DataFusion - needs decoupling between Index operations and DataFusion physical nodes. I'll expand on this more soon.

[wkalt]

To me the naming of lance/lancedb suggests that lance is the data file format and lancedb is the database implementation built on top. As noted above however there is leakage. Lance includes engine functionality that could be separated from the format, and lancedb does not include a good interface for third-party engine integration. You cannot easily integrate lancedb into a custom engine at the datafusion layer (low-level) or the SQL layer (high level), so you are forced to use lance. This leads to confusion because we end up supporting two fairly-similar looking ways to solve a given problem. Datasets and tables can be substituted in many cases, and for users who land first on datasets and later need to transition to tables/lancedb for some reason, the transition is an unnecessary inconvenience that results from this unclear separation of concerns.

If with time we tend toward putting more engine functionality in lancedb instead of lance and begin to scope lance more to format essentials and immediate consequences, then direct lance consumers would begin to lose out because they would need to re-implement that functionality in their own engines.

For those who are integrating with external engines, would doing so via a SQL interface in lancedb be easier? Alternatively, would it be disqualifying because you need a lower level of control?

[wjones127]

To me the naming of lance/lancedb suggests that lance is the data file format and lancedb is the database implementation built on top.

Yes, that is correct. Note that—despite the name—this PMC doesn't control LanceDB and is governed independent of it. Our current stance is to treat Spark, DuckDB, and similar engines on equal footing here.

From a governance perspective, I'm not sure I'd want to put LanceDB as the integration point for engines with Lance unless LanceDB was donated to this organization. Would it be good for the Lance project if most integrations actually went through a library that was controlled by a third party? I'm not so sure.

[zhangyue19921010]

Really glad to see this discussion — at the pace things are moving, it's the right time to revisit the architecture.

Let me ground this in a fairly typical model-training pipeline we run, since you asked for real use cases rather than architecture diagrams. It spans both ends of the gradient you described: distributed Lance-Spark / Lance-Ray operations at the top, and low-level Lance-Core APIs underneath.

What we use the distributed engines for

append, update columns, and add columns — all at scale. This part works well today, and notably it works because the fragment-level primitives are engine-agnostic: write_fragments / FileFragment.create_from_file / merge_columns produce FragmentMetadata, and the engine commits them via LanceOperation::Append / Merge / DataReplacement. This is exactly the CompactionTask shape you called out as the good example — a task object the engine schedules, not a plan Lance executes. For us this is the strongest argument that the engine-agnostic direction is workable: it's already proven in the write path.

What we fall back to pylance for

Fine-grained retrieval and point lookups, specifically:

1. FileFragment.take for batched sample debugging — pulling specific rows out of a known fragment.
2. BTree-backed point lookups for small-sample debugging.
3. Building and optimizing ZONEMAP indices.

The pattern is: bulk write and scan go through distributed engines; selective lookups come back to the pylance SDK — not by preference, but because the engine integrations don't expose these paths with acceptable performance.

[wjones127]

@yanghua @timsaucer @eddyxu As PMC members, would welcome your input here.

Also @LuciferYang for your perspective as a Spark PMC member and someone who's worked on the MergeInsert implementation.

[LuciferYang]

@wjones127 Thanks for pinging me, and for framing it as core-vs-high-level. That's exactly the tension we've been living with.

The team I work on runs a distributed multimodal data-preparation pipeline (image/video plus large binary payloads, tables with hundreds of millions of rows across many fragments) on Ray + Lance, built directly on Lance's primitives. I've also worked on the MergeInsert implementation and come from the Spark side, which shapes how I read the boundary.

What the framework layer uses. Our framework layer (the Ray integration) rides entirely on the engine-agnostic, task-object primitives: workers call write_fragments / merge_columns / update_columns / create_index(fragment_ids=...) and hand back FragmentMetadata, and the driver does a single commit through LanceOperation::{Append, Overwrite, Merge, Update, CreateIndex} with an explicit read_version. The distinction that matters for the scope question: at this layer we drive the fragment-level column primitives directly. Our column-upsert path, for instance, is built on update_columns and committed as LanceOperation::Update. We never touch the higher-level, query-planned operations (dataset.update(...), dataset.delete(...), merge_insert), and we use Scanner only for column projection, filter pushdown, fragment scoping, and row-id/count reads, never its SQL planning.

Where we do reach higher. At the application layer we use the higher-level surface where it fits: merge_insert for keyed updates, a scalar (BTREE) index for point lookups, and an external engine (DuckDB) over the Arrow data for ad-hoc SQL rather than an in-format one. That matches what @zhangyue19921010 described: bulk writes through the fragment primitives, selective work through higher-level ops. The query-planning layer is something we'd rather compose and swap than bake into the format.

What internals we reach into. On the older Lance version we run in production we reached into a handful of internals; checking against current Lance, several are already closed: the dataset is picklable, and version enumeration and storage-option accessors are public. The one still open for us is mapping a row we've read back to its owning fragment. There's no public resolver for that, so we depend on the row-address layout, which also stops holding once row ids are stable rather than address-based. A public row-to-fragment resolver, correct under both schemes, would remove the coupling. It's partly a consequence of how we distribute the work, so it may be more our concern than a general one.

Where I'd draw the line. The Spark DataSource V2 analogy is close: DSv2 separates the physical Write / BatchWrite (workers produce commit messages, the driver commits them atomically) from the catalog and query engine, and that boundary has held up across many data sources. Lance's task-object plus driver-side commit(read_version=...) is the same shape, and I'd treat it as the durable core: engine-agnostic write/commit and fragment-level primitives on one side; Scanner with its SQL planning and the merge-into / update / delete logical+physical plans on the other, as high-level surface engines should be able to compose or replace. That a full distributed framework rides on the write/commit + fragment core (plus basic scan reads) is, to me, the clearest signal of where that core sits.

On merge_insert specifically. Since that's why you tagged me: I'd group it with the Scanner decoupling you already flagged rather than treat it as separate. The commit at the end is already the engine-agnostic task object (a LanceOperation); what runs as DataFusion logical and physical plans is the match and merge exec, and that's the part worth pulling out. The shape I'd want: expose the underlying primitive for the common keyed case (which target rows a key set matches) so an engine can build the join and the write itself, while merge_insert stays as a composable default. We rely on that turnkey op at the application layer, so this is about opening up the primitive underneath, not removing anything. To wkalt's point: the bulk write/commit path needs that lower level of control, so a SQL interface in LanceDB wouldn't cover it.

[yanghua]

I don’t think the boundary can be described simply as “format in Lance, query execution elsewhere.” In Lance-Spark’s normal local scan path, Spark owns query-level planning, partition planning, and distributed scheduling, while each Spark input partition invokes Lance’s native Scanner for a specific fragment. That Scanner builds and executes a DataFusion physical plan locally.

The local plan is not merely a generic filter-and-project pipeline. It enforces Lance-specific behavior around index coverage, fallback for unindexed fragments, overlay-stale index rows, prefilter eligibility, refinement, deletion visibility, and late materialization. The code therefore suggests that asking each engine to reproduce this behavior would create duplicated correctness-sensitive implementations.

What appears to be missing is a stable engine-neutral boundary between connector-level planning and the DataFusion-specific physical plan. Today Lance-Spark maintains its own representation of resolved snapshots, fragment splits, zonemap pruning, statistics, and pushed versus residual predicates, while the JNI boundary reconstructs a Rust Scanner from a large option set. A serializable Lance-owned planning/task protocol could make that boundary explicit. DataFusion could continue as the native reference executor without being exposed in the protocol’s public types.

I would also distinguish scheduler-agnostic from dependency-neutral. CompactionTask is serializable and designed for execution on remote workers, but it remains part of the DataFusion-dependent lance crate. MergeInsert, meanwhile, combines join planning, duplicate semantics, semantic and commit retries, execution-dependent write-mode choices, and transaction conflict resolution, so it should not be assumed to decompose into the same task model without additional mutation and transaction abstractions.

Suppose we want to change the situation. My preference would therefore be to retain a Lance-owned native runtime, define an engine-neutral planning/task/commit boundary around it, and let host engines own global relational planning and distributed scheduling.

[wjones127]

The local plan is not merely a generic filter-and-project pipeline. It enforces Lance-specific behavior around index coverage, fallback for unindexed fragments, overlay-stale index rows, prefilter eligibility, refinement, deletion visibility, and late materialization. The code therefore suggests that asking each engine to reproduce this behavior would create duplicated correctness-sensitive implementations.

I agree with this. We discussed a bit about this at the sync meeting, and it was clear that the Lance library does need to provide the logic for this. I agree that we can't rely on the engines to reproduce all of this on their own. The question is whether we can make this dependency-neutral or at least scheduler agnotistic.

One guiding principal we should have is that Lance should provide the tricky logic to make sure tables are read and written correctly. We shouldn't rely on individual library authors to be Lance experts.

A serializable Lance-owned planning/task protocol could make that boundary explicit. DataFusion could continue as the native reference executor without being exposed in the protocol’s public types.

Knowing this all needs to be (a) serializable and (b) more stable is helpful. I think those are good requirements to keep in mind.

In delta-kernel=rs, what they did is had executors be totally pluggable but provided a default engine that could be used. They take it to an extreme where it's just Arrow and Tokio, no DataFusion. I don't think we have to go that far. But we can make DataFusion the default engine, but try to make it optional when possible.

MergeInsert, meanwhile, combines join planning, duplicate semantics, semantic and commit retries, execution-dependent write-mode choices, and transaction conflict resolution, so it should not be assumed to decompose into the same task model without additional mutation and transaction abstractions.

Merge insert is a good one to talk about, because it is complex. I'm tempted to say that the engines themselves should handle a lot of the first part of the query (joins, index lookups), but the write part should be handled by Lance. We could provide an abstraction that is basically an engine agnostic version of FullSchemaMergeInsertExec.