Lance Table Schema Semantic Type Contract

[Xuanwo]

Revision 2 (2026-09-23). Revised to address review feedback: the writer no longer normalizes representations before encoding, the design distinguishes representation-only types from value-transforming types, and it adds extension semantic types and an explicit enablement rule. The previous revision is in the edit history. Two open decisions are listed at the end.

Summary

This design makes Field.logical_type in the table schema name a column's semantic type: the value domain and query semantics that every engine must preserve. Arrow layout choices that do not change values (32- or 64-bit offsets, view layouts, dictionary encoding, decimal width) leave the logical type. The writer encodes whatever valid representation it receives. The reader returns the layout named by an advisory lance-schema:output-encoding field hint or by a request override. Only value-transforming types (json, blob) convert between input and storage. Applications can define their own semantic types as Arrow extension types on top of a core type. Readers that do not recognize an extension read it as its storage type, so new application types need no format vote.

Engines get a type system they can map without knowing Arrow encodings. Users can append LargeUtf8, Utf8View, or dictionary data to a string column, and a read returns the layout they wrote with. Lance can add view types and pick storage layouts without format changes. The contract is table-level only: data files, their encodings, and LanceFileVersion are unchanged.

Problem

logical_type currently maps one-to-one to an Arrow type, so a single string mixes three concerns:

• semantics: string, json, decimal precision and scale
• user-facing Arrow layout: string vs large_string
• storage strategy: dict:string:int16:false, decimal:128:10:2 vs decimal:256:10:2

This has concrete costs:

• Compatibility checks compare layouts, not values. Appending LargeUtf8 data to a string column fails even though every value fits.
• Every new layout needs a new type. Supporting Arrow view layouts (#5817) would add two more logical types, and engines that do not use Arrow would have to understand them. This design follows the consolidation direction of #5864.
• Value conversion has no single boundary. JSON is written as JSONB but supplied as text. In July 2026, five separate write and read paths were each fixed for skipping this conversion (fix: convert Arrow JSON to Lance JSON in single-fragment create path #7469, fix: return Arrow JSON from FragmentSession.take #7470, fix: return Arrow JSON from FileFragment.take #7471, fix: convert Arrow JSON when updating columns via merge/update path #7472, fix: convert Arrow JSON in merge-insert full-fragment-rewrite fast path #7473), after an earlier fix for nested JSON (fix: handle nested JSON conversion recursively #7060).
• New semantic types have no extension point. Proposed media types such as video, image regions, or audio segments would each need a new core type and a format vote.

Proposed Design

Semantic Types

A semantic type is a distinct type exactly when it changes the value domain, precision, or computation semantics. Layout and encoding choices never create a new type. By this rule, int32 and int64 remain distinct because width bounds the values. float and double remain distinct because width sets arithmetic precision. string and large_string are one type.

Core semantic types fall into three classes.

Unchanged types. These keep their current logical_type string, and each has exactly one Arrow representation: null, bool, int8…int64, uint8…uint64, halffloat, float, double, date32:day, date64:ms, time32:*, time64:*, duration:*, timestamp:*, struct, map, fixed_size_list:<elem>:<n> (including lance.bfloat16 elements), and fixed_size_binary:<n>.

Representation-only types. Several Arrow layouts hold the same values. The writer encodes any listed physical layout unchanged, and each data file may use a different one.

Semantic type (logical_type)	Parameters	Physical layouts in data files	Additional accepted input	Output encodings (default first)
string	none	Utf8, LargeUtf8, Dictionary<int*, Utf8 | LargeUtf8>	Utf8View	utf8, large_utf8, utf8_view, dictionary:<key>:<value>
binary	none	Binary, LargeBinary, Dictionary<int*, Binary | LargeBinary>	BinaryView	binary, large_binary, binary_view, dictionary:<key>:<value>
list	child field	List, LargeList	none	list, large_list
decimal:<p>:<s>	precision, scale	Decimal128(p, s), Decimal256(p, s)	none	smallest of decimal128 / decimal256 that holds p, then the other

Dictionary encoding is a layout of string or binary values, not a type. A view input is written as a non-view layout that holds every value; the writer chooses which one.

Value-transforming types. Input values are converted to a different stored representation, and output converts them back. Only these types need conversion logic in the writer and reader.

Semantic type	Accepted input	Physical layout	Output encodings (default first)
json	arrow.json over Utf8, LargeUtf8, or Utf8View (validated, then converted to JSONB); lance.json over LargeBinary (unchanged)	LargeBinary + lance.json (JSONB)	arrow.json (text), lance.json (JSONB)
blob	as defined by the Blob specification	Blob v1: LargeBinary + lance-encoding:blob. Blob v2: struct + lance.blob.v2	read modes defined by the Blob specification

This design only classifies blob. Its representations and read modes remain governed by the Blob specification.

Output Encoding

lance-schema:output-encoding is an optional field metadata entry that names the Arrow layout a read returns for that field. Its values are the output encodings listed in the tables above. For nested types, each child field carries its own entry. A reader picks the output layout in this order:

1. a per-request override from the caller
2. the field's lance-schema:output-encoding entry
3. the default output encoding of the semantic type

When a writer creates a column (table creation, add column, or alter type), it records the input Arrow layout as the field's output encoding if that layout differs from the default. This way a LargeUtf8 input reads back as LargeUtf8. Appends do not change the entry. Changing the entry is a metadata-only schema update.

The output encoding is advisory. It never affects which values are stored, schema compatibility, or correctness. A reader that does not recognize a value, or cannot produce that layout, uses the type default instead.

Extension Semantic Types

An extension semantic type is a core field that carries ARROW:extension:name and, optionally, ARROW:extension:metadata. The core type is its storage type. Example: logical_type = "fixed_size_list:float:4" with ARROW:extension:name = "example.bbox". A field carries one extension name, so an extension cannot yet be layered on a type that already uses one (json, Blob v2, lance.bfloat16 elements); see open decision 2.

• Value preservation: extension types annotate values and do not transform them. Stored values are exactly the storage type's values, so a reader that does not recognize the extension returns correct data of the storage type.
• Metadata preservation: implementations must keep extension metadata unchanged through schema evolution, compaction, and any other rewrite, even for extensions they do not recognize.
• Reserved prefix: names starting with lance. are reserved for Lance. A Lance-owned extension may transform values only if a format version or feature flag gates it; lance.blob.v2 requires data storage version 2.2. Other extensions should use a project prefix such as lancedb. or a reverse-DNS name.
• No flag: adding an extension semantic type requires no feature flag and no change to this specification.

Schema Compatibility

Two fields are compatible for append, merge, and update when their semantic types and semantic parameters are equal. Output encodings, physical layouts, and dictionary keys are not compared. Consequences:

• An append supplying any accepted input for the column's semantic type succeeds.
• An append supplying Decimal256(10, 2) to decimal:10:2 succeeds.
• An append supplying Decimal128(12, 2) to decimal:10:2 fails, because the value domain differs. Changing precision is a type change, not a compatible append.

Field IDs, nullability, and nested structure follow the existing rules.

Data Files and the Physical Schema

A data file's schema records the exact physical layout encoded in that file, using the existing Arrow-mapped strings (large_string, dict:string:int16:false, decimal:128:10:2, …). These strings keep their current meaning inside data files. The table schema is the only source of semantics: readers resolve semantics through field IDs in the table schema and never infer them from a data file's physical layout. The encodings, footer, page metadata, and global buffers of data files are unchanged.

Contract and Invariants

• Writer boundary: every array handed to the encoder has the exact Arrow type and extension metadata recorded for that field in the data file schema. Representation-only inputs satisfy this by passing through unchanged. Value-transforming inputs satisfy it after conversion, applied recursively through struct, list, and map children.
• Lossless conversion: conversions between accepted input, physical layout, and output are lossless. An operation that would lose data fails instead of truncating.
• Metadata only: output encodings and extension metadata never require rewriting data files.

Failure Semantics

Condition	Result
Input Arrow type is not an accepted input for the field's semantic type	The write fails before any file is written. The error names the field path, semantic type, and input type.
json input is not valid JSON	The write fails. The error names the field path and row.
An array reaches the encoder in a layout different from its declared physical layout	Internal invariant violation. The write fails with an error naming the field path, the expected layout and extension, and the actual ones.
The requested output layout cannot hold a value (for example, one utf8 value over the 32-bit offset limit)	The read fails with an error naming the field, the value size, and the requested encoding. The caller can override to large_utf8 or utf8_view. Readers may emit smaller batches to fit, but must never truncate.
A writer tries to set lance-schema:output-encoding to a value that is invalid for the field's type or parameters (for example, decimal128 for precision 40)	The schema update fails.
A reader encounters an unrecognized or unsupported lance-schema:output-encoding value	The reader uses the type's default output encoding.

Compatibility and Migration

Feature flag. A table that follows this contract sets a new feature flag, FLAG_SEMANTIC_TYPES, in both reader_feature_flags and writer_feature_flags. The spec PR assigns its bit. Once set, the flag remains set in all later versions, including restores.

Without the flag, older clients would misread these tables in ways that are hard to detect:

• They interpret string as exactly Utf8, ignore the output encoding, and return different Arrow types than the table specifies.
• They reject appends that this contract accepts.
• They cannot parse decimal:<p>:<s> at all (evidence: the current TryFrom<&LogicalType> requires four :-separated parts).

The flag turns all of these into one clear "unsupported" error.

Legacy tables (flag not set) keep their current behavior exactly. Each logical_type names one Arrow type, schema compatibility compares those types, and no output encoding applies. New readers do not rewrite legacy schemas.

Enablement. New tables created with data storage version 2.3 or later set the flag (open decision 1). Tables on earlier storage versions stay legacy unless explicitly upgraded.

Upgrade. Data files do not change, so upgrading an existing table is a single metadata-only commit. The commit rewrites each legacy alias to its canonical type plus the output encoding that reproduces the current read type, then sets the flag. After the upgrade, reads return the same Arrow types as before. There is no downgrade.

Legacy logical_type	Canonical type	Output encoding recorded
string, binary, list	same	none
list.struct	list (the child field is already struct)	none
large_string	string	large_utf8
large_binary	binary	large_binary
large_list, large_list.struct	list	large_list
dict:<value>:<key>:false	canonical type of <value>	dictionary:<key>:<value layout>
decimal:128:<p>:<s>	decimal:<p>:<s>	none (p <= 38, so the default is already decimal128)
decimal:256:<p>:<s>	decimal:<p>:<s>	decimal256 when p <= 38; none otherwise

Writers of flagged tables write only canonical names in the table schema. Readers of flagged tables still interpret a legacy alias as its canonical type plus the implied output encoding.

Scenario	Behavior
Any reader or writer, legacy table	Unchanged.
New writer, legacy table	Writes legacy strings; never sets the flag implicitly.
New reader or writer, flagged table	Follows this contract.
Older reader or writer, flagged table	Rejected by the feature flag.
Any reader, data files	Unchanged; data file schemas keep the Arrow-mapped vocabulary.

Validation

This contract is accepted when all of the following are observable:

• An upgraded legacy table reads back the same Arrow types, values, and field IDs as before the upgrade.
• Older clients reject flagged tables with the "unsupported" feature-flag error.
• On a flagged table, appends of Utf8, LargeUtf8, Utf8View, and Dictionary data to one string column succeed across fragments. Reads return the recorded output encoding, and request overrides change it.
• Nested json (struct<json>, list<struct<json>>, map<string, json>) writes JSONB and reads back as arrow.json through create, append, update, merge-insert, and take.
• An unrecognized extension type round-trips through a reader that does not know it, and its metadata survives compaction.
• The failure cases in the table above return the specified errors.

Open Decisions

1. Enable by default for new 2.3 tables, or require explicit opt-in. Recommendation: default-on. Storage version 2.3 is not yet stable (stable_file_version() is 2.2), and 2.3 data files already require new readers, so default-on adds no new compatibility break. Explicit opt-in only would leave most new tables on the legacy contract. Closing this needs PMC agreement.
2. How extensions compose with types that already use an extension name. Blob-backed media types such as video need this, because Blob v2 fields already carry ARROW:extension:name = "lance.blob.v2". Recommendation: record extension semantic types under a Lance-owned key, lance-schema:extension-name (plus lance-schema:extension-metadata). On output, the reader maps these keys to ARROW:extension:name and ARROW:extension:metadata only when the field has no Arrow extension of its own. The alternative, reusing only the Arrow slot, cannot express media types over Blob v2. Closing this needs agreement with #8739 (structured Blob v2 descriptor extensions) on where media metadata lives.

Implementation Notes

The spec change touches docs/src/format/table/schema.md and versioning.md. The implementation keeps semantic-type dispatch internal to the Rust core, with no public plugin trait. The write path gets a single boundary check, extending today's SchemaAdapter. Reads choose layouts through the existing ReaderProjection, which is designed to decode a column into another in-memory layout of the same semantic type when the decoder supports it; conversions a decoder lacks (for example, between decimal widths) run after decoding. The writer boundary check changes no format and can land before the spec vote. The upgrade commit is a small metadata rewrite and is planned for the first implementation, so existing tables can adopt view types and relaxed appends.

[Xuanwo]

This proposal is an extension from @wjones127's #5864

[westonpace]

+1, this is a great writeup (though it is a lot of information 😆). Is this a proposal or just discussion?

First, I agree with almost everything here. This would be a good system.

1. Is the table in Canonical Semantic Type meant to be complete? It's missing integer types, boolean, maybe others.

2. I think my biggest question is around the physical storage type. My initial thinking is that we don't need this. The file format can choose whatever storage representation it wants and it might even change from file to file.

I think you are arguing that we need to centralize on one type before we send it to the writer. For example, if the user has a string field they could provide string, large_string, string_view, and dictionary<i32, string> and we should be able to store it. If I understand correctly you are saying it is the table format's responsibility to pick one canonical physical type (e.g. string) before sending it to the writer.

However, what happens if the table format picks string and the user provides a large_string that is actually really large and cannot be cast to string? We would have to fail the write which doesn't seem good.

On the other hand, if we pick large_string as the canonical physical type (since we can always cast all four representations to that type) then we are doing a lot of unneeded casting most of the time.

Why can't we just give the writer whatever it is the user gave us? If they give us string we give that to the writer which may or may not dictionary encode it based on cardinality. If they give us large_string then the writer will bitpack the offsets as part of encoding so there's no benefit converting to string even if we could. If they give us string_view then the writer already knows how to do the cast. If they give us dictionary<i32, string> then the writer says "great, it's already dictionary encoded" and just stores it as dictionary encoded.

Or am I misunderstanding something here?

[Xuanwo]

Is this a proposal or just discussion?

Half and half. Maybe I should attach the context as a seperate docs for users.

Is the table in Canonical Semantic Type meant to be complete?

The Canonical Semantic Type table is meant as examples, not a complete list yet. I make that clearer or expand it.

I think my biggest question is around the physical storage type.

I don’t mean that the table format must pick one canonical physical Arrow type for each semantic type. The table schema should own semantics and hints; the writer/file encoder can choose the concrete physical representation. I think I need to tweak the wording a bit to avoid overemphasizing JOSNB's case, where I want to enforce large_string.

The key invariant is only that the file schema and arrays agree before encoding.

My initial thinking is that we don't need this. The file format can choose whatever storage representation it wants and it might even change from file to file.

Does this mean that the reader needs to perform more dynamic casts, rather than just the pre-defined ones?

My current understanding is:

user's schema is large_string, but during encoding, we find that those string are quiet small, we can just encode them as string, even some more compat forms.