Delta to Iceberg conversion loses the field ids of map and list children under column mapping

[slachiewicz]

Search before asking

I searched the open and closed issues; nothing covers this.

Describe the problem

Converting a Delta table that has column mapping enabled and contains a map or array column produces an Iceberg table that can't be read, once the Delta writer is 3.x. Reading it fails in Iceberg's Parquet reader:

java.lang.IllegalArgumentException: [col-127c1280-..., key_value, key] required binary key (STRING) is not in the store: [...]

ITConversionController#testColumnMappingEnabledDeltaToIceberg covers this path and passes on Delta 2.4, which the build still uses. It fails against Delta 3.3, which is what surfaced this.

Root cause

Delta 3.x writes Parquet field IDs; Delta 2.4 doesn't. The difference is visible in the footer of a file written by the same code against each version, for a table with delta.columnMapping.mode = name:

delta-core 2.4.0     optional group col-0284b2db-... (MAP) { repeated group key_value { ... } }
delta-spark 3.3.3    optional group col-d660aa27-... (MAP) = 2 { repeated group key_value { ... } }

XTable's Delta to Iceberg path relies on Iceberg's name mapping, which IcebergConversionTarget#createAndSetNameMapping populates with Delta's physical names. Iceberg applies a name mapping only to files that carry no field IDs at all. Delta 3.x's IDs therefore switch Iceberg to ID-based resolution, and Delta assigns column mapping IDs to struct fields only, never to a map key, a map value, or a list element. Those nested fields end up with no ID in the file and no name mapping to fall back on, so they can't be resolved. A map key is required, so the read fails rather than returning nulls.

Delta's own answer is delta.enableIcebergCompatV2, which hooks DeltaParquetWriteSupport specifically to, in its words, "write nested field IDs for list and map types to the parquet schema". With that enabled the IDs are in the file, and Delta records them per field under delta.columnMapping.nested.ids, keyed by the path the child takes in the file, for example col-1234.key.

What the protocol says

PROTOCOL.md defines delta.columnMapping.nested.ids under "Writer Requirements for IcebergCompatV2", so Delta writes it only when delta.enableIcebergCompatV2 is set. Under plain column mapping, in mode=id and mode=name alike, a list's element and a map's key and value have no identity mechanism at all, and those Parquet nodes carry no field_id.

Writing field_id for every StructField is a general column mapping requirement rather than a mode-gated one; the modes differ only in how a reader resolves the IDs. That matches the footer diff above: the 3.x file has IDs on the struct fields, and Iceberg switches to ID-based resolution as soon as any ID is present.

There is no delta.columnMapping.nested.physicalNames key. A collection node's physical path is its nearest ancestor StructField's physicalName plus .element, .key, or .value.

What this is not

Ruled out by experiment:

• Iceberg 1.10.2 in place of 1.9.2.
• Parquet 1.13.1 in place of 1.15.2.
• An ID or name bug on the XTable side. The Delta schema JSON and the ID-to-physical-name map that IcebergSchemaExtractor builds are identical between Delta 2.4 and 3.3.

Suggested fix

The two cases need different answers, and only the first is a carrying problem:

1. IcebergCompatV2 enabled. The IDs exist in the file. DeltaSchemaExtractor reads delta.columnMapping.nested.ids and puts those IDs on the converted map key, map value, and list element fields. IcebergSchemaExtractor already prefers a field's own ID over one it generates, so the Iceberg schema then matches the file. [911] Carry Delta's nested field ids into the converted schema #912 covers this.

2. IcebergCompatV2 disabled. There is nothing to carry. Any ID in the generated Iceberg schema for a collection node would have to be invented, which is the condition that produces the unreadable table. On Delta 3.x such a table isn't convertible to a readable Iceberg table when it contains collections, so the open question is whether XTable refuses the sync, warns at sync time, or documents the limit.

   The Spark 3.5 step carries a characterization test for the current behavior, ITConversionController#testColumnMappingWithoutIcebergCompatIsNotReadableAsIceberg. It builds a column mapped table without IcebergCompatV2, syncs it to Iceberg, asserts the sync reports SUCCESS, and asserts that reading the resulting Iceberg table throws. Refusing the sync has the advantage that the failure is otherwise silent until someone reads a collection column, which is what that test pins. If the project prefers to refuse or warn, that test is the place that changes.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

This issue was created with AI assistance.