[887] Add a Hudi backed secondary index for Iceberg tables - #935
Draft
vinishjail97 wants to merge 13 commits into
Draft
[887] Add a Hudi backed secondary index for Iceberg tables#935vinishjail97 wants to merge 13 commits into
vinishjail97 wants to merge 13 commits into
Conversation
Adds an `xtable.hudi.target.table_version` config (values 6 or 9, default 9) so the Hudi target can write the Hudi 1.x table format (timeline layout V2, column-stats index V2) instead of being pinned to version 6. Target changes: - HudiTargetConfig parses the new config from the target's additional properties; HudiTableManager.initializeHudiTable and the write config now honour the selected version (write version derives from the table itself). - Enable the column-stats index for all tables and disable the partition-stats index independently (hoodie.metadata.index.partition.stats.enable, added in apache/hudi#19111) so column stats work for partitioned external-file tables. - Select the timeline archiver by layout version (TimelineArchivers.getInstance) so version 9 uses the V2/LSM archiver; switch to the HoodieCleanStat builder. Source changes: - HudiConversionSource selects and orders instants by completion time on version 9 (timeline layout V2) and by requested time on version 6, so a commit that completes out of order relative to its requested time is no longer skipped during incremental sync. Tests: - Bump hudi.version to 1.3.0-SNAPSHOT to pick up apache/hudi#19111. - Parameterize TestHudiFileStatsExtractor over versions 6 (8 columns, decimal excluded) and 9 (9 columns, decimal present). - Add TestHudiTargetConfig and an out-of-order-completion incremental sync test on a version 9 source table.
…arget Column stats are now generated for partitioned Hudi targets (column-stats index enabled, partition-stats index disabled independently), so drop the `if (!partitioned)` guards that previously skipped the column-stats assertions for partitioned tables. The partitioned/non-partitioned parameterization is unchanged; the col-stats checks now run for both.
Source test tables already enable the column-stats index unconditionally and the suite passes with array/map schemas, so remove the leftover commented-out schemaContainsArrayOrMap guard and its stale apache#773 note.
…ble version 9 Re-enables the three conversion cases apache#772 disabled for the Hudi 1.x target: - Un-partitioned Paimon -> Hudi (ITConversionController and ITConversionService). BaseFileUpdatesExtractor now emits Hudi's external file-group-prefix format (Hudi PR #17788) for bucketed files instead of folding the "bucket-N" directory into the partition path: the file is registered under its true partition (empty for un-partitioned) with fileId "bucket-N/<file>" and the 3-arg marker "<file>_<commit>_fg%3Dbucket-N_hudiext". Applied consistently across the snapshot path, the diff path, and file-id derivation; non-bucketed sources are unaffected. - HUDI (partitioned on the nested column "nested_record.level") -> ICEBERG. This depends on the Hudi reader fix in apache/hudi#19123, so it will only pass once that fix is available in the Hudi snapshot the build resolves.
A savepoint instant reuses the requested time of the commit it pins, so orderByCompletionTimeAndDedup (the table version 9 / completion-time ordering path) dropped it when deduping the merged commit lists by requested time alone: putIfAbsent kept the data commit and silently discarded the savepoint. The version 6 path (mergeAndDedupLists) dedups by full instant equality, which includes the action, so it never had this problem. Include the action in the dedup key. The intended dedup (the same instant appearing in both the pending list and the newly-completed list) still collapses, but distinct actions sharing a requested time survive, and the version 9 backlog matches version 6: commit, savepoint (no-op), restore, commit.
… versions 6 and 9
Flip HudiTargetConfig.DEFAULT_TABLE_VERSION from NINE to SIX so the
default output stays readable by released Hudi readers; version 9
remains fully supported via xtable.hudi.target.table_version=9.
Parameterize the Hudi test suites so every run exercises both table
versions instead of only the default:
- ITHudiConversionTarget: partitioned x {SIX, NINE} via a MethodSource
cross-product; the target client sets the version through
HudiTargetConfig.HUDI_TABLE_VERSION.
- ITHudiConversionSource: source tables are created at {SIX, NINE} via
the table-type/partition MethodSource cross-products, and the
parameterized tests write through TestSparkHudiTable (Spark writer)
instead of the Java client.
- ITConversionController: combinations targeting HUDI are emitted once
per version; getTableSyncConfig gained an overload that applies the
version to the Hudi target properties.
- TestHudiTargetConfig/TestHudiConversionTarget assert against
DEFAULT_TABLE_VERSION instead of a hard-coded version.
Version 9 source coverage in ITHudiConversionSource depends on two
Hudi fixes validated against a locally patched 1.3.0-SNAPSHOT:
apache/hudi#19126 (column stats on map/array-nested leaves during MOR
log-append) and the savepoint backlog fix in the previous commit.
Hudi master moved two packages that XTable depends on, in the refactor(common) commits apache/hudi#19193 and apache/hudi#19195: - org.apache.hudi.stats -> org.apache.hudi.metadata.stats, covering ValueMetadata, ValueType and HoodieColumnRangeMetadata. XTable's XTableValueMetadata and its test live inside Hudi's package to reach package-private members, so the files move directories as well. - org.apache.hudi.avro.HoodieAvroUtils -> org.apache.hudi.common.avro. HoodieAvroUtils. The generated org.apache.hudi.avro.model classes did not move. This is required to build against a Hudi release that carries the reorganization, alongside the hudi.version bump this branch already makes.
Lets the Hudi conversion target build the Hudi record index and a secondary index for the files it registers, and adds HudiBackedIcebergSecondaryIndex, which syncs an Iceberg table to Hudi with those indexes enabled and resolves values of the indexed column to the data file, row position and partition that hold them. - HudiTargetConfig parses the secondary index column, the record index file group bounds, the secondary index parallelism and the execution engine. - HudiExecutionEngineProvider abstracts the Java and Spark engines used to commit and to run the table services; the Spark engine reuses the active Spark session. - HudiConversionTarget enables the indexes on the metadata table config and writes file ids with the plain string encoding, since the registered files keep their original names. - IcebergPartitionConverter converts the partition path of a matched file to the Spark InternalRow Iceberg exposes as _partition. Depends on apache/hudi#19869 for the record key generation of files without record keys and for the replace commit handling of both indexes.
vinishjail97
changed the title
2 tasks
Hudi 1.2.1 is planned to carry apache/hudi#19111, #19126, #19123 and #19869, which is everything this branch needs from Hudi 1.x. It does not carry the hudi-common package reorganization (apache/hudi#19193, #19195), so the imports move back to their 1.2.x locations.
HoodieCleanStat only became a Lombok builder on Hudi master; 1.2.x still exposes the seven argument constructor that main uses.
XTable's pom pins hudi.version to 1.2.1-SNAPSHOT, which no repository publishes, so the build fails at dependency resolution. Build only the Hudi modules XTable depends on from a pinned SHA of apache/hudi release-1.2.0 plus the changes proposed for 1.2.1 (apache/hudi#19111, #19126, #19123, #19869), cached on that SHA. Every step marked TEMPORARY, together with the raised timeout, must be removed once XTable depends on a released Hudi 1.2.1.
3 tasks
vinishjail97
force-pushed
the
hudi-secondary-index-iceberg
branch
2 times, most recently
from
September 9, 2026 06:30
1c7b0da to
29d998f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is the purpose of the pull request
Lets the Hudi conversion target build a Hudi record index and secondary index for the files it registers, and adds a
HudiBackedIcebergSecondaryIndexthat uses those indexes to resolve values of an Iceberg column to the data file, row position and partition that hold them. Engines can use the lookup to locate the rows touched by a merge or delete without joining against the whole table.The files XTable registers carry no Hudi record keys, so Hudi keys each row by
<file path relative to the table>_<row position>(apache/hudi#19869). A secondary index lookup therefore returns the file and the row position directly, which is what the index exposes asIndexLookupResult(mirroring Iceberg's_file,_posand_partitionmetadata columns).Brief change log
Hudi target configuration (
HudiTargetConfig)xtable.hudi.target.secondary.index.column: column to build the secondary index on. Enables the global record index as well, since the secondary index resolves through it. Requiresxtable.hudi.target.table_version=9.xtable.hudi.target.metadata.record.index.min.filegroup.count/...max.filegroup.count: file group bounds of the record index (must be set together).xtable.hudi.target.metadata.index.secondary.parallelism: parallelism of the secondary index generation.xtable.hudi.target.execution.engine:java(default) orspark. Building the record and secondary indexes reads every registered file, so the Spark engine is the practical choice for those.Execution engines (
org.apache.xtable.hudi.engine)HudiExecutionEngineProviderabstracts the engine context,HoodieTablecreation and the commit.JavaExecutionEngineProviderkeeps today's behaviour;SparkExecutionEngineProvidercommits throughSparkRDDWriteClienton the active Spark session.HudiConversionTargettakes the provider and the parsedHudiTargetConfig, enables the indexes on the metadata table config when a secondary index column is set, and sets_hoodie.writes.fileid.encoding=1because the registered file names are not UUID based Hudi file ids.hudi-spark-clientis added as aprovideddependency ofxtable-core, with its Lance and Vortex artifacts excluded because they are compiled against Spark 3.5; the Spark engine expects the Hudi Spark bundle on the classpath of the Spark application.Index API (
org.apache.xtable.index)Index<T>withdoesIndexExist,syncIndexandlookup.HudiBackedIcebergSecondaryIndexsyncs the Iceberg table to Hudi (table version 9, Spark engine, secondary index on the requested column) under<data location>/.hoodieand looks keys up withHoodieBackedTableMetadata#readSecondaryIndexDataTableRecordKeysWithKeys.IcebergPartitionConverterturns the partition path of a matched file into the SparkInternalRowIceberg exposes as_partition.Verify this pull request
This change added tests and can be verified as follows:
ITHudiBackedIcebergSecondaryIndex: builds the index for a partitioned and an unpartitioned Iceberg table, checks that every lookup resolves to the same file and row position as Iceberg's_file/_posmetadata columns and to the right partition value, that unknown keys return nothing, and that a second sync picks up newly added files.TestHudiTargetConfig: parsing and validation of the new properties.TestHudiConversionTarget: updated for the engine provider.TestIcebergPartitionConverter: type conversion of partition values.Run locally against a Hudi 1.3.0-SNAPSHOT built from apache/hudi#19869 (
-Dspark3.4 -Dscala-2.12): xtable-core unit tests andITHudiBackedIcebergSecondaryIndex,ITHudiConversionTarget,ITHudiConversionSourceandITConversionController(59 tests) pass.Limitations / follow-ups