[611] RFC: Module Restructuring by the-other-tim-brown · Pull Request #612 · apache/incubator-xtable

the-other-tim-brown · 2024-12-30T15:30:08Z

What is the purpose of the pull request

Issue: #611
This is a proposal for how the project can be restructured to allow the user to only include the dependencies that they desire. It also contains a proposal for allowing the project to support multiple versions of a spec, like Iceberg v2 and v3.

Brief change log

Adds RFC

Verify this pull request

NA

the-other-tim-brown · 2024-12-30T15:31:01Z

+This will allow the user to easily add jars and the required dependencies when executing the standalone RunSync command as well as avoid issues with version differences when adding these jars to an existing class-path that may contain table format related dependencies already.
+In addition to this, the jars will still be published to allow users to compose their own shaded jar if they have specific needs or customizations that they want.
+
+We can also create these modules with a suffix indicating which version of the format is supported to allow for different implementations or versions of dependencies for the same format. For example, have an xtable-iceberg-v2 and xtable-iceberg-v3 module. 


@ashvina this is based off of our conversation, can you include some of the more practical items for why you would want this split?

the-other-tim-brown · 2025-01-06T19:53:41Z

One open question, should the existing xtable-hudi-support module be moved under the proposed xtable-hudi module?

ashvina · 2025-02-09T06:48:44Z

I like the proposal of adding the version number in Iceberg module. I think it may soon apply to Delta as well. Do you think it would be consistent to add the version proactively to all modules, or do you think it makes it verbose?

I am less concerned with how verbose it may be but more concerned around maintenance and how we will test these conflicting dependency versions in the future.

ashvina · 2025-02-09T07:00:55Z

+These conflicts may only surface as runtime errors when there is difference in method signature for example. 
+This can make the experience brittle for the end user by requiring them to closely follow any upgrades within the XTable repo as well.
+In this scenario the user will also have dependencies on all 3 table formats by default.
+


In addition to the footprint of the fat jar and the class conflicts, the root causes of most issues in production, another reason is the performance of XTable. Currently, bootstrapping XTable is slow. One of the reasons is the service providers that load all table formats even if they are not needed. Splitting the modules will also split the service providers, which in turn would skip loading all formats. Currently, they service providers are hardcoded, resulting in the unnecessary class loading and checks.

ashvina · 2025-02-09T07:07:47Z

+This can make the experience brittle for the end user by requiring them to closely follow any upgrades within the XTable repo as well.
+In this scenario the user will also have dependencies on all 3 table formats by default.
+
+## Implementation


The current implementation description outlines what users will need to do once the modules are created (thanks for adding it).

However, I anticipate the process of splitting the modules to be a bit involved task and won't be as simple as moving classes to new packages. There could be several sub-tasks, such as resolving test dependencies, moving any format classes in the API module, and more. It might be helpful to list these tasks here and create trackable sub-tasks that can be distributed among contributors.

Additionally, could you share how you imagine this change to take effect? For instance, clarify whether this change will happen all at once in one PR or incrementally in multiple PRs.

Here is the current draft: #618
It can be split up to pull out the test module into its own smaller change and then each format can also have its own PR after that.

vinishjail97

Added comments.

vinishjail97 · 2026-08-17T19:31:36Z

+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+# RFC-2: Module Structure


I think RFC-2 may already be taken? rfc/rfc-2/2 - Deletion Info Conversion.md landed in dc20996, after you opened this. Would you mind renumbering, and does the rfc/rfc-<n>/rfc-<n>.md layout work here too?

vinishjail97 · 2026-08-17T19:31:36Z

+
+## Implementation
+In order to allow users to include only the dependencies that they need for their use case, the repo will need to have modules per table format and per catalog client in the future.
+This will allow the user to include only the relevant modules at runtime to fit their needs. Furthermore, each of these sub-modules will publish a shaded jar that shades the format specific dependencies. 


Does "shades" here mean relocating packages, or just bundling them? I ask because xtable-utilities/pom.xml today has 480 <include> entries and no <relocation>, so I am not sure the existing bundle avoids the conflicts described in the Background.

If we do relocate, how would we handle the lookups that go through class-name strings, such as the payload class in hoodie.properties or Iceberg's catalog-impl? Would ServicesResourceTransformer cover the META-INF/services side?

My original plan was to shade the format dependencies as well so the user can run the conversion without needing to provide a long list of jars to run the code. If this is not possible given the table format limitations then we will still require the user to provide this list.

Even if that is the case, I think this RFC is still useful for users that are building their own bundled jars. Splitting out the formats into their own dependencies the user can import will help limit the transitive dependencies.

vinishjail97 · 2026-08-17T19:31:36Z

+## Implementation
+In order to allow users to include only the dependencies that they need for their use case, the repo will need to have modules per table format and per catalog client in the future.
+This will allow the user to include only the relevant modules at runtime to fit their needs. Furthermore, each of these sub-modules will publish a shaded jar that shades the format specific dependencies. 
+This will allow the user to easily add jars and the required dependencies when executing the standalone RunSync command as well as avoid issues with version differences when adding these jars to an existing class-path that may contain table format related dependencies already.


What does each format bundle include, and what stays provided? My worry is that if xtable-api is packaged (or relocated) per bundle, the ConversionTarget interface differs per bundle and ServiceLoader finds nothing. Is the intent that only third-party format deps are relocated?

Only the third-party dependencies. The idea here is that if the user has Iceberg 1.11 and we're on 1.6 then we risk some APIs changing and then we have some runtime errors. If we shade our version, then we know the APIs are as we expect.

vinishjail97 · 2026-08-17T19:31:37Z

+In this scenario the user will also have dependencies on all 3 table formats by default.
+
+## Implementation
+In order to allow users to include only the dependencies that they need for their use case, the repo will need to have modules per table format and per catalog client in the future.


On @ashvina's question about sub-tasks, would it help to call out the core-to-format imports that need inverting first? The ones I noticed are ConversionTargetFactory importing DeltaConversionTargetConfig and DeltaKernelConversionTarget, and CatalogUtils importing HudiCatalogPartitionSyncTool. Could those be separate PRs ahead of the module moves?

I think it is a bit detailed to include the movements in this RFC but yes, I think they can be separated out ahead of time.

vinishjail97 · 2026-08-17T19:31:37Z

+In order to allow users to include only the dependencies that they need for their use case, the repo will need to have modules per table format and per catalog client in the future.
+This will allow the user to include only the relevant modules at runtime to fit their needs. Furthermore, each of these sub-modules will publish a shaded jar that shades the format specific dependencies. 
+This will allow the user to easily add jars and the required dependencies when executing the standalone RunSync command as well as avoid issues with version differences when adding these jars to an existing class-path that may contain table format related dependencies already.
+In addition to this, the jars will still be published to allow users to compose their own shaded jar if they have specific needs or customizations that they want.


Today all four targets sit in one META-INF/services file in xtable-core. After the split, what does a user see if they ask for ICEBERG without that module on the classpath? Would it be worth committing to an error that names the missing jar, given ServiceLoader.hasNext() can throw ServiceConfigurationError?

The expectation is that an error is thrown and yes, we should improve the message so the remediation is clear.

vinishjail97 · 2026-08-17T19:31:37Z

+
+We can also create these modules with a suffix indicating which version of the format is supported to allow for different implementations or versions of dependencies for the same format. For example, have an xtable-iceberg-v2 and xtable-iceberg-v3 module. 
+
+![img.png](assets/images/xtable_dependencies.jpg)


Could the diagram be refreshed to include Paimon, Parquet, Kernel, xtable-aws, xtable-hive-metastore and xtable-service? Also, where do the two Delta paths land? delta-core pulls in Spark and Scala, Kernel does not, so does xtable-delta become one module or two?

For the Delta Table format, there will be two modules.

vinishjail97 · 2026-08-17T19:31:37Z

+This will allow the user to easily add jars and the required dependencies when executing the standalone RunSync command as well as avoid issues with version differences when adding these jars to an existing class-path that may contain table format related dependencies already.
+In addition to this, the jars will still be published to allow users to compose their own shaded jar if they have specific needs or customizations that they want.
+
+We can also create these modules with a suffix indicating which version of the format is supported to allow for different implementations or versions of dependencies for the same format. For example, have an xtable-iceberg-v2 and xtable-iceberg-v3 module. 


Does the suffix mean the format spec version or the library version? I ask because iceberg-core 1.9.2 already writes both spec v2 and v3, whereas Hudi 0.x vs 1.x genuinely cannot share a module (hence main-hudi-0x). Would it be simpler to add the suffix only when two library versions must coexist? That might also answer @ashvina's question above.

vinishjail97 · 2026-08-17T19:31:37Z

+## Rollout/Adoption Plan
+
+- Are there any breaking changes as part of this new feature/functionality?
+Yes, users will need to add new jars to their class-paths when running commands. Additionally, users may need to import different modules into their builds if they are constructing their own jar.


For users who upgrade in place and keep running the same bundle command, the failure would land at runtime rather than build time. Would it be worth publishing an all-formats bundle for a release or two, so RunSync keeps working and only users who want a small jar have to change?

I don't think so, I think those jars would be far too large. We can make the error messages useful and be sure to include this in the release notes.

vinishjail97 · 2026-08-17T19:31:37Z

+
+## Test Plan
+
+Describe in few sentences how the RFC will be tested. How will we know that the implementation works as expected? How will we know nothing breaks?


Could we fill in the Test Plan? A few things I would find convincing:

ITConversionController green against the split modules (related to Set delta kernel as the default path for 0.5.0 release #886).

The bundle validation flow you mentioned on dev@, running each bundle alone.

A run with a conflicting version of that format library on the classpath, since that is what proves the relocation claim.

Jar sizes before and after, given the size goal in the abstract.

Would it also be worth noting the merge order here, i.e. test module first, then one PR per format?

Yes, updating

vinishjail97 · 2026-08-17T19:31:37Z

+- @the-other-tim-brown
+
+## Approvers
+- @<approver1 github username>


A few nits: the Approvers placeholders and Status are unfilled, rfc/README.md referenced on line 31 does not exist yet, and the image path on line 58 renders broken on GitHub (rfc/rfc-1/rfc-1.md has the same issue). Missing newline at EOF too.

Proposal

25ea0a4

the-other-tim-brown commented Dec 30, 2024

View reviewed changes

update to show that table formats can use core for common logic

cdb0df9

the-other-tim-brown mentioned this pull request Jan 6, 2025

[DRAFT] Restructure Repo for Table Format specific modules #618

Draft

vinishjail97 added the rfc PR's proposing an RFC label Feb 5, 2025

ashvina reviewed Feb 9, 2025

View reviewed changes

vinishjail97 mentioned this pull request Aug 11, 2026

Cross-format indexing support in Apache XTable #887

Open

2 tasks

vinishjail97 reviewed Aug 17, 2026

View reviewed changes

the-other-tim-brown added 4 commits September 5, 2026 14:17

update based on feedback

a73b80c

Merge remote-tracking branch 'origin/main' into 611-modules-rfc

eae3d30

rename

033bde9

move image

b32e262


		We can also create these modules with a suffix indicating which version of the format is supported to allow for different implementations or versions of dependencies for the same format. For example, have an xtable-iceberg-v2 and xtable-iceberg-v3 module.

		![img.png](assets/images/xtable_dependencies.jpg)


		## Test Plan

		Describe in few sentences how the RFC will be tested. How will we know that the implementation works as expected? How will we know nothing breaks? No newline at end of file

Uh oh!

Conversation

the-other-tim-brown commented Dec 30, 2024

What is the purpose of the pull request

Brief change log

Verify this pull request

Uh oh!

Choose a reason for hiding this comment

Uh oh!

the-other-tim-brown commented Jan 6, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vinishjail97 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants