Skip to content

Fix module inference for compound extensions and tie-break order (#9099) - #9109

Open
zhengdaodeguL wants to merge 1 commit into
huggingface:mainfrom
zhengdaodeguL:fix/load-extension-inference-tar-parquet-9099
Open

zhengdaodeguL wants to merge 1 commit into
huggingface:mainfrom
zhengdaodeguL:fix/load-extension-inference-tar-parquet-9099

Conversation

@zhengdaodeguL

Copy link
Copy Markdown

Fixes #9099

infer_module_for_data_files_list had two bugs:

  1. Compound extensions miscounted: it counted every component after the first dot, so data.tar.gz contributed .tar and was inferred as the webdataset module — even though the tarball's contents are never inspected and may be parquet/CSV/JSON. Now .tar only counts when it is the final suffix, so data.tar.gz yields no inferred module while data.csv.gz still correctly infers csv (and data.tar still infers webdataset).

  2. Reversed tie-break: the docstring promises that in a draw ".parquet is the favorite, and then alphabetical order", but the sort used reverse=True, so the alphabetical tie-break ran backwards (a.csv + b.json picked json). The sort key is now ascending with parquet as the explicit favorite, matching the docstring. Count precedence (most common extension wins) and metadata-file exclusion are unchanged.

Thanks to PCTEJA for reporting both issues with a precise root-cause analysis.

Verification

  • 30 new parametrized cases in tests/test_load.py covering compound extensions (data.tar.gz → None, train.csv.gz → csv, data.parquet.tar.xz → parquet, zip:// URLs), tie-breaks (csv/json → csv, arrow/lance → arrow, json/parquet → parquet), count precedence (2×json beats parquet), and metadata exclusion. All 46 inference tests pass; the 9 behavior-changing cases fail on base (verified with a source-only base control).
  • Targeted suites: 104 passed; ruff lint/format clean.

…gingface#9099)

Only infer WebDataset from a final .tar suffix, preserving known data
suffixes inside compound extensions. Resolve format ties by preferring
parquet and then alphabetical order while retaining count and metadata
precedence.

Thanks to PCTEJA for reporting both issues in huggingface#9099.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant