Skip to content

Ignore backup extensions when discovering data files - #9110

Open
codingfeng-fufu wants to merge 1 commit into
huggingface:mainfrom
codingfeng-fufu:fix-data-file-extension-filter-9100
Open

codingfeng-fufu wants to merge 1 commit into
huggingface:mainfrom
codingfeng-fufu:fix-data-file-extension-filter-9100

Conversation

@codingfeng-fufu

Copy link
Copy Markdown

What this fixes

Fixes #9100. When Datasets discovers files in a directory, previous.csv.bak can be selected as a CSV file because the extension filter matches every dot-separated component. If the backup still contains valid CSV, its stale rows can be included without an error.

Check the final filename extension. If it is a recognized single-file compression extension, also check the preceding data extension. This keeps formats such as .csv.gz and .parquet.xz discoverable.

Validation

  • Added CSV, Parquet, and ZIP regression cases. All three failed on the main resolver at 39fd698d and passed with the patch.
  • Ran the local file-discovery tests from tests/test_data_files.py with the patched main module and installed Datasets 4.4.1 dependencies. Result was 170 passed, 1 skipped, 56 deselected in 0.66s. The skipped test requires symlinks on Windows. The selection excludes Hub and external fixtures.
  • Ran load_dataset(directory) with an ordinary CSV, a gzipped CSV, and a CSV backup. The original resolver loaded [1, 2, 999]. Substituting the patched resolver loaded [1, 2], including the gzip record.

OpenAI Codex assisted with investigation, implementation, and local validation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

resolve_pattern() allowed_extensions filter accepts intermediate suffixes — c.parquet.tmp and b.csv.bak pass as parquet/CSV data files

1 participant