Skip to content

Fix CSV round trips with declared timestamp features - #9112

Open
jeeth-kataria wants to merge 2 commits into
huggingface:mainfrom
jeeth-kataria:fix/csv-timestamp-schema
Open

jeeth-kataria wants to merge 2 commits into
huggingface:mainfrom
jeeth-kataria:fix/csv-timestamp-schema

Conversation

@jeeth-kataria

@jeeth-kataria jeeth-kataria commented Nov 20, 2026 •

Copy link
Copy Markdown

A CSV written by Dataset.to_csv() cannot be read back with a declared timestamp[us] feature on 39fd698. Pandas rejects datetime64[us] as a read_csv dtype before Arrow sees the data. This is the CSV reader case mentioned in #9082.

Read declared timestamps as objects and let the existing Arrow schema cast parse them. The regression exercises both public calls and checks microseconds, a missing timestamp, the feature schema, and leading zeros in a separate string column.

Verified on CPU: the regression failed before the change and passed afterward. tests/io/test_csv.py and tests/packaged_modules/test_csv.py: 52 passed. Ruff and git diff --check passed. Codex assisted with the implementation and tests.

Follow-up regression coverage in 30392ea also checks UTC microseconds and nanoseconds with nonzero sub-microsecond digits. Both failed on 39fd698 and passed on the PR. The assertions compare Arrow arrays directly, including the timestamp type and null. Both CSV test files now pass all 54 cases.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant