Repository navigation
Fix CSV round trips with declared timestamp features - #9112
Open
jeeth-kataria wants to merge 2 commits into
Open
jeeth-kataria wants to merge 2 commits into
jeeth-kataria wants to merge 2 commits into
Conversation
Assisted-by: Codex
Assisted-by: Codex
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A CSV written by Dataset.to_csv() cannot be read back with a declared timestamp[us] feature on 39fd698. Pandas rejects datetime64[us] as a read_csv dtype before Arrow sees the data. This is the CSV reader case mentioned in #9082.
Read declared timestamps as objects and let the existing Arrow schema cast parse them. The regression exercises both public calls and checks microseconds, a missing timestamp, the feature schema, and leading zeros in a separate string column.
Verified on CPU: the regression failed before the change and passed afterward.
tests/io/test_csv.pyandtests/packaged_modules/test_csv.py: 52 passed. Ruff and git diff --check passed. Codex assisted with the implementation and tests.Follow-up regression coverage in 30392ea also checks UTC microseconds and nanoseconds with nonzero sub-microsecond digits. Both failed on 39fd698 and passed on the PR. The assertions compare Arrow arrays directly, including the timestamp type and null. Both CSV test files now pass all 54 cases.