Skip to content

Write the encoding passed to to_csv - #8835

Open
mertkayacs wants to merge 6 commits into
huggingface:mainfrom
mertkayacs:fix-to-csv-encoding
Open

mertkayacs wants to merge 6 commits into
huggingface:mainfrom
mertkayacs:fix-to-csv-encoding

Conversation

@mertkayacs

@mertkayacs mertkayacs commented Oct 17, 2026 •

Copy link
Copy Markdown

Dataset.to_csv(path, encoding="utf-16") wrote UTF-8. CsvDatasetWriter._batch_csv hands the batch to pandas with path_or_buf=None, so pandas returns a str and never applies encoding; the writer then encoded that str with a hard-coded "utf-8". The docstring points at pandas.DataFrame.to_csv for the keyword arguments, which documents encoding and errors, and both were accepted without effect.

The writer now takes encoding (default utf-8, as in pandas) and errors (default strict) out of the pandas keyword arguments and encodes with one codecs incremental encoder for the whole file, in the parent process. One encoder, rather than str.encode per batch, so that an encoding with a byte order mark writes it once at the start instead of before every batch; the batches themselves are still built in the worker processes when num_proc > 1, and _batch_csv now returns the str. The default path writes the same bytes as before, and an empty dataset still writes an empty file: the encoder's final flush, which is where a byte order mark would otherwise land, runs only after at least one batch was written. JsonDatasetWriter keeps its hard-coded UTF-8, since DataFrame.to_json has no encoding argument.

Verification

$ python -m pytest tests/io/test_csv.py -q   # Python 3.12, pyarrow 26.0.0, pandas 3.0.6
33 passed, 4 warnings in 1.10s
$ python -m pytest tests/test_arrow_dataset.py -q -k to_csv
2 passed, 4 warnings in 4.43s
$ ruff check src/datasets/io/csv.py tests/io/test_csv.py && ruff format --check src/datasets/io/csv.py tests/io/test_csv.py
All checks passed!
2 files already formatted
# not run: the rest of tests/

With src/datasets/io/csv.py back at 18378f7 and the tests kept, test_dataset_to_csv_encoding[1] and [2] fail with UnicodeError: UTF-16 stream does not start with BOM. The test now checks the raw bytes as well: the utf-16 file starts with the byte order mark and contains it exactly once, and dataset.select([]) written with encoding="utf-16" gives written == 0 and a 0-byte file (the previous head of this branch wrote the 2-byte mark alone). test_csv.py 33 passed; tests/test_arrow_dataset.py -k "to_csv or to_sql" 4 passed. A probe: encoding="ascii", errors="replace" writes ? for é.

Note: #8834 touches the same lines of csv.py and test_csv.py; whichever of the two lands second gets a rebase from me.

Fixes #8832

@mertkayacs

Copy link
Copy Markdown
Author

Two small follow-ups: an empty dataset wrote a two-byte UTF-16 byte order mark where main wrote nothing, so the encoder's final flush now runs only after a batch was written, and the test checks the raw bytes for exactly one mark instead of relying on the round trip alone. #8834 touches the same lines, so whichever lands second I will rebase.

@jeeth-kataria

Copy link
Copy Markdown

I tested this branch at c827696f combined with main at 39fd698d. The merge applies cleanly, but exporting a zero-row dataset raises TypeError: main’s empty-dataset path passes the string from _batch_csv straight to the binary output stream. Main now writes a header by default for a zero-row dataset.

I pushed a commit you can cherry-pick after updating from main. It encodes the header with the configured encoder and returns zero bytes for header=False. Tests cover default and custom headers, UTF-8 and UTF-16, the byte count, and a single UTF-16 byte-order mark. All 12 empty-export cases failed on the combined revision before the fix. With the fix, all 59 tests across both CSV suites passed locally. Codex assisted with implementation and verification.

@mertkayacs

Copy link
Copy Markdown
Author

Thanks, I could reproduce it: after merging main, every zero-row export failed with that TypeError, whatever the encoding or header setting. I merged main into the branch and cherry-picked your commit as it is, so you keep the authorship. The CSV suites pass locally, and UTF-16 still writes a single byte order mark.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

to_csv ignores the encoding argument and always writes UTF-8

2 participants