Repository navigation
Write the encoding passed to to_csv - #8835
mertkayacs wants to merge 6 commits into
Conversation
|
Two small follow-ups: an empty dataset wrote a two-byte UTF-16 byte order mark where main wrote nothing, so the encoder's final flush now runs only after a batch was written, and the test checks the raw bytes for exactly one mark instead of relying on the round trip alone. #8834 touches the same lines, so whichever lands second I will rebase. |
|
I tested this branch at I pushed a commit you can cherry-pick after updating from main. It encodes the header with the configured encoder and returns zero bytes for |
|
Thanks, I could reproduce it: after merging main, every zero-row export failed with that TypeError, whatever the encoding or header setting. I merged main into the branch and cherry-picked your commit as it is, so you keep the authorship. The CSV suites pass locally, and UTF-16 still writes a single byte order mark. |
Dataset.to_csv(path, encoding="utf-16")wrote UTF-8.CsvDatasetWriter._batch_csvhands the batch to pandas withpath_or_buf=None, so pandas returns astrand never appliesencoding; the writer then encoded thatstrwith a hard-coded"utf-8". The docstring points atpandas.DataFrame.to_csvfor the keyword arguments, which documentsencodinganderrors, and both were accepted without effect.The writer now takes
encoding(defaultutf-8, as in pandas) anderrors(defaultstrict) out of the pandas keyword arguments and encodes with onecodecsincremental encoder for the whole file, in the parent process. One encoder, rather thanstr.encodeper batch, so that an encoding with a byte order mark writes it once at the start instead of before every batch; the batches themselves are still built in the worker processes whennum_proc > 1, and_batch_csvnow returns thestr. The default path writes the same bytes as before, and an empty dataset still writes an empty file: the encoder's final flush, which is where a byte order mark would otherwise land, runs only after at least one batch was written.JsonDatasetWriterkeeps its hard-coded UTF-8, sinceDataFrame.to_jsonhas noencodingargument.Verification
With
src/datasets/io/csv.pyback at 18378f7 and the tests kept,test_dataset_to_csv_encoding[1]and[2]fail withUnicodeError: UTF-16 stream does not start with BOM. The test now checks the raw bytes as well: the utf-16 file starts with the byte order mark and contains it exactly once, anddataset.select([])written withencoding="utf-16"giveswritten == 0and a 0-byte file (the previous head of this branch wrote the 2-byte mark alone).test_csv.py33 passed;tests/test_arrow_dataset.py -k "to_csv or to_sql"4 passed. A probe:encoding="ascii", errors="replace"writes?foré.Note: #8834 touches the same lines of
csv.pyandtest_csv.py; whichever of the two lands second gets a rebase from me.Fixes #8832