Repository navigation
fix: Global - honor a UTF-16 byte order mark when no encoding is given - #1280
Open
aaronkanzer wants to merge 1 commit into
Open
aaronkanzer wants to merge 1 commit into
aaronkanzer wants to merge 1 commit into
Conversation
allotrope_from_file / allotrope_from_io assume UTF-8 when encoding is None, so a UTF-16 export (BOM FF FE or FE FF) fails with AllotropeParsingError unless the caller knows to pass an encoding. Three committed test fixtures fail this way through the public API (Vi-Cell BLU example01_utf16.csv, GENESYS On-Board Genesys150_ExampleFile..., VISIONlite Fixed_tab_sep.csv); the test harness only passes because it retries with chardet. A UTF-16 BOM can never decode as UTF-8, so honoring it only turns these failures into successes. The check covers decode() and the read_csv wrapper, and leaves a stream's position untouched (readers such as MSD Workbench consume a header line before calling read_csv).
ajcariaga16
approved these changes
Oct 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
allotrope_from_file/allotrope_from_ioassume UTF-8 whenencodingis None, so a UTF-16 export (BOMFF FE/FE FF) raisesAllotropeParsingErrorunless the caller passes an encoding. Three committed fixtures fail this way through the public API:beckman_vi_cell_blu/testdata/Beckman_Vi-Cell-BLU_example01_utf16.csvthermo_fisher_genesys_on_board/testdata/Genesys150_ExampleFile_Fixed_10_2_2024_1_57_42_PM.csvthermo_fisher_visionlite/testdata/Fixed_tab_sep.csvThe test harness passes them only because it retries with
chardetafter a UTF-8 failure, so the default path a user hits is not covered.A UTF-16 BOM can never decode as UTF-8, so honoring it only turns failures into successes. The check covers
decode()and theread_csvwrapper, and leaves stream position untouched (MSD Workbench reads a header line before callingread_csv; a version that rewound the stream brokemsd_workbench_test_2.csv, which is whybom_encodingreturns early when the stream is not at offset 0).Testing: full suite passes locally (1,254 tests,
tests/schema_genexcluded for missing optional dependencies in my environment), including 5 new tests intests/parsers/utils/encoding_test.py. black 22.3.0 and mypy 1.8.0 clean on the changed files. The three fixtures above now parse withallotrope_from_file(path, vendor)and noencoding.Open question: Latin-1 exports (for example the Agilent Gen5 Image fixtures, which contain a raw
0xB5micro sign) still needencoding="chardet". Would you take a follow-up that documents that option in the README, or one that falls back to it after a UTF-8 failure?