Skip to content

fix: Global - honor a UTF-16 byte order mark when no encoding is given - #1280

Open
aaronkanzer wants to merge 1 commit into
Benchling-Open-Source:mainfrom
aaronkanzer:fix/detect-utf16-bom-by-default
Open

aaronkanzer wants to merge 1 commit into
Benchling-Open-Source:mainfrom
aaronkanzer:fix/detect-utf16-bom-by-default

Conversation

@aaronkanzer

Copy link
Copy Markdown

allotrope_from_file / allotrope_from_io assume UTF-8 when encoding is None, so a UTF-16 export (BOM FF FE / FE FF) raises AllotropeParsingError unless the caller passes an encoding. Three committed fixtures fail this way through the public API:

  • beckman_vi_cell_blu/testdata/Beckman_Vi-Cell-BLU_example01_utf16.csv
  • thermo_fisher_genesys_on_board/testdata/Genesys150_ExampleFile_Fixed_10_2_2024_1_57_42_PM.csv
  • thermo_fisher_visionlite/testdata/Fixed_tab_sep.csv

The test harness passes them only because it retries with chardet after a UTF-8 failure, so the default path a user hits is not covered.

A UTF-16 BOM can never decode as UTF-8, so honoring it only turns failures into successes. The check covers decode() and the read_csv wrapper, and leaves stream position untouched (MSD Workbench reads a header line before calling read_csv; a version that rewound the stream broke msd_workbench_test_2.csv, which is why bom_encoding returns early when the stream is not at offset 0).

Testing: full suite passes locally (1,254 tests, tests/schema_gen excluded for missing optional dependencies in my environment), including 5 new tests in tests/parsers/utils/encoding_test.py. black 22.3.0 and mypy 1.8.0 clean on the changed files. The three fixtures above now parse with allotrope_from_file(path, vendor) and no encoding.

Open question: Latin-1 exports (for example the Agilent Gen5 Image fixtures, which contain a raw 0xB5 micro sign) still need encoding="chardet". Would you take a follow-up that documents that option in the README, or one that falls back to it after a UTF-8 failure?

allotrope_from_file / allotrope_from_io assume UTF-8 when encoding is None,
so a UTF-16 export (BOM FF FE or FE FF) fails with AllotropeParsingError
unless the caller knows to pass an encoding. Three committed test fixtures
fail this way through the public API (Vi-Cell BLU example01_utf16.csv,
GENESYS On-Board Genesys150_ExampleFile..., VISIONlite Fixed_tab_sep.csv);
the test harness only passes because it retries with chardet.

A UTF-16 BOM can never decode as UTF-8, so honoring it only turns these
failures into successes. The check covers decode() and the read_csv
wrapper, and leaves a stream's position untouched (readers such as MSD
Workbench consume a header line before calling read_csv).
@aaronkanzer
aaronkanzer requested review from a team and slopez-b as code owners October 20, 2026 17:04
@aaronkanzer
aaronkanzer requested a review from tamargrey October 20, 2026 17:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants