Repository navigation
Fix silent float-to-int truncation in Dataset.from_dict (#9102) - #9107
zhengdaodeguL wants to merge 1 commit into
Conversation
…9102) Use Arrow safe casts when encoding Python and NumPy floats as integer features so fractional values raise ArrowInvalid. Cover from_dict, from_list, nested features, and lossless conversions. Reported by PCTEJA.
1AbdullahButt
left a comment
There was a problem hiding this comment.
I ran Dataset.from_dict with features={"x": Value(...)} on main and on this branch (pyarrow 26.0.1, Python 3.11, in Docker).
| input | target | main | this PR |
|---|---|---|---|
[1.9, 2.5] |
int64 | [1, 2] (silent) |
ArrowInvalid |
[1.0, 2.0] |
int64 | [1, 2] |
[1, 2] |
[nan] |
int64 | ValueError |
ArrowInvalid |
[300.0] |
int8 | ArrowInvalid |
ArrowInvalid |
[np.float16(2.5)] |
int64 | [2] |
ArrowInvalid: Float value 16640 was truncated |
[np.longdouble(2.0)] |
int64 | [2] |
ArrowNotImplementedError: Unsupported numpy type 13 |
IterableDataset.from_generator yielding 1.9 |
int64 | [{'x': 1}] |
ArrowInvalid |
The main fix works. map, add_item, cast and from_pandas already raised on main, so this also makes from_dict and encode_example agree with them. The new tests pass: pytest tests/test_arrow_dataset.py -k "from_dict or float_to_int or safe_int" gives 17 passed here and 4 on main.
Two things I found:
- Regression for
np.longdouble.pa.scalar()does not acceptnp.longdouble, so a whole-number value likenp.longdouble(2.0)used to work and now fails.np.longdoubleis a subclass ofnp.floating, so it goes into the new branch. - Wrong value in the error for
np.float16.pa.scalar(np.float16(2.5))makes a half-float scalar, and the cast error prints its raw bits (16640) instead of2.5. Whole-number float16 values still work.
Converting to a Python float first fixes both for me: pa.scalar(float(value)).cast(self.pa_type).as_py(). With that change, np.longdouble(2.0) gives 2 and np.float16(2.5) reports Float value 2.500000 was truncated. A np.longdouble and a np.float16 case in the new parametrized test would cover this.
(I made this review with help from Claude Code. I ran the table above on main and on this branch, the float(value) change in a local patch, and the filtered test_arrow_dataset.py tests. I did not run the full test suite.)
Fixes #9102
Dataset.from_dict({"x": [1.9, 2.5]}, features=Features({"x": Value("int64")}))silently returned{'x': [1, 2]}— wrong data, no error. The library's owncast_array_to_featurepath raisesArrowInvalidfor the same values, so this was an inconsistency, not intended semantics.Root cause:
Value.encode_exampleused plainint(value)for integer features, which truncates floats. The fix routes Python and NumPy floats through Arrow's safe cast (pa.scalar(value).cast(...)), so fractional values raiseArrowInvalidwhile lossless conversions (1.0→1) still work. This coversfrom_dict,from_list, and nested features, since they share the same encoding path.Thanks to PCTEJA for reporting the issue with a precise root-cause analysis.
Verification
tests/test_arrow_dataset.py: 10 cases (float lists, NumPy float arrays incl. float32, nested lists, nested dicts — via bothfrom_dictandfrom_list) assertingArrowInvalidon truncation, plus 3 control cases proving lossless conversions still work. All 13 pass; the 10 regression cases fail on base (verified with a source-only base control).rufflint/format clean.