Skip to content

Fix silent float-to-int truncation in Dataset.from_dict (#9102) - #9107

Open
zhengdaodeguL wants to merge 1 commit into
huggingface:mainfrom
zhengdaodeguL:fix/from-dict-int64-silent-truncation-9102
Open

zhengdaodeguL wants to merge 1 commit into
huggingface:mainfrom
zhengdaodeguL:fix/from-dict-int64-silent-truncation-9102

Conversation

@zhengdaodeguL

Copy link
Copy Markdown

Fixes #9102

Dataset.from_dict({"x": [1.9, 2.5]}, features=Features({"x": Value("int64")})) silently returned {'x': [1, 2]} — wrong data, no error. The library's own cast_array_to_feature path raises ArrowInvalid for the same values, so this was an inconsistency, not intended semantics.

Root cause: Value.encode_example used plain int(value) for integer features, which truncates floats. The fix routes Python and NumPy floats through Arrow's safe cast (pa.scalar(value).cast(...)), so fractional values raise ArrowInvalid while lossless conversions (1.0 → 1) still work. This covers from_dict, from_list, and nested features, since they share the same encoding path.

Thanks to PCTEJA for reporting the issue with a precise root-cause analysis.

Verification

  • New regression tests in tests/test_arrow_dataset.py: 10 cases (float lists, NumPy float arrays incl. float32, nested lists, nested dicts — via both from_dict and from_list) asserting ArrowInvalid on truncation, plus 3 control cases proving lossless conversions still work. All 13 pass; the 10 regression cases fail on base (verified with a source-only base control).
  • Targeted suites: 315 passed, 7 skipped; ruff lint/format clean.

…9102)

Use Arrow safe casts when encoding Python and NumPy floats as integer features so fractional values raise ArrowInvalid. Cover from_dict, from_list, nested features, and lossless conversions.

Reported by PCTEJA.

@1AbdullahButt 1AbdullahButt left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran Dataset.from_dict with features={"x": Value(...)} on main and on this branch (pyarrow 26.0.1, Python 3.11, in Docker).

input target main this PR
[1.9, 2.5] int64 [1, 2] (silent) ArrowInvalid
[1.0, 2.0] int64 [1, 2] [1, 2]
[nan] int64 ValueError ArrowInvalid
[300.0] int8 ArrowInvalid ArrowInvalid
[np.float16(2.5)] int64 [2] ArrowInvalid: Float value 16640 was truncated
[np.longdouble(2.0)] int64 [2] ArrowNotImplementedError: Unsupported numpy type 13
IterableDataset.from_generator yielding 1.9 int64 [{'x': 1}] ArrowInvalid

The main fix works. map, add_item, cast and from_pandas already raised on main, so this also makes from_dict and encode_example agree with them. The new tests pass: pytest tests/test_arrow_dataset.py -k "from_dict or float_to_int or safe_int" gives 17 passed here and 4 on main.

Two things I found:

  1. Regression for np.longdouble. pa.scalar() does not accept np.longdouble, so a whole-number value like np.longdouble(2.0) used to work and now fails. np.longdouble is a subclass of np.floating, so it goes into the new branch.
  2. Wrong value in the error for np.float16. pa.scalar(np.float16(2.5)) makes a half-float scalar, and the cast error prints its raw bits (16640) instead of 2.5. Whole-number float16 values still work.

Converting to a Python float first fixes both for me: pa.scalar(float(value)).cast(self.pa_type).as_py(). With that change, np.longdouble(2.0) gives 2 and np.float16(2.5) reports Float value 2.500000 was truncated. A np.longdouble and a np.float16 case in the new parametrized test would cover this.

(I made this review with help from Claude Code. I ran the table above on main and on this branch, the float(value) change in a local patch, and the filtered test_arrow_dataset.py tests. I did not run the full test suite.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dataset.from_dict() silently truncates floats to ints — 1.9 becomes 1 with no error, while the library's own safe cast raises

2 participants