Skip to content

Cast dictionary-encoded storage such as pandas Categorical columns in ClassLabel - #8833

Open
mertkayacs wants to merge 4 commits into
huggingface:mainfrom
mertkayacs:fix-classlabel-dictionary-storage
Open

mertkayacs wants to merge 4 commits into
huggingface:mainfrom
mertkayacs:fix-classlabel-dictionary-storage

Conversation

@mertkayacs

@mertkayacs mertkayacs commented Oct 17, 2026 •

Copy link
Copy Markdown

ClassLabel.cast_storage handles pa.StringArray and pa.IntegerArray and hands anything else to array_cast(storage, int64). A pandas Categorical column arrives as a pa.DictionaryArray (dictionary<values=large_string, indices=int8>), so Arrow tries to parse the category names as integers:

ArrowInvalid: Failed to parse string: 'positive' as a scalar of type int64

This breaks Dataset.from_pandas(df, features=Features({"label": ClassLabel(...)})) and load_dataset("parquet", ..., features=...) for a categorical label column. Without features both work, because array_cast to a plain string type lets Arrow decode the dictionary itself; ClassLabel is the one feature with its own cast_storage that did not.

cast_storage now accepts a dictionary array. For a dictionary of label names it maps each category that occurs once (pc.unique on the indices) and takes the ids by index with pc.take, so a null index stays null and a category no row uses is not looked up (pandas keeps unused categories after filtering). Any other dictionary value type (label ids, or a float dictionary that array_cast already converted on main) is decoded and continues through the existing integer path, so nothing that cast before raises now. The Union in the signature and the docstring list the new input.

The plain str column of a pandas 3 DataFrame arrives as large_string, which cast_storage does not map either (#8681); #8688 fixes that on adjacent lines of the same function. The two changes are independent and merge cleanly: with this PR alone the Categorical column works and the str column still fails, with both every pandas label column casts.

I left array_cast and cast_array_to_feature in table.py alone: the other features with a cast_storage take struct or binary storage, where a dictionary does not occur, and the generic Value path already decodes dictionaries through Arrow's own cast.

Verification

$ .venv/bin/python -m pytest tests/features/test_features.py -q -n 4   # Python 3.12.3, pyarrow 26.0.0, pandas 3.0.6
208 passed, 3 skipped in 2.56s
$ .venv/bin/python -m pytest tests/test_arrow_dataset.py tests/test_table.py -q -n 6 -k "class or pandas or cast or label"
18 passed (test_arrow_dataset.py), 66 passed (test_table.py)
$ ruff check src/datasets/features/features.py tests/features/test_features.py && ruff format --check src/datasets/features/features.py tests/features/test_features.py   # ruff 0.17.1
All checks passed!
2 files already formatted
# not run: the rest of tests/

With features.py back at 18378f7 and the tests kept, the same test_features.py run gives 1 failed, 207 passed, 3 skipped: test_classlabel_cast_storage_from_dictionary fails on the first cast_storage call with the ArrowInvalid above. Follow-up after review: the name path is now gated on a string value type (string, large_string, string_view) and the test covers pd.Categorical([1.0, 0.0]), which cast to [1, 0] on main and raised ValueError: Invalid string class label 1.0 on the previous head of this branch; test_features.py 208 passed, 3 skipped, the -k "class or pandas or cast or label" selection 84 passed, 2 skipped. The end-to-end part of the test (Dataset.from_pandas with a categorical column and a ClassLabel feature) fails the same way on main and passes with the fix; I also checked load_dataset("parquet", ..., features=...) on a Parquet file written from the same DataFrame.

Fixes #8830

@mertkayacs

Copy link
Copy Markdown
Author

A second look at my own change turned up a regression: the first version sent every non-integer dictionary down the label-name path, so a float dictionary that array_cast converted on main started raising. The name path is now gated on a string value type and a test pins the float case. I also noted #8688 in the body, which fixes the plain large_string label column on adjacent lines of the same function and merges cleanly with this.

@mertkayacs

Copy link
Copy Markdown
Author

This still fails on current main without the fix. Could a maintainer approve the CI run? My other open PRs (#8834, #8835, #8837, #8839, #8848) are waiting on the same approval.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ClassLabel.cast_storage fails on a pandas Categorical column (dictionary-encoded Arrow storage)

1 participant