Repository navigation
Cast dictionary-encoded storage such as pandas Categorical columns in ClassLabel - #8833
Open
mertkayacs wants to merge 4 commits into
Open
mertkayacs wants to merge 4 commits into
mertkayacs wants to merge 4 commits into
Conversation
Author
|
A second look at my own change turned up a regression: the first version sent every non-integer dictionary down the label-name path, so a float dictionary that |
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ClassLabel.cast_storagehandlespa.StringArrayandpa.IntegerArrayand hands anything else toarray_cast(storage, int64). A pandasCategoricalcolumn arrives as apa.DictionaryArray(dictionary<values=large_string, indices=int8>), so Arrow tries to parse the category names as integers:This breaks
Dataset.from_pandas(df, features=Features({"label": ClassLabel(...)}))andload_dataset("parquet", ..., features=...)for a categorical label column. Withoutfeaturesboth work, becausearray_castto a plain string type lets Arrow decode the dictionary itself;ClassLabelis the one feature with its owncast_storagethat did not.cast_storagenow accepts a dictionary array. For a dictionary of label names it maps each category that occurs once (pc.uniqueon the indices) and takes the ids by index withpc.take, so a null index stays null and a category no row uses is not looked up (pandas keeps unused categories after filtering). Any other dictionary value type (label ids, or a float dictionary thatarray_castalready converted on main) is decoded and continues through the existing integer path, so nothing that cast before raises now. TheUnionin the signature and the docstring list the new input.The plain
strcolumn of a pandas 3 DataFrame arrives aslarge_string, whichcast_storagedoes not map either (#8681); #8688 fixes that on adjacent lines of the same function. The two changes are independent and merge cleanly: with this PR alone theCategoricalcolumn works and thestrcolumn still fails, with both every pandas label column casts.I left
array_castandcast_array_to_featureintable.pyalone: the other features with acast_storagetake struct or binary storage, where a dictionary does not occur, and the genericValuepath already decodes dictionaries through Arrow's own cast.Verification
With
features.pyback at 18378f7 and the tests kept, the sametest_features.pyrun gives1 failed, 207 passed, 3 skipped:test_classlabel_cast_storage_from_dictionaryfails on the firstcast_storagecall with theArrowInvalidabove. Follow-up after review: the name path is now gated on a string value type (string,large_string,string_view) and the test coverspd.Categorical([1.0, 0.0]), which cast to[1, 0]on main and raisedValueError: Invalid string class label 1.0on the previous head of this branch;test_features.py208 passed, 3 skipped, the-k "class or pandas or cast or label"selection 84 passed, 2 skipped. The end-to-end part of the test (Dataset.from_pandaswith a categorical column and aClassLabelfeature) fails the same way on main and passes with the fix; I also checkedload_dataset("parquet", ..., features=...)on a Parquet file written from the same DataFrame.Fixes #8830