Skip to content

Handle empty text arrays in normalize_bytes2str - #847

Open
AryanRG wants to merge 1 commit into
google:mainfrom
AryanRG:fix-empty-text-normalization
Open

AryanRG wants to merge 1 commit into
google:mainfrom
AryanRG:fix-empty-text-normalization

Conversation

@AryanRG

@AryanRG AryanRG commented Oct 30, 2026

Copy link
Copy Markdown

enp.normalize_bytes2str(np.empty((2, 0, 3), dtype='S8')) raises ValueError because np.vectorize cannot infer an output type from an empty input. This also affects filtered batches with no remaining text values.

Return a Unicode cast for empty text arrays, preserving their shape. The regression tests cover byte, Unicode and object arrays, filtered batches, scalar arrays, UTF-8 decoding and numeric inputs.

Validated the new tests and nearby utility tests with NumPy 1.26.4 and 2.5.4.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant