Return errors for undersized ArrayData validity buffers - #11044
Open
emecii wants to merge 1 commit into
Open
Conversation
Move validity-buffer bounds checking into the validated ArrayDataBuilder path before BooleanBuffer construction. Cover safe IPC decoding and streaming without constructing invalid Arrow arrays. Fixes apache#7124. Implementation and regression tests generated with OpenAI Codex.
albertlockett
approved these changes
Sep 10, 2026
albertlockett
left a comment
albertlockett
left a comment
Contributor
There was a problem hiding this comment.
Looks reasonable to me
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #7124.
Rationale for this change
Validated IPC decoding uses
ArrayDataBuilder, which constructs aBooleanBufferbefore checking whether the validity buffer is large enough. Malformed input can therefore panic instead of returning anArrowError.ArrayData::try_newalready performs the required check, but its direct builder callers do not benefit from it.What changes are included in this PR?
Move the existing bounds check into the builder's validated path, before
BooleanBufferconstruction. Keep the error wording, checked length-plus-offset arithmetic, and unsafe skip-validation behavior.Add builder regressions for short/empty buffers, offsets, overflow, explicit/inferred null counts and valid inputs. Add public record-batch decoder and
StreamReaderregressions using directly encoded malformed IPC, without constructing invalid Arrow arrays with unsafe code.Are these changes tested?
The short-buffer and overflow builder regressions and both IPC regressions fail with panics before the fix, then pass afterward.
Local CPU validation on Rust 1.98.0 / macOS aarch64:
cargo test --locked --workspace: 6,934 tests/doctests passed, 29 ignored.cargo test --locked -p <crate> --all-featuresruns forarrow-data,arrow-arrayandarrow-ipc: 1,200 passed, 3 ignored.arrow-dataandarrow-ipclibrary/integration tests: 201 passed.arrow-data/force_validateandarrow-array/force_validateenabled.-D warnings, formatting, Typos 1.37.0 and diff whitespace checks passed.An additional combined all-feature run across those three crates fails in six existing IPC tests that intentionally construct invalid arrays. The same six failures reproduce on unchanged main when
force_validateis unified into that test run; the separate commands used by CI pass.The existing IPC
StreamReader/read_10benchmark was exercised on main and this patch. Shared-host timings varied substantially between repetitions, so no performance conclusion is claimed.Are there any user-facing changes?
An undersized validity buffer now produces a recoverable error through the validated builder/IPC path instead of a panic. No public API signature changes.
Implementation and regression tests generated with OpenAI Codex.