Skip to content

fix(arrow-array): deprecate FixedSizeListArray::value_offset, which wraps past i32::MAX - #11070

Open
Cintu07 wants to merge 1 commit into
apache:mainfrom
Cintu07:fix/fixed-size-list-value-offset
Open

Cintu07 wants to merge 1 commit into
apache:mainfrom
Cintu07:fix/fixed-size-list-value-offset

Conversation

@Cintu07

@Cintu07 Cintu07 commented Sep 13, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

FixedSizeListArray::value_offset casts a usize offset to i32, so once a row starts past i32::MAX child values it returns a negative number. the avro FixedSizeList encoder cast that back to usize and panicked writing the batch.

What changes are included in this PR?

  • deprecate value_offset in favour of i * value_length() as usize, same as FixedSizeBinaryArray::value_offset in [arrow-array] use usize arithmetic in FixedSizeBinaryArray, aggressive overflow checks #9910, and panic instead of wrapping
  • the avro encoder uses idx * elem_len instead
  • take_value_indices_from_fixed_size_list computes the offset in usize and returns an error when it doesn't fit u32
  • the value_offset asserts come out of the tests, and the slice test checks row values instead

Are these changes tested?

yes. fixed_size_list_encoder_int32_sliced is new and covers the avro change. the overflow itself needs 384 MiB so it isn't a unit test, but the repro from the issue panics on 59.3.0 and writes the batch on this branch. arrow-array 972, arrow-select 446 and arrow-avro 515 tests pass, and clippy is clean.

Are there any user-facing changes?

FixedSizeListArray::value_offset is deprecated and panics past i32::MAX. take on a FixedSizeList returns an error past u32::MAX instead of wrapping.

…::MAX

value_offset narrows a usize offset to i32 with as, so past i32::MAX
child values it returns a negative number. The avro FixedSizeList
encoder cast that back to usize and panicked writing the batch.

Deprecate it in favour of i * value_length() as usize, as was done for
FixedSizeBinaryArray::value_offset, and panic instead of wrapping. The
avro encoder and take work the offset out in usize, and take returns an
error when it does not fit its u32 indices.

Closes apache#11059.
@github-actions github-actions Bot added arrow Changes to the arrow crate arrow-avro arrow-avro crate arrow-select arrow-array labels Sep 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

arrow Changes to the arrow crate arrow-array arrow-avro arrow-avro crate arrow-select

Projects

None yet

Development

Successfully merging this pull request may close these issues.

FixedSizeListArray::value_offset wraps past i32::MAX, and the avro writer panics on it

1 participant