Skip to content

fix(variant): preserve nulls for missing shredded list indexes - #11052

Open
emecii wants to merge 1 commit into
apache:mainfrom
emecii:fix/variant-get-shredded-list-null-11050
Open

fix(variant): preserve nulls for missing shredded list indexes#11052
emecii wants to merge 1 commit into
apache:mainfrom
emecii:fix/variant-get-shredded-list-null-11050

Conversation

@emecii

@emecii emecii commented Sep 10, 2026

Copy link
Copy Markdown

Which issue does this PR close?

Rationale for this change

An out-of-bounds list lookup and a present Variant null both produce null child columns after take. Without preserving index validity separately, shredded input turns missing list positions into valid Variant nulls, unlike unshredded input.

What changes are included in this PR?

  • Carry the list index validity mask through shredded path traversal.
  • Apply that mask to the output while preserving explicit Variant null elements as valid values.
  • Update list-like coverage and add a regression test distinguishing an empty list, an explicit null element, and a typed element.

Are these changes tested?

Yes.

  • cargo test -p parquet-variant-compute
  • cargo clippy -p parquet-variant-compute --all-targets --all-features -- -D warnings
  • cargo fmt --all -- --check

Are there any user-facing changes?

Yes. variant_get now returns an Arrow null for an out-of-bounds lookup on shredded list-like values, matching unshredded input semantics.

AI usage

OpenAI Codex assisted with investigating the issue, generating the implementation and tests, running the listed local validation, and drafting this PR description.

@github-actions github-actions Bot added the parquet-variant parquet-variant* crates label Sep 10, 2026
@sdf-jkl

sdf-jkl commented Sep 10, 2026

Copy link
Copy Markdown
Member

I'll take a look tomorrow

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

parquet-variant parquet-variant* crates

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Variant] variant_get returns Variant null instead of SQL NULL for out-of-bounds access on shredded lists

2 participants