Conversation
| return Ok(make_array(left_data)); | ||
| match left.data_type() { | ||
| DataType::RunEndEncoded(_, values) => { | ||
| if !values.is_nullable() && right.iter().any(|value| value == Some(true)) { |
There was a problem hiding this comment.
| if !values.is_nullable() && right.iter().any(|value| value == Some(true)) { | |
| if !values.is_nullable() && right.has_true() { |
There was a problem hiding this comment.
Done in 9ec63d5. The should-null bitmap is computed once (right.values() & validity), then this check uses BooleanBuffer::has_true() on that mask.
| } | ||
|
|
||
| /// Applies `nullif` to arrays that represent logical nulls in their children. | ||
| fn nullif_take(left: &dyn Array, right: &BooleanArray) -> Result<ArrayRef, ArrowError> { |
There was a problem hiding this comment.
perhaps we can reuse the computation we do here:
arrow-rs/arrow-select/src/nullif.rs
Lines 58 to 71 in b372d1f
instead of needing to check validity in this iterator
There was a problem hiding this comment.
Done. should_null is now computed once and reused by both the generic nullif path and nullif_take, so we no longer inspect is_valid/value per index.
There was a problem hiding this comment.
- same fixes as fix(arrow-select): preserve nullability for REE and Union take #10994? should we wait for it to land first?
There was a problem hiding this comment.
Yes — the take.rs changes here are a subset of #10994. I'll wait for that to land, then rebase this PR so it only contains the nullif.rs changes.
Compute the should-null bitmap once and reuse it for both the generic nullif path and the take-based REE/Union path. This avoids iterating BooleanArray validity in nullif_take and uses BooleanBuffer::has_true for the non-nullable REE check.
|
@Jefffrey on the relationship with #10994: This PR's
Those I'd like to wait for #10994 to land, then rebase this PR so it only contains the |
nullifcreates nulls by adding a top-level validity bitmap. This works formost arrays, but RunEndEncoded and Union arrays derive logical nullability from
their child arrays instead.
As a result,
nullifpreviously returned non-null values at positions wherethe condition was true for these array types.
Closes #10990
nulliffor RunEndEncoded and Union arrays totake, using nullindices for the positions selected by the condition.
represent the requested nulls.
indices.
takeneeds to represent a null index.cases.