IN LIST: add UInt16 bitmap filter#23012
Draft
geoffreyclaude wants to merge 5 commits into
Draft
Conversation
This was referenced Jun 18, 2026
Replaces HashSet<u8> with a 32-byte stack-allocated bitmap. Provides O(1) membership testing via bit-shifting, significantly reducing memory overhead and improving cache locality. Triggers for UInt8 arrays.
48e938d to
55f3836
Compare
Implements an 8 KB heap-allocated bitmap for UInt16. Maintains O(1) performance while handling the larger value space. Triggers for UInt16 arrays.
55f3836 to
81ec379
Compare
This was referenced Jun 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
INperformance with specialized implementations #19390.Rationale for this change
#23011 uses a bitmap checklist for
UInt8, where there are 256 possible values.UInt16is the same idea with a larger value range: 0 through 65,535.That is still small enough to represent directly. A
UInt16bitmap needs one bit for each possible value:Then a lookup is still simple: use the input value as the bit position and check whether that bit is set. For example, if the list contains
42, bit42is set, and every input row with value42can be recognized with one bit test.This PR keeps the scope narrow: it adds the unsigned 2-byte bitmap path. Signed 1-byte and 2-byte types reuse these bitmaps in the next PR.
What changes are included in this PR?
UInt16BitmapConfig, backed by a heap-allocated 65,536-bit bitmap.UInt16constant-list filtering to the bitmap path.IN/NOT INnull behavior as the generic path.UInt16boundary values, sliced arrays, nulls, andNOT IN.Are these changes tested?
Yes.
cargo fmt --all --checkcargo test -p datafusion-physical-expr bitmap_filter_u16 --libcargo clippy -p datafusion-physical-expr --all-targets --all-features -- -D warningsAre there any user-facing changes?
No. This is an internal performance optimization only.
Benchmark note
No local
in_list_strategynumbers are included for this PR because the benchmark harness does not currently include a directUInt16case. The availablei16rows measure the signed reinterpretation path added in #23013, not this PR's unsignedUInt16bitmap filter.