Conversation
TermFieldReader() and prepareField() used to hand rv.postings[i]/ rv.iterators[i] back in as a same-shaped "prealloc" hint on every lookup, but a recycled reader's shape is frequently wrong (a mix of 1-hit and general-encoded terms on the same field), so the hint was often discarded and allocated over anyway. Both call sites now release a value they're done with via releasePostings() -- a type assertion against the new, optional segment.Releasable capability (scorch_segment_api bitpack-simd branch), the same opt-in idiom as segment.OptimizablePostingsIterator -- and fetch a fresh one with prealloc left nil, letting the segment implementation's own pool decide what to hand back. Values that don't implement Releasable (e.g. the unadorned/synthetic iterators built by query optimization) make this a no-op, same as before. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Member
Author
|
Putting this off for a future release -- reverting the split/pooling approach in zapx. Closing rather than merging. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Part of the bitpack-simd effort (zapx blevesearch/zapx#447, scorch_segment_api companion PR): replaces the prealloc-passing pattern in
TermFieldReader()andprepareField()with pooled reuse.rv.postings[i]/rv.iterators[i]used to be handed back in as a same-shaped "prealloc" hint on every lookup, but a recycled reader's shape is frequently wrong (a mix of 1-hit and general-encoded terms landing on the same field), so the hint was often discarded and allocated over anyway.releasePostings()-- a type assertion against the new, optionalsegment.Releasablecapability -- on a value they're done with, and fetch a fresh one withpreallocleftnil, letting the segment implementation's own pool decide what to hand back.segment.Releasableis an opt-in capability, the same idiom as the existingsegment.OptimizablePostingsIterator: values that don't implement it (e.g. the unadorned/synthetic iterators built by query optimization) makereleasePostings()a no-op, unchanged from before.go.modbumped to thescorch_segment_apibitpack-simdbranch, which addsReleasableplusMergeablePostingsList/MergeablePostingsIteratoras optional interfaces (not required additions toPostingsList/PostingsIterator, so zapx v11-v17 keep compiling unchanged).Test plan
go build ./...,go vet ./...cleango test ./...cleango test -race ./index/scorch/...clean