Repository navigation
Benchmark reciprocal rank fusion against the current hybrid fusion #1748
Description
Summary
Hybrid search merges its full-text and vector legs with fused = max(f, v) + 0.3 × min(f, v) (FUSION_FORMULA_VERSION = "max+0.3*min/v1", repository/search_reader.py). Evaluate reciprocal rank fusion (RRF) against it on the retrieval benchmarks, and adopt RRF only if it measures better.
This is deliberately kept out of the #1746 search rework. That rework changes cost, not ranking, so its before-and-after benchmarks must match exactly. Changing fusion at the same time would mix a quality change into a performance change, and a recall shift could not be attributed to either.
Why look at it
- The current formula mixes two scales. Full-text scores are divided by the query's best full-text score, so the top lexical hit is always 1.0 however weak it is. Vector scores are raw cosine similarity, about 0.5 to 0.95. A weak keyword match can outrank a strong semantic one, which overlaps with Hybrid search and the similar-notes advisory return weak vector matches #1718 (weak vector matches in hybrid).
- RRF ranks by position only,
score = Σ 1 / (k + rank)withk ≈ 60, so the scales stop mattering. It also discards magnitude: a barely related vector hit at rank 1 counts as much as a near-exact one.
Candidates
- The current formula (baseline).
- RRF with
k = 60, plus a small sweep ofk. - The current formula with calibrated full-text scores, for example a fixed bm25 / ts_rank normalization in place of the per-query maximum.
- Any of the above with the Hybrid search and the similar-notes advisory return weak vector matches #1718 weak-match floor applied.
Measure
test-int/test_search_performance_benchmark.py::test_benchmark_search_quality_recall_by_mode(hit@1, recall@5, MRR@10 for hybrid)just test-semantic-report(semantic quality and reranker quality)benchmarks/LoCoMo suite: recall@5/10, MRR,content_hit(just bench-run-longwithBM_LOCAL_PATH)
Run with the reranker off and on, since a reranker reorders the fused prefix and can mask fusion differences.
Acceptance
Adopt a new formula only if it does not regress any mode on any benchmark and improves hybrid hit@1 or MRR. Bump FUSION_FORMULA_VERSION when it changes.
Related
- Rework search: rank by id in the database, fetch text only for the returned page #1746 search rework (keeps the current fusion)
- Hybrid search and the similar-notes advisory return weak vector matches #1718 hybrid search returns weak vector matches
- Add a rerank stage to search: ~half of LoCoMo benchmark misses are ranking failures, and there is ~20x latency headroom #950 reranking
Activity
claude commentedon Oct 11, 2026
Claude finished @phernandez's task in 22s —— View job
Triage
enhancementapplied)repository/search_reader.py) plus the benchmark suites. This isn't a Cloud issue.test_benchmark_search_quality_recall_by_mode,just test-semantic-report, the LoCoMobenchmarks/suite). Each needs runs with the reranker off and on, and the candidates need aksweep and the optional Hybrid search and the similar-notes advisory return weak vector matches #1718 floor.Duplicates: I found none. The related issues are #1746 (search rework), #1718 (weak vector matches in hybrid) and #950 (reranking, closed). The issue already scopes this separately from #1746 so that ranking changes don't get mixed into the performance benchmarks.
Notes for whoever picks this up:
FUSION_FORMULA_VERSIONand any score thresholds assume before adopting it.No questions are blocking. I did triage only and made no code changes.
vitoramaral10 commentedon Nov 5, 2026
A real-world datapoint from a non-English knowledge base, in case it helps the evaluation.
Setup: basic-memory 1.0.0, SQLite + sqlite-vec, 167 notes (~10k vector chunks) written in Portuguese, embeddings
bge-m3(1024 dims) through an OpenAI-compatible server, reranker off, defaultsemantic_min_similarity(0.55).Probe: 6 paraphrased questions whose answer note shares few or no words with the question, plus 4 exact-name lookups. A hit means the expected note is in the top 3.
max+0.3*min/v1)fuse())In the cases where semantic mode found the answer, hybrid with the current formula followed the text-mode ranking instead. This is the effect described above: the top FTS hit is normalized to 1.0 even when it only matched common words, while the vector score stays raw cosine. Portuguese questions carry many function words that
unicode61keeps, so FTS matches are noisy and the normalized top hit wins almost every time.The sample is tiny and is not a benchmark. I'm posting it because the effect looks larger on non-English corpora, so a non-English set in the benchmark might be worth including. The patch replaced the fused-score line with
Σ 1 / (k + 1 + rank)over both legs; nothing downstream depended on the score scale.