Repository navigation
fix(memory): Rank each sentence of a request during recall - #2642
sentry-junior[bot] wants to merge 3 commits into
Conversation
Co-Authored-By: David Cramer <23610+dcramer@users.noreply.github.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Co-Authored-By: David Cramer <23610+dcramer@users.noreply.github.com>
A request with two asks scored the memory for one ask below the 0.55 recall threshold. The handoff integration eval has failed since #2585 for this reason: the reranker scored its gut-check memory 0.48 against the whole message and 0.86 against "Gut check" alone. Co-Authored-By: David Cramer <23610+dcramer@users.noreply.github.com>
sentry-junior
Bot
changed the title
|
this patch is awfully confusing and feels a bit like duct tape |
|
|
|
Closing unmerged: the handoff eval goes away with the handoff/turn router removal, so this fix is no longer needed. The multi-request recall gap from #2585 stays open. The full eval message scored 0.475 and "Gut check" scored 0.856, against the 0.55 cutoff. @thomasjiangcy owns that as its own fix, with a multi-request memory eval case. |
Fixes the integration eval
coding/handoff.eval.ts› "when the turn hands off, do the new request". It has failed on every shard-2 run since #2585.Cause. #2585 (
55c00b77e) admits a recall candidate only when the reranker scores it at least0.55against the whole user message (packages/junior-memory/src/recall.ts). The eval's message has two requests: "Switch to the other configured profile first. Gut check". I got livevoyage/rerank-2.5-litescores in CI with a temporary probe:So the memory never reached the prompt. In the failing transcripts the model reads files and edits code, which the memory forbids, and never adds the verdict reaction. The last green run replayed a recording made before #2585: it has no rerank request. The eval did not catch this on #2585 because the integration evals do not run for changes to
packages/junior-memory.Fix. When the message has more than one sentence (
Intl.Segmenter), recall also ranks the candidates against each of the first 5 sentences and keeps the best score per memory. The rerank calls run in parallel. The threshold and the single-sentence path do not change, so small-talk queries still score as they did in #2585's calibration.Tests. A new storage test uses the live scores above. It fails without the fix.
junior-memorytests (98) and typecheck pass. The PR has thetrigger-evalslabel so the integration and behavioral evals run.The branch has two temporary probe commits. The final diff does not include them; squash on merge.
via @dcramer.
--
View Session [Sentry]