Repository navigation
fix(evals): Drop the session affinity header before Roach - #2636
Closed
sentry-junior[bot] wants to merge 1 commit into
Closed
sentry-junior[bot] wants to merge 1 commit into
sentry-junior[bot] wants to merge 1 commit into
Conversation
Since #2590, Junior sends the Conversation ID to AI Gateway in x-session-affinity. Roach takes each header whose name has "session" as a credential and redacts its value in the response that the client gets. A model call that names a child agent's Conversation ID then got <<redacted>>, so stopAgent failed with "<<redacted>> is not a child agent" in the subagents integration eval. The eval AI Gateway dispatcher now drops the header. AI Gateway only groups requests by it. Co-Authored-By: David Cramer <23610+dcramer@users.noreply.github.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
dcramer
approved these changes
Nov 4, 2026
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Since #2590, Junior sends the Conversation ID to AI Gateway in
x-session-affinity. Roach treats every header whose name containssessionas a credential (CREDENTIAL_HEADERingetsentry/roachsrc/secrets.ts). It redacts the header value in the response that the client gets (src/recorder.ts: "The client gets the redacted response too"). So when a model names a child agent's Conversation ID, the agent gets<<redacted>>instead.That is why
subagents.eval.ts› "when a child agent is no longer needed, stop it without a result turn" has failed on every run of integration shard 2 since #2590 landed. In run 41728176957, the model calledstopAgenttwice withconversation_id: "<<redacted>>", and both calls failed with<<redacted>> is not a child agent. A third call passed only because the model built the ID in codemode as"agent:" + "212aaafa…".The eval AI Gateway dispatcher now removes the header from requests. AI Gateway uses this header only to group requests, and
model-message-history.test.tsstill checks that the product sends it.Still open:
handoff.eval.ts› "when the turn hands off, do the new request" has also failed since the same range (4e14179..89f35a0). This change does not fix it. In the failing run, the reranker was called once, and after the handoff the model ranbashand added no reaction. The "gut check" memory forbids both. This suggests that the reranker threshold in #2585 (0.55) dropped the memory, but nobody has confirmed it. Confirming it needs a live rerank score for that query.via @dcramer.
--
View Session [Sentry]