Repository navigation
Fix behavioral eval cases that fail on unrelated PRs #2413
Description
Some behavioral eval cases fail or error on most PRs, including PRs that do not touch product code. The evals summary check goes red when any case fails or errors, so these cases make the check red on almost every PR. Reviewers then learn to ignore it.
Seen on #2371, #2374, #2376, #2377 and #2410:
evals/google/scheduling.eval.ts: "when a recurring slot is for two people and the requester's afternoons are holds, check both and propose a time". It errors withSTACK_TRACE_ERRORon almost every run, and sometimes hits the 60s idle timeout. fix(google): Let the requester be optional in findMeetingTimes #2375 fixed a relatedfindMeetingTimesbug, but not this error.evals/agent/delegation.eval.ts: "when a request splits into independent research tracks, start a child agent for each track". It often hits the 60s "agent not idle" timeout.evals/google/scheduling.eval.ts: "when the requester corrects a one-sided answer, check the other attendee without asking again". It sometimes hits the 60s timeout.evals/coding/subscriptions.eval.ts: "looks up and subscribes to an exact deployment before GitHub creates it". It fails on some runs.
For each case, find out whether:
- the product has a bug (fix the product),
- the eval checks wording or needs the model to do something wrong (fix the eval, following
policies/evals.md; do not rewrite the user's message to make it pass), or - the 60s idle limit is too short for a task that needs several tool calls (change the limit for that case, with a reason).
Do not skip or remove a case only to make the check green.
via David Cramer.
--
Activity
fix(evals): Assert the deployment watch, not the lookup step
fix(google): Drop unproven blockedBy result from findMeetingTimes
sentry-junior commentedon Oct 30, 2026
Recurring two-person slot: use case and proposed change
The use case. This case copies a real thread. The requester and Sam want a weekly 90-minute slot on Wednesdays and Fridays, early afternoon, to stream pairing. The requester blocks every afternoon with "DNS" (do not schedule) events so other people don't book over them. They are now booking into that time themselves. The requester still attends. Only their calendar is the wrong signal for this window. The real availability comes from what they said ("generally free 1–5pm"). Sam's calendar is the real constraint.
Why the tool doesn't fit it. Two gaps in
findMeetingTimes:optionalAttendees. That field means "people the requester calls optional". The requester is not optional, so the model has to say something false to get the right result. It did that in 1 of 178 calls. In the follow-up case, where the thread says outright that the blocks are placeholders, it does use the field and passes. My reading is that the name of the field is part of the miss. That is inference.maxResults: 3, the first results can all fall on Mon/Tue/Thu. The only way to scope to Wed/Fri is one call per day. That fits the 4–6 retries per run that end at the 60 s limit.Proposed change (one PR in
junior-google, required-by-default unchanged):holds: string[]: attendees who said the busy times on their calendar in this window are holds they can move. They stay attendees and keep their working hours. Their busy times don't block a slot. Slots that avoid them come first. A slot that overlaps them lists the person inholdConflicts. Remove the "put the requester here" line fromoptionalAttendees, so that field only means optional people.weekdays: ("Mon"…"Fri")[]: only return slots on these days.findMeetingTimestest intests/calendar.test.tsfor both inputs. No new eval case and no change to the user message. Proof is a few live runs of this case on the branch that propose 1:30–3:00 PM PT within 60 s.Out of scope: making one time fit every week. The model can still compare the Wed and Fri results.
fix(evals): Stop child research after the delegation decision
fix(evals): Give delegation children fixed docs pages
fix(evals): Stop delegation children before they call the model
fix(evals): Let delegation children research fixed short docs pages
fix(evals): Cover NOT NULL and INSTANT limits in the delegation docs …
fix(evals): Serve each mocked docs page only at its own URL
fix(evals): Mock the SQLite docs page without www too
fix(evals): Give the delegation docs pages a few paragraphs again
fix(evals): Fix two flaky behavioral evals (#2420)
ref(google): Replace holds with stated availability in findMeetingTimes
fix(google): Count findMeetingTimes window on the local clock
ref(google): Drop statedAvailability from findMeetingTimes
feat(google): Add a weekdays filter to findMeetingTimes and fix the 1…