Repository navigation
feat(evals): Record and replay model calls with Roach - #564
Merged
Merged
Conversation
Evals send Anthropic and pi.dev traffic through Roach. A run with ROACH_TOKEN records misses, and a run without it only replays. Each test gets its own session, so parallel workers still work. The lockfile updates once @sentry/roach is on npm. Co-Authored-By: David Cramer <23610+dcramer@users.noreply.github.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
CI now runs evals on OpenRouter (#563). roach.json allowed only api.anthropic.com, so Roach refused OpenRouter calls and every pipeline eval stopped with "Provider unavailable". Allow and match the OpenRouter and OpenAI model APIs, and redact their keys.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 8669e7d. Configure here.
Warden mirrors WARDEN_<PROVIDER>_API_KEY onto the SDK name only when that name is unset. A run with a different native key would send a value that secretEnv did not list.
Roach learns and redacts every credential header on its own. Model API keys go only in headers, so listing their env vars adds nothing. Co-Authored-By: David Cramer <23610+dcramer@users.noreply.github.com>
Warden gives each finding a random XXX-XXX ID (generateShortId). The verifier request and every later call in a run include it, so those calls never replayed. Add a finding-id value pattern to each model rule. Co-Authored-By: David Cramer <23610+dcramer@users.noreply.github.com>
Pi auxiliary calls (the judge) put process.cwd() in the system prompt inside <cwd>. The checkout path differs between CI and local runs, so judge requests replayed in CI and missed locally. Add a value pattern for the packages/evals path so the key does not depend on the checkout. Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: David Cramer <23610+dcramer@users.noreply.github.com>
sentry-junior
Bot
force-pushed
the
feat/roach-evals
branch
from
November 5, 2026 03:00
0353aa6 to
39f3804
Compare
dcramer
added a commit
to getsentry/toolkit
that referenced
this pull request
Nov 6, 2026
The MCP evals (`packages/mcp-server-evals`) call OpenRouter on every run. This sends those calls through [Roach](https://github.com/getsentry/roach), Sentry's shared record/replay proxy. CI replays recorded calls and records only the new ones. The change is config only, in the same shape as getsentry/warden#564: - `roach.json` sets tenant `getsentry/toolkit`, allows `https://openrouter.ai`, and has one `model` rule for `POST https://openrouter.ai/api/v1/`. It has no `values` patterns: two CI runs sent byte-identical requests, and no miss came from a value that changes between runs. The API key travels in `Authorization`, which Roach redacts on its own, so there is no `secretEnv`. - `vitest.config.ts` adds the Roach global setup and setup file. The Roach setup file goes **before** `src/setup-env.ts`, so MSW, which throws on unhandled requests, does not catch Roach's own session calls. - `eval.yml` passes `ROACH_TOKEN` to the eval step and moves the job to Node 24, because Roach needs 24.14+. The workflow runs on `pull_request`, not `pull_request_target`, so fork PRs get no secrets and stay replay-only. The fork policy does not change. - `@sentry/roach` 0.3.0 is in the catalog, and in `minimumReleaseAgeExclude` because it was published today. Remove that entry once 0.3.0 is older than the minimum release age. 0.3.0 includes getsentry/roach#30 (Roach's session calls get past MSW) and #31 (`ROACH_MODE=off` runs on Node 22). The lockfile change touches only Roach entries. - `docs/testing/overview.md` has a short "Eval Recordings" section. **Record and replay in CI** Tested on unreleased Roach `main` before 0.3.0 shipped ([run 41970024566](https://github.com/getsentry/toolkit/actions/runs/41970024566)): | | Attempt 1 (records) | Attempt 2 (replays) | |---|---|---| | Replayed | 9 | 207 | | Live | 227 | 30 | | Missed | 0 | 0 | | Refused | 0 | 0 | | Evals passed | 108/121 | 109/121 | | Job duration | 5m57s | 3m12s | All 30 live calls in attempt 2 came from the tests that failed in attempt 1. Roach drops the recordings of failed tests, so an eval that keeps failing calls the model live on every run. The `ROACH_TOKEN` org secret already reaches this workflow (`***` in both runs). <!-- junior-request-attribution:start --> via @dcramer. <!-- junior-request-attribution:end --> <!-- junior-session-footer:start --> <!-- junior-conversation-id:slack%3AC0ACFA5JBDX%3A1793830221.324719 --> -- [View Session](https://junior-prod.sentry.dev/conversations/slack%3AC0ACFA5JBDX%3A1793830221.324719) [[Sentry]](https://sentry.sentry.io/explore/conversations/slack%3AC0ACFA5JBDX%3A1793830221.324719/?project=4510944073809921) <!-- junior-session-footer:end --> --------- Co-authored-by: sentry-junior[bot] <264270552+sentry-junior[bot]@users.noreply.github.com> Co-authored-by: David Cramer <23610+dcramer@users.noreply.github.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Evals now send their Anthropic and pi.dev traffic through Roach. A PR run that changes nothing about prompts or fixtures replays recorded model calls rather than paying for 63 live pipelines.
The Warden side is config only:
packages/evals/roach.jsonsets the tenant, the allowed hosts, and the secret env vars. It also lists the request values that change between runs: the temp eval repo path (/tmp/warden-eval-<name>-XXXXXX,local/warden-eval-...), commit SHAs, UUIDs, and ISO timestamps. I took these from a captured request body.vitest.config.tsadds@sentry/roach/vitestas the global setup and@sentry/roach/vitest/setupas a setup file. Each test gets its own session, so parallel workers stay parallel.evals.ymlpassesROACH_TOKENto the eval step. With the token, misses go live and get recorded. Without it, the run only replays.INTERNAL.mddocumentsROACH_TOKEN,ROACH_MODE=offfor local prompt work, andROACH_MODE=record.Uses
@sentry/roach0.2.0 from npm. The first CI run withROACH_TOKENrecords every call live. Runs after it replay.Fork PRs still skip evals. Replay-only runs for forks are a follow-up.
via @dcramer.
--
View Session [Sentry]