feat(evals): Record and replay eval model calls with Roach by sentry-junior[bot] · Pull Request #1972 · getsentry/toolkit

sentry-junior · 2026-11-05T01:55:35Z

The MCP evals (packages/mcp-server-evals) call OpenRouter on every run. This sends those calls through Roach, Sentry's shared record/replay proxy. CI replays recorded calls and records only the new ones.

The change is config only, in the same shape as getsentry/warden#564:

roach.json sets tenant getsentry/toolkit, allows https://openrouter.ai, and has one model rule for POST https://openrouter.ai/api/v1/. It has no values patterns: two CI runs sent byte-identical requests, and no miss came from a value that changes between runs. The API key travels in Authorization, which Roach redacts on its own, so there is no secretEnv.
vitest.config.ts adds the Roach global setup and setup file. The Roach setup file goes before src/setup-env.ts, so MSW, which throws on unhandled requests, does not catch Roach's own session calls.
eval.yml passes ROACH_TOKEN to the eval step and moves the job to Node 24, because Roach needs 24.14+. The workflow runs on pull_request, not pull_request_target, so fork PRs get no secrets and stay replay-only. The fork policy does not change.
@sentry/roach 0.3.0 is in the catalog, and in minimumReleaseAgeExclude because it was published today. Remove that entry once 0.3.0 is older than the minimum release age. 0.3.0 includes fix(client): Keep control calls out of in-process HTTP mocks roach#30 (Roach's session calls get past MSW) and Initial work on stdio #31 (ROACH_MODE=off runs on Node 22). The lockfile change touches only Roach entries.
docs/testing/overview.md has a short "Eval Recordings" section.

Record and replay in CI

Tested on unreleased Roach main before 0.3.0 shipped (run 41970024566):

	Attempt 1 (records)	Attempt 2 (replays)
Replayed	9	207
Live	227	30
Missed	0	0
Refused	0	0
Evals passed	108/121	109/121
Job duration	5m57s	3m12s

All 30 live calls in attempt 2 came from the tests that failed in attempt 1. Roach drops the recordings of failed tests, so an eval that keeps failing calls the model live on every run.

The ROACH_TOKEN org secret already reaches this workflow (*** in both runs).

via @dcramer.

--

View Session [Sentry]

vercel · 2026-11-05T01:55:40Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
sentry-local	Ready	Preview	Nov 5, 2026 7:57pm UTC

@dcramer

The Eval workflow on `main` reports green, but no evals have run. `eval:ci` fails at startup: ``` Failed to load custom Reporter from vitest-evals/reporter cause: Package subpath './reporters' is not defined by "exports" (ERR_PACKAGE_PATH_NOT_EXPORTED) ``` The cause is that vitest-evals 0.4.0 imports `vitest/reporters`, which vitest 5 removed. `continue-on-error` on "Run evals" hides the failure, so no `eval-results.json` gets written and the status check step throws. Example run: https://github.com/getsentry/toolkit/actions/runs/41951775819 This PR moves the eval package to vitest-evals 0.18.0, which supports vitest 5 (getsentry/vitest-evals#113, peer range `vitest >=5 <6`). The whole repo stays on vitest 5; there is no vitest 4 pin. ## Changes - **Bump `vitest-evals` to `^0.18.0`.** I added `vitest-evals@0.18.0` and its `@vitest-evals/core@0.18.0` and `@vitest-evals/report-ui@0.18.0` dependencies to `minimumReleaseAgeExclude`, because they were published today. Drop those entries once they are past the 24h window. - **Import from `vitest-evals/legacy`.** 0.18 keeps the scorer-first API (`describeEval` with `data`/`task`/`scorers`, `ToolCallScorer`, `Score`, `ScoreFn`, `BaseScorerOptions`) under `vitest-evals/legacy`. This is an import-path change only, in the 33 eval imports and the example in `docs/testing/overview.md`. The eval code is unchanged. - **Remove the unused root `vitest-evals` dev dependency.** Nothing at the root imports it. - **Set `SENTRY_CLI_NO_TELEMETRY=1` for the scorer's `sentry mcp` process.** - `ToolPredictionScorer` starts `pnpm --dir packages/cli run cli mcp` from source. On `main`, that process exits immediately with ``Package subpath './client' is not defined by "exports"``, and every scorer then fails with `MCPClientError: Connection closed`. - The exit comes from `lazyTelemetry()` in `src/lib/db/index.ts`. It loads `../telemetry.js` with `createRequire`, so its imports resolve `@sentry/api/client` under the `require` condition, but `@sentry/api` exports `./client` only for `import`. - Evals shouldn't send CLI telemetry anyway. The CLI bug itself is not fixed here: `pnpm --filter sentry run cli help` also fails from source. It needs its own CLI fix. `continue-on-error` and the eval status check are unchanged. ## Verification - **Local, no LLM key:** with no key in the sandbox, `pnpm eval:ci` on vitest 5.0.5 loads the reporter, collects and runs all 27 files and 121 evals, and writes `eval-results.json`. On `main` it stops at startup. Every eval failed, and the only cause was the missing provider key. No `Connection closed` errors. - **Checks:** `pnpm run tsc`, `pnpm run lint`, and `pnpm install --frozen-lockfile` pass. The full `pnpm run test` didn't finish inside my sandbox's time limit, so I'm relying on CI for it. ## Related - #1132 (draft, "Migrate to vitest-evals harness") moves these evals to the harness API. It touches the same eval files, so expect import conflicts there. This PR keeps the current scorer-first suite running until then. - #1972 (Roach recordings for evals) depends on this, because Roach records nothing while the suite doesn't run. Both PRs change `pnpm-lock.yaml` and `pnpm-workspace.yaml`, so #1972 needs a rebase after this merges.  via @dcramer.    -- [View Session](https://junior-prod.sentry.dev/conversations/agent%3A1de0f02c391e347294cc7ce28353bcfe) [[Sentry]](https://sentry.sentry.io/explore/conversations/agent%3A1de0f02c391e347294cc7ce28353bcfe/?project=4510944073809921)  Co-authored-by: sentry-junior[bot] <264270552+sentry-junior[bot]@users.noreply.github.com> Co-authored-by: David Cramer <23610+dcramer@users.noreply.github.com>

The MCP evals call OpenRouter on every run. Roach, Sentry's shared recording proxy, now sits in front of those calls: it replays a call it has a recording for, and records the rest when the run has ROACH_TOKEN. The setup is config only: roach.json names the OpenRouter host and one rule for its API, vitest.config.ts adds the Roach global setup and setup file, and the Eval workflow passes ROACH_TOKEN and moves to Node 24, which Roach needs. Fork PRs get no secrets, so they only replay. The Roach setup file loads before setup-env.ts so that MSW does not catch Roach's own control calls. That needs getsentry/roach#30. Co-Authored-By: David Cramer <23610+dcramer@users.noreply.github.com>

0.3.0 has the client changes the evals need, so drop the git dependency on unreleased roach main.

sentry-junior Bot assigned dcramer Nov 5, 2026

sentry-junior Bot temporarily deployed to Actions November 5, 2026 01:55 Inactive

vercel Bot deployed to Preview November 5, 2026 01:56

View deployment

dcramer approved these changes Nov 5, 2026

View reviewed changes

dcramer marked this pull request as ready for review November 5, 2026 02:51

github-actions Bot added the risk: medium PR risk score: medium label Nov 5, 2026

sentry-junior Bot mentioned this pull request Nov 5, 2026

fix(evals): Run the eval suite on vitest-evals 0.18 #1973

Merged

sentry-junior Bot force-pushed the feat/roach-evals branch from c44947f to bbc5c92 Compare November 5, 2026 04:47

sentry-junior Bot temporarily deployed to Actions November 5, 2026 04:48 Inactive

vercel Bot deployed to Preview November 5, 2026 04:49

View deployment

sentry-junior Bot temporarily deployed to Actions November 5, 2026 04:54 Inactive

sentry-junior Bot mentioned this pull request Nov 5, 2026

test(evals): Migrate evals to the vitest-evals harness API #1974

Draft

build(evals): Use @sentry/roach 0.3.0

b34c56c

0.3.0 has the client changes the evals need, so drop the git dependency on unreleased roach main.

sentry-junior Bot force-pushed the feat/roach-evals branch from bbc5c92 to b34c56c Compare November 5, 2026 19:56

sentry-junior Bot temporarily deployed to Actions November 5, 2026 19:56 Inactive

vercel Bot deployed to Preview November 5, 2026 19:57

View deployment

dcramer approved these changes Nov 5, 2026

View reviewed changes

dcramer merged commit cecfb1e into main Nov 6, 2026
46 checks passed

dcramer deleted the feat/roach-evals branch November 6, 2026 05:27

Navigation Menu

Repository navigation

feat(evals): Record and replay eval model calls with Roach - #1972

feat(evals): Record and replay eval model calls with Roach#1972
dcramer merged 2 commits into
mainfrom
feat/roach-evals

sentry-junior Bot commented Nov 5, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Nov 5, 2026 •

edited

Loading

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

sentry-junior Bot commented Nov 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vercel Bot commented Nov 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

sentry-junior Bot commented Nov 5, 2026 •

edited

Loading

vercel Bot commented Nov 5, 2026 •

edited

Loading