[sentry-junior]

Add a manual Evals sample workflow. It runs the behavioral eval files that you select N times, live, against one ref, and shows the pass count of each case. Run it one time for main and one time for a branch to compare a prompt change, for example #2371.

Why

A pull request eval run does not measure a prompt change well:

• Evals replay model answers from Roach recordings. A passed test writes its recordings. A failed test writes nothing. So a pass replays for each later run with the same requests, and only failures go live again.
• Across 8 recent behavioral runs on PRs that did not change the prompt, each case had only about 3 distinct answers. "Pass rate across runs" mostly counts replays.
• Each case gets one sample per run.
• evals-behavioral.yml runs only on pull_request, so there is no run for main to compare with.

What

• .github/workflows/evals-behavioral-sample.yml (workflow_dispatch):
  • Inputs: ref (default main), files (space-separated, relative to packages/junior-evals), optional filter (-t), samples (default 5, max 10).
  • plan checks the inputs and resolves ref to one commit, so all samples run the same code.
  • sample is a matrix of N parallel jobs. Each runs the files with VITEST_EVALS_REPLAY_MODE=off (every request is live, nothing is recorded) in the network jail, with the same services, cloudflared, secrets, and snapshot cache as evals-behavioral.yml. Each job publishes its vitest-evals summary and uploads its JSON report.
  • summary writes a table with passed/runs for each case.
• scripts/sample-summary.mjs counts the passed runs per case. It reuses scenariosFromReport from report.mjs (now exported), so case names and status rules are the same as the evals.sentry.dev upload.
• CI runs all scripts/*.test.mjs, so the new test runs in test / eval reporting.
• packages/junior-evals/README.md: new "Live Samples" section. policies/evals.md does not cover how to measure prompt changes, so it is not changed.

Example A/B:

gh workflow run evals-behavioral-sample.yml --repo getsentry/junior \
  -f ref=main -f files="evals/conversation/output.eval.ts" -f samples=5
gh workflow run evals-behavioral-sample.yml --repo getsentry/junior \
  -f ref=<pr-branch> -f files="evals/conversation/output.eval.ts" -f samples=5

Design notes

• Vitest 4 repeats is a per-test option with no CLI flag. A repeated test gets one result, fails if any repeat fails, and keeps only the metadata of the last repeat. It cannot give a pass count, so this uses a job matrix.
• This is a new workflow, not a workflow_dispatch branch in evals-behavioral.yml. A branch there needs conditions in most steps of the PR gate. The services block and the cloudflared step are copied. A composite action for cloudflared would have to exist on every ref that you sample, so old branches could not run.
• The workflow does not upload to evals.sentry.dev and does not publish a check. report.mjs refuses duplicate case names, and N samples of one case would be duplicates. The PR upload path does not change.
• Inputs go through env, not inline ${{ }} in scripts. files must match evals/<path>.eval.ts.

Open questions

• There is still no base run on main, so the Sentry Evals check reports "Waiting for a run of base commit". This PR does not add a push-to-main trigger.
• Should results go to evals.sentry.dev, for example one run per sample or a separate dataset?

Checks

• actionlint 1.7.7 with shellcheck 0.10.0 on the new workflow and ci.yml: clean (custom blacksmith-* runner label ignored).
• node --test packages/junior-evals/scripts/*.test.mjs: 6 pass.
• Input checks and argument building tested locally in bash (bad counts, --config, .., empty files, -t with spaces).
• Prettier check on the changed files: clean.
• Not tested: a real dispatch. workflow_dispatch can only run after the workflow is on the default branch.

via David Cramer.

--

View Junior Session [Sentry]

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
junior-docs	Ready	Preview	Oct 29, 2026 5:57pm UTC

[sentry-evals]

Sentry Evals

junior-behavioral

Run	Status	Passed	Failed	Errored	Cost	Tokens	Duration
Head	Error	80	1	2	$0.57	0	523.85s

Base commit 0f21f18 has no eval run to compare against.

junior-guardian

Run	Status	Passed	Failed	Errored	Cost	Tokens	Duration
Head	Passed	94	0	0	$0.03	0	27.03s

Base commit 0f21f18 has no eval run to compare against.

junior-integration

Run	Status	Passed	Failed	Errored	Cost	Tokens	Duration
Head	Passed	102	0	0	$0.70	0	406.20s

Base commit 0f21f18 has no eval run to compare against.

junior-router

Run	Status	Passed	Failed	Errored	Cost	Tokens	Duration
Head	Passed	12	0	0	$0.00	0	3.71s

Base commit 0f21f18 has no eval run to compare against.

[dcramer]

this overcomplicates our life right now