From 4a9594b4d6050d73607b08bfe39eff6692edb79f Mon Sep 17 00:00:00 2001 From: "sentry-junior[bot]" <264270552+sentry-junior[bot]@users.noreply.github.com> Date: Thu, 5 Nov 2026 02:47:57 +0000 Subject: [PATCH] docs: Explain how to find values that change between runs A value that changes on each run and that no pattern finds makes a request go live on every run, with no error. Add the check to do after setup, a way to capture request bodies to compare, and the sources found in Warden. Co-Authored-By: David Cramer <23610+dcramer@users.noreply.github.com> --- README.md | 74 +++++++++++++++++++++++++++++++++++++++++++++++++++- TELEMETRY.md | 1 + 2 files changed, 74 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index e107932..5aa7916 100644 --- a/README.md +++ b/README.md @@ -37,7 +37,8 @@ gives its proxy variables to the test workers. The setup file record in parallel workers at the same time. A passed test writes its new recordings. A failed test drops them. In `replay` mode, a test with a miss fails. At the end, the global setup writes the stats of the run to stdout -and to the job summary. +and to the job summary. After you add Roach, do the check in +[Find changing values](#find-changing-values). - `ROACH_TOKEN`: the write token. Only the global setup reads it. Without it, the run can only replay, as in jobs of forks. @@ -271,6 +272,77 @@ The proxy adds each request without a recording to `stats().misses`, with its session and key. `describeRecordingMisses()` in `report.ts` gives the first miss of each test, which is the one to fix. +## Find changing values + +Do this check when you add Roach to a project. Do it again when a test +that passed does not replay on the next run. + +A value that changes on each run and that no pattern finds gives a new key +on each run. In `auto` mode, that request goes live and records again on +each run. The run is slow, and it costs as much as a run without Roach. +The requests after it in the test also go live, because the live response +is not the same as the recorded one. No error shows this. The tests pass. + +1. Run the suite two times with the token. On the second run, each test + that passed must replay all of its requests. Look at `live` in the stats + of the second run. It must be 0, or near 0. In CI, also compare the + time of each test. A test that replays takes about as long as its code + without the network. [`TELEMETRY.md`](TELEMETRY.md) shows how to find + the runs and keys that stay live. +2. Find the first live request of a test that did not replay. Run only + that test with `ROACH_MODE=replay`. It needs no token. The test fails + at the first miss, and the run report shows its key. Fix the first miss + of a test before the others. The others often come from it. +3. Get the body of that request from two runs, and compare them. A + recording does not keep the request body, so get it in the client. For + example, add this file to `setupFiles` after + `@sentry/roach/vitest/setup`, and run the test two times: + + ```ts + // Local only. Writes each fetch request to DUMP_FILE. + import { appendFileSync } from "node:fs"; + import { expect } from "vitest"; + + const dumpFile = process.env.DUMP_FILE ?? "requests.jsonl"; + const fetchBefore = globalThis.fetch; + globalThis.fetch = async (input, init) => { + const request = new Request(input, init); + const record = { + test: expect.getState().currentTestName, + method: request.method, + url: request.url, + body: await request.clone().text(), + }; + appendFileSync(dumpFile, JSON.stringify(record) + "\n"); + return fetchBefore(request); + }; + ``` + + The first difference between the two bodies is the changing value. + This file sees only `fetch`. A client that uses `node:http` directly + needs its own hook. + +4. Add a narrow pattern for the value to `values` of the rule. See + [Changing values](#changing-values). Then do step 1 again. + +Values that changed in real projects: + +- Ids that the code under test makes, such as UUIDs and random short ids. + For example, Warden gives each finding a random `XXX-XXX` id, and the + next model request quotes it. +- Paths of temporary directories, commit SHAs of a repository that a test + makes, and times and dates. +- The local environment. Global git config, such as a hook that adds a + commit trailer, commit signing, or `user.name`, changes the commits that + a test makes. `HOME`, the user name, the locale, and the time zone can + also show in a request. Then local keys are not the same as CI keys, so + a local `replay` misses while CI replays. Make the test harness ignore + them. For git, set `GIT_CONFIG_GLOBAL=/dev/null` and + `GIT_CONFIG_NOSYSTEM=1`, and set the author and committer in the test. +- Order. The proxy sorts JSON object keys, but not arrays. An array whose + order depends on timing, such as the results of parallel tool calls, + gives a new key when the order changes. + ## Service `cli.ts service ` starts the service. The config is diff --git a/TELEMETRY.md b/TELEMETRY.md index 79d7e19..af9aab7 100644 --- a/TELEMETRY.md +++ b/TELEMETRY.md @@ -80,6 +80,7 @@ sentry explore sentry/roach --dataset metrics --metric roach.recording.write \ Runs with the most live requests. When the same tests stay live from run to run, a value in the key changes on each run. Then look at the keys of one run. +"Find changing values" in `README.md` shows how to find the value. ```bash sentry explore sentry/roach --dataset spans --period 24h \