diff --git a/AGENTS.md b/AGENTS.md index 51a9624..6bcd000 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -32,7 +32,7 @@ - The repository is a pnpm workspace. The root is private and has only the tools. `packages/roach` is the npm package `@sentry/roach`: the client, the vitest integration, and the action. `packages/service` is the private `@sentry/roach-service`: the service and the proxy. Put a shared dependency version in the catalog of `pnpm-workspace.yaml`, and refer to it as `catalog:`. - Use Node built-ins and the `openssl` command. `@sentry/roach` has no runtime dependencies. `@sentry/node` is the only one of `@sentry/roach-service`, and only `packages/service/src/service.ts` imports it. The service imports `@sentry/roach` by package name (`workspace:*`), never by a relative path. `@sentry/roach` never imports the service. `packages/roach/tests/package.test.ts` fails when an export or the action loads a package that is not a Node built-in or vitest, or when a file in `packages/roach/src/` is not used. -- `deploy/gcp/` is the production setup (Terraform). Change it in the same change when the service config (`RoachServiceConfig`) changes. CI runs `terraform validate` on it. +- `deploy/gcp/` is the production setup (Terraform). Change it in the same change when the service config (`RoachServiceConfig`) changes. The `Image` workflow runs `terraform fmt -check` and `terraform validate` on it. - Node runs both packages as TypeScript with type stripping. Use only erasable syntax, and import local files with the `.ts` extension. - Use functions and plain objects, not classes. - Start each source file with a comment that says what the file owns. Give each export a short JSDoc. diff --git a/README.md b/README.md index 629d338..7280615 100644 --- a/README.md +++ b/README.md @@ -127,7 +127,8 @@ A repository adds one step and a `roach.json` file (see The action (`packages/roach/src/action.ts`) starts a run with the token in `auto` mode, or without it in `replay` mode. It runs `run` in bash with the proxy variables -of the run, then ends the run and writes its stats to the job summary. +of the run, then ends the run and writes its stats to the job summary. The +`url` input sets the service. Default: `https://roach-proxy.getsentry.net`. - Only `run` uses the proxy. Later steps, such as cache and artifact uploads, do not. `run` does not get the write token or the `ACTIONS_*` @@ -204,8 +205,9 @@ A run takes `RunConfig` (`packages/roach/src/types.ts`): `tenant`, `mode`, `rule look there first when requests fail with HTTP 403. A tunnel to a port other than 443 shows as `host:port`. The stats name 32 origins for each reason, and count the others as `other origins`. -- `rules`: the traffic to record. `name` uses letters, digits, `_`, `.`, - and `-`. `match` has `method`, `url`, and `headers`. `url` is a prefix, +- `rules`: the traffic to record. `name` starts with a letter or a digit, + has at most 64 characters, and uses letters, digits, `_`, `.`, and `-`. + `match` has `method`, `url`, and `headers`. `url` is a prefix, or a list of 1 to 32 prefixes. A URL that starts with one of them matches. Use a list when the same API is on many hosts, such as one rule with the same `values` for Anthropic, OpenRouter, and OpenAI. The @@ -218,9 +220,10 @@ A run takes `RunConfig` (`packages/roach/src/types.ts`): `tenant`, `mode`, `rule or of the run. Use `secrets` of the run for the credentials of a CI job that a response can show but no request header has, such as a token that goes to a model in a prompt. The service keeps them only in memory, for - the run. A run can have 32 of them. It also learns the value of each - request header and query parameter whose name can mean a credential, - such as one with `auth`, `token`, `key`, `secret`, `cookie`, or `session`. + the run. A run can have 32 of them, each of at most 16 Ki characters. It + also learns the value of each request header and query parameter whose + name can mean a credential: a name with `auth`, `token`, `secret`, + `key`, `cookie`, `session`, `password`, `credential`, or `signature`. The match is broad on purpose. It does not learn `x-session-affinity`, because that header has a conversation id and not a credential. Recordings show `<>` in place of each known value. The URL of @@ -363,8 +366,10 @@ stream. Each response has an `x-roach` header: `replayed`, `live`, The key of a recording is `/.json`, under `/` in the store. The hash is the hash of the rule name, the method, the URL, the key headers, and the body. A JSON body has sorted keys. A recording keeps the -response and the session that recorded it. It does not keep the request -body. A request sample keeps it. See [Misses](#misses). +response, the session that recorded it, and the method and the redacted +URL of the request. Of the response headers, it keeps only `content-type` +and `location`. It does not keep the request body. A request sample keeps +it. See [Misses](#misses). - Many sessions can be open at once. A request belongs to the session of its proxy URL (`url` of the session). A request through the proxy URL of @@ -382,7 +387,7 @@ body. A request sample keeps it. See [Misses](#misses). Ids and times change on each run, and a response often repeats them. For example, the model archives the memory with the id that its request showed. `values` of a rule maps a name to a regular expression source. -`VALUE_PATTERNS` in `values.ts` has `uuid`, `isoTime`, `date`, `epochMs`, +`VALUE_PATTERNS` in `packages/roach/src/values.ts` has `uuid`, `isoTime`, `date`, `epochMs`, `sha256`, and `gitCommit`. When two patterns match at the same place, the first one wins. The proxy matches values in JSON text. Start a custom pattern with `NOT_AFTER_WORD`, not `\b`: after an escape such as `\n`, `\b` sees the `n` @@ -459,12 +464,13 @@ Avoid nested quantifiers, such as `(a+)+`. ## Misses The proxy adds each request without a recording to `stats().misses`, with -its session and key. `describeRecordingMisses()` in `report.ts` gives the +its session and key. The stats keep the first 500 misses of a run. +`describeRecordingMisses()` in `packages/roach/src/report.ts` gives the first miss of each test, which is the one to fix. The run report of the vitest integration and of the GitHub Action shows these misses. The proxy also finds where the first miss of each session changed -(`misses.ts`). Sentry gets the text of the change, on the `roach.request` +(`packages/service/src/misses.ts`). Sentry gets the text of the change, on the `roach.request` span of the miss (see [Sentry](#sentry)). The report of a run with the write token tells only where the change is: @@ -607,10 +613,8 @@ config is `RoachServiceConfig` in `packages/service/src/service.ts`. `deploy/gcp refuses a request that no rule records, so nothing goes live. The stats count these requests in `refused.replayOnly`. Every other mode needs the write token. So treat each recording as readable by anyone who can - make its request. Do not record responses that must stay private. Only - runs with the write token get the change of a miss in their stats, and - only its offset (see [Misses](#misses)). So CI of a fork does not show - it. Sentry shows it. + make its request. Do not record responses that must stay private. For + what a run without the token sees of a miss, see [Misses](#misses). - **Runs.** Each run has its own id, token, mode, rules, sessions, and stats. A proxied request names its run in `Proxy-Authorization`, as `Basic base64(:)`. The proxy URL of the run has them, so @@ -619,13 +623,14 @@ config is `RoachServiceConfig` in `packages/service/src/service.ts`. `deploy/gcp their value patterns. A tenant that calls a new origin or uses a new pattern changes only its own config. The service needs no change and no restart. Two checks keep one run from harming the others: - - The proxy sends requests only to public addresses (`addresses.ts`). + - The proxy sends requests only to public addresses + (`packages/service/src/addresses.ts`). It checks the address of each connection after DNS, so a host name cannot point a run at the metadata server, at the VPC, or at the VM. Only tests and local development set `privateNetworks`. - A key has a time limit. The service makes the keys of runs without - the token in a worker process with a memory limit (`key-worker.ts`, - `request-key.ts`). So a slow pattern or a large body fails only its + the token in a worker process with a memory limit + (`packages/service/src/key-worker.ts`). So a slow pattern or a large body fails only its own request, and it does not stop the service. - **Sentry.** With `sentryDsn`, each read and write of a recording sends a span to Sentry. Each proxied request sends a span and metrics. Each @@ -635,7 +640,8 @@ config is `RoachServiceConfig` in `packages/service/src/service.ts`. `deploy/gcp of the upstream or of GCS, gets HTTP 502 for a proxied request or HTTP 500 for a control call. Its text is only `internal error, Sentry event `, because the message of the error can have a response body. Only - Sentry gets the error. The service log gets its class and HTTP status. + Sentry gets the error. The service log gets its class, its code, and + the HTTP status of a failed GCS call. - **Certificate authority.** `ca` holds the PEM certificate and key that clients trust. Keep it across restarts. Host certificates live for 7 days, and the service makes new ones before they expire. @@ -663,10 +669,11 @@ Each proxied request is one trace. Each span has `roach.tenant`, token). - `roach.request`: one proxied request, named ` `, with `roach.origin`, `roach.rule`, `roach.result`, `roach.session` (the name - of the session, such as the test), and `http.response.status_code`. A - refused request also has `roach.reason`. When a replayed or live - response has token usage, the span also has the `gen_ai` attributes of - the model call (`usage.ts`). Sentry then calculates its cost in USD as + of the session, such as the test), `http.request.method`, and + `http.response.status_code`. A refused request also has `roach.reason`. + When a replayed or live response has token usage, the span also has the + `gen_ai` attributes of the model call + (`packages/service/src/usage.ts`). Sentry then calculates its cost in USD as `gen_ai.cost.total_tokens`. The span of the first miss of a session also has `miss.sample` (`found`, `withheld`, or `none`), in every run. When the proxy finds a change, the span also has `miss.changed_at` (the @@ -680,8 +687,8 @@ Each proxied request is one trace. Each span has `roach.tenant`, `roach.result`: `replayed` (the store has the recording) or `missed`. The first miss of a session also reads a request sample. Its `roach.key` starts with `_requests/`. - - `roach.upstream`: the live request, with `roach.origin` and - `http.response.status_code`. + - `roach.upstream`: the live request, with `roach.origin`, + `http.request.method`, and `http.response.status_code`. - `roach.store.write`: a write of recordings, with `roach.recordings` and `roach.written`. A session writes when it ends, so this span is usually its own trace. A second span writes the request samples of the same @@ -704,21 +711,9 @@ removes the path from the name of an error event, and adds no trace headers to the requests that it sends. An error event has the message of the error, which can have a response body. -To see how well the cache works: - -- **Hit rate.** Sum `roach.request`, group by `roach.result`, and filter - by `roach.tenant` and `roach.rule`. The hit rate is - `replayed / (replayed + live + missed)`. -- **Misses.** `missed` is a request that failed in `replay` mode. `live` - in `auto` mode is a new recording. When `live` stays high from run to - run, a value in the key changes on each run. -- **Time.** Compare `roach.request.duration` for `live` and `replayed`. -- **Dropped.** `roach.recording.write` with `roach.result:dropped` counts - recordings of failed sessions. The next run sends these requests live - again. -- **Changing keys.** Search spans for `span.op:roach.key`, group by - `roach.key`, and count the runs. A key that only one run uses shows that - a value changes on each run. +[`TELEMETRY.md`](TELEMETRY.md) shows how to get the hit rate, the misses, +the time that replays save, the dropped recordings, and the keys that +change on each run. ### Limits @@ -759,7 +754,8 @@ To see how well the cache works: [Changing values](#changing-values). - Request headers that do not come in 10 s get HTTP 408, and a request whose body does not come in 300 s too. This is also true in a tunnel - and with `proxyProtocol`. + and with `proxyProtocol`. With `proxyProtocol`, a connection that sends + no byte, or no full PROXY header, in 10 s closes. - A `CONNECT` tunnel that sends no byte for 10 s after the `200` closes. - A connection or tunnel that sends and gets no byte for 10 minutes closes, also while a request waits for its response. This is true for @@ -872,13 +868,10 @@ To deploy by hand, restart the VM: `gcloud compute instances reset roach --zone `. It pulls `:main`, the last green commit. -The service reads its config only when it starts. With `proxyProtocol`, -it reads a PROXY header when a connection has one, and it uses the -address of the connection when it does not. A service without -`proxyProtocol` cannot read the header, so a load balancer that sends the -header before the service reads it fails each request with HTTP 400. To -turn on the PROXY header, apply in this order. Run the commands in -`deploy/gcp`: +The service reads its config only when it starts. When a change to the +service config must reach the VM before a change to the load balancer, +such as `proxyProtocol` before `proxy_header`, apply in this order. Run +the commands in `deploy/gcp`: 1. Merge the change, and let the `Deploy` workflow deploy the image. 2. Apply only the config secret: @@ -888,34 +881,30 @@ turn on the PROXY header, apply in this order. Run the commands in -target=google_secret_manager_secret_version.config ``` -3. Restart the VM, so that it reads the new config. It then reads a - header when a connection has one: +3. Reset the VM, so that it reads the new config: ```sh gcloud compute instances reset roach --project roach-511216 --zone us-east1-b ``` -4. When the service answers, apply the rest. The load balancer then - sends the header (`proxy_header` of `google_compute_target_ssl_proxy` - and of `google_compute_health_check`): +4. When the service answers, apply the rest: ```sh terraform apply -var-file=roach.tfvars ``` -No step fails requests with HTTP 400. But step 1 and step 3 each reset -the VM, and a reset ends the open runs, so their CI jobs fail (see -[Deploy](#deploy)). Do step 3 when few runs are open. From step 3 to -step 4, the service sees the addresses of the load balancer, so all -clients share the limit of runs for each client. +Step 1 and step 3 each reset the VM, and a reset ends the open runs, so +their CI jobs fail. Do step 3 when few runs are open, and do step 4 at +once after it. -The load balancer writes its header before the first byte of the client, -so a client cannot put its own header first. A client can send its own -header only when the service has `proxyProtocol` and the load balancer -does not send the header: from step 3 to step 4 above. So do step 4 at -once. A false header changes only the address for the limit of runs for -each client, and the firewall lets only the Google ranges connect to the -VM. +With `proxyProtocol`, the service reads a PROXY header when a connection +has one, and it uses the address of the connection when it does not. The +load balancer writes its header before the first byte of the client, so a +client cannot put its own header first. A client can send its own header +only when the service has `proxyProtocol` and the load balancer does not +send the header: from step 3 to step 4 above. So do step 4 at once. A +false header changes only the address for the limit of runs for each +client, and the firewall lets only the Google ranges connect to the VM. To add a project, add its repository to the repositories of the `ROACH_TOKEN` organization secret, in the GitHub settings of the @@ -955,10 +944,17 @@ The service also has: - `POST /__roach/runs` with a `RunConfig`: start a run. Send `Authorization: Bearer ` for any mode but `replay`. Returns - the `id`, `token`, proxy `url`, `controlUrl`, and `caCert` of the run. + HTTP 201 with the `id`, `token`, proxy `url`, `controlUrl`, and `caCert` + of the run. A body that is not JSON or a config that is not valid gets + HTTP 400, a body over 1 MiB gets HTTP 413, a mode other than + `replay` without the token gets HTTP 401, and a run past the limit of + runs without a token gets HTTP 429 (see [Limits](#limits)). - `DELETE /__roach/runs/`: end the run. Returns its stats. - `GET /__roach/ca.pem`: the CA certificate. No token. +A call of a run with a wrong token gets HTTP 404, as a call of a run that +does not exist does. + ## Release Run the `Release` workflow, and choose `minor`, `patch`, or `major`. Craft @@ -994,14 +990,21 @@ no image, and Craft cannot publish it. ```sh pnpm install -pnpm check # format, lint, types, unused code, and tests, as CI runs them +pnpm check # format, lint, types, unused code, and tests ``` -`AGENTS.md` has the conventions and the commands for one file. - -- `packages/service/tests/recording.test.ts` tests modes, sessions, changing values, and - misses. -- `packages/service/tests/service.test.ts` tests the service in the test process. +CI runs the tests only with coverage (`pnpm test:coverage`), and packs +`@sentry/roach`. It does not run `pnpm check`, so run it before you push. `AGENTS.md` has the conventions and the commands for one +file. + +- `packages/roach/tests/package.test.ts` checks what the npm package + loads and ships. +- `packages/service/tests/recording.test.ts` tests modes, sessions, + changing values, and misses. +- `packages/service/tests/redaction.test.ts` tests the redaction of + credentials in recordings and request samples. +- `packages/service/tests/service.test.ts` tests the service in the test + process: access, limits, and the control API. - `packages/service/tests/vitest.test.ts` runs vitest with the integration on a small project, against a service in the test process. - `packages/service/tests/deployed.test.ts` runs the service as it runs in production: from diff --git a/TELEMETRY.md b/TELEMETRY.md index 88f0078..8952b1e 100644 --- a/TELEMETRY.md +++ b/TELEMETRY.md @@ -3,8 +3,9 @@ Use this file to answer questions about Roach from its Sentry data. Examples: "how many requests replayed?", "why does this run miss?", and "how much time do recordings save?". "Sentry" in `README.md` lists the metrics, spans, and -errors. The code in `packages/service/src/service.ts` and -`packages/service/src/recorder.ts` is authoritative. +errors. The code in `packages/service/src/service.ts`, +`packages/service/src/server.ts`, and `packages/service/src/recorder.ts` is +authoritative. ## Where To Query @@ -63,29 +64,33 @@ For metrics, use `-f dataset=tracemetrics`, ## Pivots -| Pivot | Meaning | Found In | -| --------------------------- | ------------------------------------------ | --------------------------------- | -| `roach.tenant` | repository of the run | metrics, spans, errors | -| `roach.mode` | `auto`, `replay`, `record`, `off` | metrics, spans, errors | -| `roach.rule` | recording rule, such as `model` | metrics, spans | -| `roach.result` | `replayed`, `live`, and others (see Words) | metrics, spans | -| `roach.origin` | upstream origin | metrics, spans | -| `roach.reason` | why the proxy refused a request | `roach.request` metric and span | -| `roach.run.id` | `name` of the run, else the service id | spans, errors | -| `roach.ci.run_id` | `GITHUB_RUN_ID` of the run | spans, errors | -| `roach.ci.job` | `GITHUB_JOB` of the run | spans, errors | -| `roach.ci.repo` | `GITHUB_REPOSITORY` of the run | spans, errors | -| `roach.ci.ref` | `GITHUB_REF` of the run | spans, errors | -| `roach.session` | session (test) name of a request | `roach.request` | -| `roach.key` | recording key of a request | `roach.key`, `roach.store.read` | -| `roach.write` | the run has the write token (boolean) | `roach.run.start` | -| `http.response.status_code` | status to the client | `roach.request`, `roach.upstream` | -| `gen_ai.request.model` | model of a model call | `roach.request` | -| `miss.sample` | `found`, `withheld`, or `none` for a miss | `roach.request` | -| `miss.changed_at` | offset of the change of a miss | `roach.request` | -| `miss.compared` | recording key of a miss change | `roach.request` | -| `miss.recorded` | recorded text of a miss change | `roach.request` | -| `miss.request` | sent text of a miss change | `roach.request` | +| Pivot | Meaning | Found In | +| --------------------------- | ------------------------------------------------ | ------------------------------------ | +| `roach.tenant` | repository of the run | metrics, spans, errors | +| `roach.mode` | `auto`, `replay`, `record`, `off` | metrics, spans, errors | +| `roach.rule` | recording rule, such as `model` | metrics, spans | +| `roach.result` | `replayed`, `live`, and others (see Words) | metrics, spans | +| `roach.origin` | upstream origin | metrics, spans | +| `roach.reason` | why the proxy refused a request | `roach.request` metric and span | +| `roach.run.id` | `name` of the run, else the service id | spans, errors | +| `roach.ci.run_id` | `GITHUB_RUN_ID` of the run | spans, errors | +| `roach.ci.job` | `GITHUB_JOB` of the run | spans, errors | +| `roach.ci.repo` | `GITHUB_REPOSITORY` of the run | spans, errors | +| `roach.ci.ref` | `GITHUB_REF` of the run | spans, errors | +| `roach.session` | session (test) name of a request | `roach.request` | +| `roach.key` | recording key of a request | `roach.key`, `roach.store.read` | +| `roach.write` | the run has the write token (boolean) | `roach.run.start` | +| `roach.recordings` | files that one write sends to the store | `roach.store.write` | +| `roach.written` | files that were new or changed | `roach.store.write`, `roach.run.end` | +| `http.request.method` | method of the request | `roach.request`, `roach.upstream` | +| `http.response.status_code` | status to the client | `roach.request`, `roach.upstream` | +| `gen_ai.request.model` | model that a model call asked for | `roach.request` | +| `gen_ai.response.model` | model of the response, when the request has none | `roach.request` | +| `miss.sample` | `found`, `withheld`, or `none` for a miss | `roach.request` | +| `miss.changed_at` | offset of the change of a miss | `roach.request` | +| `miss.compared` | recording key of a miss change | `roach.request` | +| `miss.recorded` | recorded text of a miss change | `roach.request` | +| `miss.request` | sent text of a miss change | `roach.request` | A metric for a `notAllowed` request has `roach.origin:other`, because the client picks that origin. Its span has the origin. @@ -155,6 +160,15 @@ sentry explore sentry/roach --dataset metrics --metric roach.recording.write \ --agg sum --period 24h -F roach.rule -F roach.result ``` +Keys that change on each run. A key that only one run uses shows that a +value changes on each run. Add `-F roach.run.id` to see the runs. + +```bash +sentry explore sentry/roach --dataset spans --period 24h \ + -q 'span.op:roach.key roach.tenant:getsentry/junior' -F roach.rule \ + -F roach.key -F 'count()' -s 'count()' +``` + Runs with the most live requests. When the same tests stay live from run to run, a value in the key changes on each run. Then look at the keys of one run. "Find changing values" in `README.md` shows how to find the value. @@ -242,7 +256,7 @@ Spent and saved, by rule and model. ```bash sentry explore sentry/roach --dataset spans --period 24h \ -q 'span.op:roach.request has:gen_ai.usage.input_tokens' \ - -F roach.result -F roach.rule -F gen_ai.request.model \ + -F roach.result -F roach.rule -F gen_ai.request.model -F gen_ai.response.model \ -F 'sum(gen_ai.cost.total_tokens)' -F 'sum(gen_ai.usage.input_tokens)' \ -F 'sum(gen_ai.usage.output_tokens)' -F 'count()' ``` @@ -275,8 +289,8 @@ and the next run makes it again. Compare `roach.recording.write` with metric does not count samples. - An error event has the message of the error. The message can have the body of a GCS response. Only Sentry gets it. The client gets only the id - of the event. The service log gets the id, the class, and the HTTP status - of the error. + of the event. The service log gets the id of the event, and the class, + the code, and the HTTP status of the error. - An error of the control API, such as a failed write at the end of a session, has no `roach.*` tags. - Cost is on spans only, not on metrics. diff --git a/packages/service/src/service.ts b/packages/service/src/service.ts index 17b5fb1..6c3d245 100644 --- a/packages/service/src/service.ts +++ b/packages/service/src/service.ts @@ -21,8 +21,9 @@ * service. Two checks keep that safe for the other tenants. The proxy * sends requests only to public addresses (`addresses.ts`), so a run * cannot reach the metadata server or the VPC. The key of a request has - * a time limit, and it is made in a worker thread (`key-worker.ts`), so - * a slow pattern fails only its own request. + * a time limit, and the keys of runs without the token are made in a + * worker process (`key-worker.ts`), so a slow pattern fails only its own + * request. * - The recordings of a tenant are under `/`, in a directory or a * GCS bucket (`gcs.ts`). Tenants never share recordings, because a key * does not include credentials.