Skip to content

feat(middleware): Log request context with large RSS growth - #129458

Draft
mwarkentin wants to merge 1 commit into
masterfrom
claude/request-rss-context
Draft

mwarkentin wants to merge 1 commit into
masterfrom
claude/request-rss-context

Conversation

@mwarkentin

Copy link
Copy Markdown
Member

Ticket: PRODENG-2971

Description:

Environments affected:
Only processes with SENTRY_REQUEST_RSS_METRICS set (currently the getsentry web canaries in SaaS US, DE, S4S2). Off everywhere else, including self-hosted.

Change plan:

AI Commentary

Why. After ~55 hours on the web canaries, request.rss_growth names the endpoints behind most large RSS jumps (repository.get_repositories, the issue events lists, the GitHub webhook, chunk-upload, ...). Reading the code points to specific causes, but confirming them needs to know how each request was made: e.g. whether the events lists are hit with 100-row pages by API clients, and whether the GitHub webhook spikes are big push events.

What it does. The log line gains these fields (the metric and its tags are unchanged, so no new cardinality):

  • duration_ms, status_code, request_bytes (from Content-Length), response_bytes (none for streaming responses)
  • user_agent (truncated to 200 characters)
  • webhook_event, from X-GitHub-Event, X-Gitlab-Event or X-Event-Key
  • organization, the organization_id_or_slug URL argument, to tell whether one organization accounts for the outliers
  • param_<name> for an allowlist of query parameters (per_page, limit, full, dataset, sort, expand, collapse, statsPeriod, referrer), plus field_count and query_length. Search text and other parameters are never logged, only the query's length.
  • worker_request_number and worker_uptime_s. A new worker's first requests grow RSS while it imports modules and fills caches; these separate warm-up from real offenders.

Keys are flat so each is a queryable attribute in Sentry logs.

Safety. Reporting runs in the middleware's finally, so it is now wrapped: any failure while building the log line is logged with logger.exception and never fails the request or masks the view's exception (e.g. Django raising on a query string with more than 1,000 parameters). The only new per-request cost is two time.monotonic() calls; everything else runs only for requests over the threshold.

Tests. tests/sentry/middleware/test_request_memory.py, 22 tests, adding:

  • the new fields on a basic report;
  • allowlisted parameters logged, search text and other parameters not;
  • the worker request count;
  • no response size for a streaming response, and none of status or size when the view raises;
  • a failure while reporting doesn't fail the request.

prek and mypy pass on both files.

Generated by Claude Code


Generated by Claude Code

The request.rss_growth log line now also records what made the request
large: duration, status, request and response sizes, user agent,
webhook event type, the organization from the URL, and the values of a
few paging and dataset query parameters (only the length of a search
query). It also records how many requests the worker had served and its
uptime, so a new worker's warm-up can be told apart from real
offenders. Reporting can no longer fail the request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Lpv3UTF9m93wLvaw2ruhQ8
@github-actions github-actions Bot added the Scope: Backend Automatically applied to PRs that change backend components label Nov 2, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Scope: Backend Automatically applied to PRs that change backend components

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants