Skip to content

fix(replays): Bound recording segment decompression at ingest - #123354

Closed
DominikB2014 wants to merge 4 commits into
masterfrom
dominikbuszowiecki/replay-994-bound-segment-decompression-at-ingest
Closed

DominikB2014 wants to merge 4 commits into
masterfrom
dominikbuszowiecki/replay-994-bound-segment-decompression-at-ingest

Conversation

@DominikB2014

@DominikB2014 DominikB2014 commented Sep 1, 2026 •

Copy link
Copy Markdown
Contributor

Backend only (src/, tests/). Relay companion: getsentry/relay#6587, which applies Relay's uncompressed size limit to every recording, not only scrubbed ones.

Problem

decompress_segment inflated recording segments with an uncapped zlib.decompress before the consumer stored them. Relay bounds the compressed payload at 10 MiB, but only applies its 100 MiB uncompressed bound for orgs that have recording scrubbing enabled with a non-empty PII config. For everyone else nothing checked how large a segment became once inflated, and the consumer stored the client's compressed bytes verbatim.

Change

  • decompress_segment now decompresses incrementally via zlib.decompressobj, requesting one byte past the limit so an oversized segment is detected without ever materializing more than one byte over the limit. Segments over the limit raise DropSilently.
  • New replay.consumer.max-segment-decompressed-size option, default 100 MiB, FLAG_AUTOMATOR_MODIFIABLE — so the limit can be raised or lowered without a deploy.
  • Drops are counted under replays.consumer.recording.oversized_segment and logged with the compressed size and the limit.
  • decompress_segment reads the option itself instead of having it passed down through process_message and parse_recording_event.

Behavior for well-formed and already-uncompressed segments is unchanged. zlib.decompressobj returns partial output on a truncated stream where the one-shot zlib.decompress raised, so that case is normalized back to zlib.error to preserve the existing handling.

Rollout note

The default matches the uncompressed bound Relay already applies to scrubbing-enabled orgs, so it should be a no-op for segments produced by SDKs. If it turns out legitimate segments sit closer to the limit than expected, replays.consumer.recording.oversized_segment will show it and the option can be raised immediately. Worth watching that metric before considering this settled.

Refs REPLAY-994

via @DominikB2014.

--

View Junior Session [Sentry]

Recording segments were inflated with an uncapped `zlib.decompress` before being stored. Relay bounds the compressed payload at 10 MiB, but only applies its uncompressed bound for orgs with recording scrubbing and a non-empty PII config, so a highly compressible segment could be stored with no bound on its inflated size.

Decompress incrementally against a byte cap instead, and drop segments that exceed it. The limit comes from a new `replay.consumer.max-segment-decompressed-size` option (default 100 MiB) so it can be tuned without a deploy; drops are counted under `replays.consumer.recording.oversized_segment`.

A follow-up applies the same bound on the read path.

Refs REPLAY-994
@linear-code

linear-code Bot commented Sep 1, 2026

Copy link
Copy Markdown

REPLAY-994

@github-actions github-actions Bot added the Scope: Backend Automatically applied to PRs that change backend components label Sep 1, 2026
@DominikB2014

Copy link
Copy Markdown
Contributor Author

@cursor review

@cursor cursor Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@getsantry

getsantry Bot commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

This pull request has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you add the label WIP, I will leave it alone unless WIP is removed ... forever!


"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

@getsantry getsantry Bot added the Stale label Sep 23, 2026
@getsantry getsantry Bot closed this Sep 30, 2026
@DominikB2014 DominikB2014 reopened this Oct 27, 2026
…replay-994-bound-segment-decompression-at-ingest

# Conflicts:
#	src/sentry/replays/consumers/recording.py

Co-Authored-By: Dominik Buszowiecki <44422760+DominikB2014@users.noreply.github.com>
Read the option where it is used instead of threading it through
process_message and parse_recording_event.

Refs REPLAY-994

Co-Authored-By: Dominik Buszowiecki <44422760+DominikB2014@users.noreply.github.com>

@cursor cursor Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

decompress_segment reads replay.consumer.max-segment-decompressed-size from
the options store, which falls through to the database on a cache miss. The
unit tests calling it without a DB only passed when another test had already
warmed the options cache, and failed when run on their own.

Refs REPLAY-994

Co-Authored-By: Dominik Buszowiecki <44422760+DominikB2014@users.noreply.github.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 55632f7. Configure here.

parse_recording_event(msgpack.packb(message))


@django_db_all

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test missing database access decorator

Medium Severity

test_process_message_invalid_recording_json still reaches decompress_segment, which now calls options.get before inflating the body. Valid headers plus an invalid payload (b"t") take that path, so without @django_db_all the options lookup hits the database blocker and the test errors. The same decorator was added to other tests on this path, but this one was missed.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 55632f7. Configure here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Scope: Backend Automatically applied to PRs that change backend components

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant