Repository navigation
fix(replays): Bound recording segment decompression at ingest - #123354
DominikB2014 wants to merge 4 commits into
Conversation
Recording segments were inflated with an uncapped `zlib.decompress` before being stored. Relay bounds the compressed payload at 10 MiB, but only applies its uncompressed bound for orgs with recording scrubbing and a non-empty PII config, so a highly compressible segment could be stored with no bound on its inflated size. Decompress incrementally against a byte cap instead, and drop segments that exceed it. The limit comes from a new `replay.consumer.max-segment-decompressed-size` option (default 100 MiB) so it can be tuned without a deploy; drops are counted under `replays.consumer.recording.oversized_segment`. A follow-up applies the same bound on the read path. Refs REPLAY-994
|
@cursor review |
|
This pull request has gone three weeks without activity. In another week, I will close it. But! If you comment or otherwise update it, I will reset the clock, and if you add the label "A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀 |
…replay-994-bound-segment-decompression-at-ingest # Conflicts: # src/sentry/replays/consumers/recording.py Co-Authored-By: Dominik Buszowiecki <44422760+DominikB2014@users.noreply.github.com>
Read the option where it is used instead of threading it through process_message and parse_recording_event. Refs REPLAY-994 Co-Authored-By: Dominik Buszowiecki <44422760+DominikB2014@users.noreply.github.com>
decompress_segment reads replay.consumer.max-segment-decompressed-size from the options store, which falls through to the database on a cache miss. The unit tests calling it without a DB only passed when another test had already warmed the options cache, and failed when run on their own. Refs REPLAY-994 Co-Authored-By: Dominik Buszowiecki <44422760+DominikB2014@users.noreply.github.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 55632f7. Configure here.
| parse_recording_event(msgpack.packb(message)) | ||
|
|
||
|
|
||
| @django_db_all |
There was a problem hiding this comment.
Test missing database access decorator
Medium Severity
test_process_message_invalid_recording_json still reaches decompress_segment, which now calls options.get before inflating the body. Valid headers plus an invalid payload (b"t") take that path, so without @django_db_all the options lookup hits the database blocker and the test errors. The same decorator was added to other tests on this path, but this one was missed.
Reviewed by Cursor Bugbot for commit 55632f7. Configure here.


Backend only (
src/,tests/). Relay companion: getsentry/relay#6587, which applies Relay's uncompressed size limit to every recording, not only scrubbed ones.Problem
decompress_segmentinflated recording segments with an uncappedzlib.decompressbefore the consumer stored them. Relay bounds the compressed payload at 10 MiB, but only applies its 100 MiB uncompressed bound for orgs that have recording scrubbing enabled with a non-empty PII config. For everyone else nothing checked how large a segment became once inflated, and the consumer stored the client's compressed bytes verbatim.Change
decompress_segmentnow decompresses incrementally viazlib.decompressobj, requesting one byte past the limit so an oversized segment is detected without ever materializing more than one byte over the limit. Segments over the limit raiseDropSilently.replay.consumer.max-segment-decompressed-sizeoption, default 100 MiB,FLAG_AUTOMATOR_MODIFIABLE— so the limit can be raised or lowered without a deploy.replays.consumer.recording.oversized_segmentand logged with the compressed size and the limit.decompress_segmentreads the option itself instead of having it passed down throughprocess_messageandparse_recording_event.Behavior for well-formed and already-uncompressed segments is unchanged.
zlib.decompressobjreturns partial output on a truncated stream where the one-shotzlib.decompressraised, so that case is normalized back tozlib.errorto preserve the existing handling.Rollout note
The default matches the uncompressed bound Relay already applies to scrubbing-enabled orgs, so it should be a no-op for segments produced by SDKs. If it turns out legitimate segments sit closer to the limit than expected,
replays.consumer.recording.oversized_segmentwill show it and the option can be raised immediately. Worth watching that metric before considering this settled.Refs REPLAY-994
via @DominikB2014.
--
View Junior Session [Sentry]