Skip to content

fix: make server-side copies O(1) memory (stop Scylla backup 503 storm)#113

Merged
ServerSideHannes merged 1 commit into
mainfrom
fix/copy-o1-memory-fixed-part-size
Jul 7, 2026
Merged

fix: make server-side copies O(1) memory (stop Scylla backup 503 storm)#113
ServerSideHannes merged 1 commit into
mainfrom
fix/copy-o1-memory-fixed-part-size

Conversation

@ServerSideHannes

Copy link
Copy Markdown
Owner

The real fix (root cause, not a bandaid)

Follow-up to #112. That patch (writer preference) treated a symptom; this fixes the cause of the Scylla backup 503 storm.

What's actually happening in prod

Pulled from the live pods (2026.7.8, 30 replicas, 1Gi limit, governor budget 192MB):

  • ~50% of requests 503 (296 × 503 vs 304 × 200 in a 4-minute window).
  • Every rejection is a copy: MEMORY_CLAMPED_TO_BUDGET copy=true requested_mb=534.83 (276×), MEMORY_REJECTED at active_mb 0.06–0.31.
  • Every failing request is partNumber=1 (291/291) — a single-part UploadPartCopy (Scylla Manager dedup copies each SSTable as one object).

Why

The copy pipeline sized its internal S3 parts as object_size / MAX_INTERNAL_PARTS_PER_CLIENT (= object/20 = 238MB for a 4.7GB SSTable), so copy_pipeline_peak535MB — larger than the 192MB budget. copy_governor_clamped_reserve then clamped the reservation to the whole budget, making the copy exclusive: admission requires active_bytes + limit ≤ limit, i.e. active_bytes == 0. Under steady backup traffic the baseline reservations of other in-flight requests keep active_bytes > 0 forever, so every large copy backpressures the full timeout → 503 → retry → same.

Writer preference (#112) can't fix this: it holds back new small requests but can't evict baselines already held, and a single copy deadlocks even against its own request-gate reservation.

The fix

Copies frame at a fixed internal part size (crypto.copy_internal_part_size, 32MB) instead of object/20, so the copy peak is O(1) in object size (~90MB) no matter how big the object is. A copy now fits the budget outright — never clamps, never exclusive, just a ~90MB reservation like anything else. Multiple copies run concurrently, bounded by the budget (so the multi-copy OOM that motivated exclusivity in #110 can't happen either — each copy is light).

A large copy needs many internal parts, so they're allocated sequentially (client_part_number=0) rather than from the fixed 20-wide per-client window. Safe because:

  • Copies are single-part uploads (verified 291/291 partNumber=1).
  • Reassembly is metadata-driven (get.py sorts by internal_part_number and uses stored ciphertext sizes), so smaller/more parts + sequential numbering read back byte-identically — no migration.
  • Part size only grows past S3's 10,000-part ceiling (>288GB objects; backup SSTables never approach it).

Tests

Rewrote the copy-governor tests that hard-coded the old "monopolize the budget" design to assert the new invariants:

  • test_copy_peak_is_o1_across_object_sizes — peak identical from 33MB to 100GB, all < 128MB.
  • test_copy_does_not_monopolize_budget, test_multiple_copies_run_concurrently_bounded_by_budget, test_copy_coexists_with_scylla_upload.
  • test_large_copy_admitted_while_baseline_reservations_held — reproduces the exact prod deadlock (copy + baseline reservations held), verified to fail against the old O(size) sizing and pass with the fix.
  • Existing framed multi-frame roundtrip, large-copy roundtrip, copy passthrough and integration copy-governor tests all pass unchanged (read-back correctness).

Full unit suite green (one pre-existing, unrelated dashboard asyncio-loop flake).

Follow-up (not in this PR)

The code fix removes the deadlock. Throughput is still capped by the 192MB budget while the pod has a 1Gi limit (maxconn 20/pod admits ~20 concurrent, but 192MB only fits ~2 copies + small traffic). Recommend bumping performance.memoryLimitMb in the argocd values toward ~640MB to use the available headroom now that copies are light and honestly accounted. That's a config change in argocd-internal-services, separate from this code PR.

Root cause of the Scylla backup 503 storm (~50% of requests): a dedup
UploadPartCopy of a 4.7GB SSTable (single client part, partNumber=1) sized its
internal S3 parts as object_size/MAX_INTERNAL_PARTS_PER_CLIENT (=object/20 =
238MB), so the copy pipeline peaked at ~535MB -- larger than the 192MB governor
budget. copy_governor_clamped_reserve then clamped the reservation to the WHOLE
budget, making the copy exclusive: it could only be admitted when active_bytes
hit exactly 0. Under steady backup traffic (baseline reservations of other
in-flight requests keep active_bytes > 0) that never happens, so every large
copy backpressured for the full timeout and returned 503 SlowDown. Confirmed in
prod: MEMORY_CLAMPED_TO_BUDGET copy=true requested_mb=534, MEMORY_REJECTED at
active_mb=0.06-0.31, 296x503 vs 304x200.

Writer preference (#112) could not fix this: it holds back *new* small requests
but cannot evict baseline reservations already held, and a single copy deadlocks
even against its own request-gate reservation.

Fix: copies frame at a FIXED internal part size (crypto.copy_internal_part_size,
32MB) instead of object_size/20, so the copy peak is O(1) in object size (~90MB)
regardless of how large the object is. A copy now fits the budget outright, never
clamps, never runs exclusive, and simply reserves ~90MB like any other request --
multiple copies run concurrently, bounded by the budget (so the multi-copy OOM
that motivated exclusivity can't happen either, because each copy is light).

A large copy needs many internal parts, so allocate them sequentially
(client_part_number=0) rather than from the fixed 20-wide per-client window.
Safe: copies are single-part uploads (verified 291/291 partNumber=1 in prod) and
reassembly is metadata-driven (get.py sorts by internal_part_number and uses
stored ciphertext sizes), so smaller/more parts and sequential numbering need no
migration and read back byte-identically. Part size grows only past S3's 10k-part
ceiling (>288GB objects, which backup SSTables never hit).

Tests: rewrote the copy-governor tests that encoded the old 'monopolize the
budget' design to assert the new invariants (peak is O(1) and size-independent,
copy never clamps, N copies run concurrently within budget, copy coexists with
uploads). Added test_large_copy_admitted_while_baseline_reservations_held which
reproduces the exact prod deadlock and is verified to fail against the old
O(size) sizing. Full unit suite green.
@ServerSideHannes ServerSideHannes merged commit 57dc8ec into main Jul 7, 2026
10 checks passed
@ServerSideHannes ServerSideHannes deleted the fix/copy-o1-memory-fixed-part-size branch July 7, 2026 07:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant