fix(restore): stream restore inputs, scope dump files per attempt, document restore 409s

[dviejokfs]

Description

Follow-ups to #1273.

1. Restores no longer load the whole backup into memory.

• Before: the MariaDB, PostgreSQL and MongoDB legacy restore helpers built an in-memory tar of the full backup before uploading it to the container. The MongoDB mongorestore sidecar paths also collected the S3 object into a Vec first. A restore needed RAM roughly the size of the database, and every in-place PostgreSQL restore of a pg_dump backup takes the legacy path.
• Now: every input is staged on the host in constant memory:
  • The S3 object is streamed chunk by chunk into a temp file owned by the attempt. The file is opened with create_new, so it never overwrites an existing one.
  • gzip is decompressed file to file on a blocking thread.
  • The file is streamed into the container with the shared container_upload helper.
• The Redis staging helpers moved into a shared restore_staging module with a typed StagingError that names the bucket, key and file.
• This also removes a production .to_str().unwrap() from the MongoDB legacy restore.

2. Dump files and containers belong to one attempt.

• Control-plane backup:
  • Before: the sidecar container and the host files were named after the backup. A retry after a failed or crashed attempt collided with the container or files the earlier attempt left behind.
  • Now: it runs on dump_capture, like the per-service engines since fix(restore): recover interrupted restores and explain restore and backup failures #1273. Each attempt gets its own sidecar name, its own directory inside the sidecar and its own host directory under <data_dir>/backups/tmp. The dump is streamed out through the Docker archive API instead of a bind mount, so Docker in a VM or on another host works.
  • stderr is copied out only on failure, and failures are typed (Export with the pg_dump stderr, EmptyDump, DumpUnreadable, Container, Upload).
• MariaDB dump:
  • Before: it wrote to a per-backup file opened with a truncating create, so a retry or a concurrent attempt could overwrite another attempt's dump.
  • Now: it streams into a directory owned by the attempt and reports typed failures.
  • A new DumpCaptureError::Exec keeps the retry policy and cancellation of the underlying exec failure.
• An attempt only ever deletes what it created. Leftovers from older binaries are left untouched rather than guessed at.

3. The start-restore 409 is fully documented.

• The OpenAPI response previously described only the unconfirmed cross-service restore.
• It now also covers restore-already-active (with active_restore_run_id) and the "backup is being deleted" conflict.
• Clients were regenerated from a server built off this branch: spec:update + generate:api for the CLI, openapi-ts for the web. Each changed by exactly one line.

Scalability

Control plane only, no hot-path code.

• Restores: memory goes from O(backup size) to O(chunk); disk use is one staged copy (plus the decompressed copy for gzip inputs) in a temp dir removed on every exit path.
• Control-plane backup: the dump now briefly lives in the sidecar's writable layer before it is copied to the host. That is the same trade-off the per-service dump_capture engines already make, in exchange for no host bind mount.

Type of change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation update

Checklist

• I have written tests that cover the changes
• All new and existing tests pass (cargo test --lib)
• cargo check --lib passes with no warnings
• My commits follow the Conventional Commits format
• Every commit is signed off (git commit -s) per the DCO
• I have updated documentation where necessary

Verification

All runs used --features temps-providers/docker-tests,temps-backup/docker-tests against a real Docker daemon:

• cargo clippy -p temps-backup -p temps-providers --lib --tests -- -D warnings: clean.
• cargo test --lib -p temps-backup: 338 passed.
• New Docker tests, all passing (none skipped):
  • PostgreSQL: plain SQL and pg_dump -Fc restores, streamed from staged files into a postgres:17-alpine container.
  • MongoDB: the existing S3 round-trip test now also restores a legacy mongodump archive through restore_from_legacy.
  • Control plane: a real dump over host networking with no bind mount. Roles come first, excluded tables keep their schema but not their data, and the sidecar and attempt dir are removed. A retry ignores an old-layout container and old files left by an earlier attempt.
  • MariaDB: two concurrent attempts of one backup each keep their own dump. Failed attempts (non-zero exit, empty dump, missing container) remove only their own file and are classified correctly.
  • Staging: gunzip round-trip, a corrupt gzip error that names the backup, and refusal to overwrite an existing staging file.
• cargo test --lib -p temps-providers: 830 passed. Three tests that this branch doesn't touch fail only on a macOS + Colima host:
  • test_init_persists_actual_port_after_conflict_retry: a host TcpListener does not occupy the port inside Colima's VM, so no retry is forced.
  • the two cluster_integration_tests monitor-health tests: they time out through the VM.
  • They exercise init and the cluster code, neither of which this PR changes; CI runs them on Linux.
• bun run spec:check: canonical. python3 scripts/source_attribution.py check: passes.

Follow-ups noticed, not changed here

• MariaDB DUMP_SHELL (from reading the code, not reproduced): if the database-list query fails, for example because of a wrong root password, the shell prints a plain-text -- No user databases to dump and exits 0. That output would be accepted as a non-gzip "dump".
• PostgreSQL restore_backup_file does not check psql's exit code.
• Cancelling a MariaDB dump does not stop the docker exec process inside the service container.

Related issues

Follow-up to #1273 (#1236, #1237, #1243).

🤖 Generated with Claude Code

[dviejokfs]

@greptile-apps review

[github-actions]

📓 Changelog preview

This is what your commits will add to the generated CHANGELOG.md at release time (via git-cliff). Do not edit CHANGELOG.md by hand — it is generated from your Conventional Commit messages.

## [Unreleased]

### Documentation

- **api:** Document every 409 the start-restore endpoint returns

### Fixed

- **restore:** Stream legacy restore inputs instead of buffering them
- **backup:** Scope control-plane and MariaDB dump files to each attempt
- **restore:** Remove partial MongoDB archives and bound uploads by stalls
- **restore:** Give the daemon time to answer after an upload's last byte

[greptile-apps]

Confidence Score: 5/5

[High risk] Refactors backup and restore streaming to use disk staging.

The PR appears safe to merge based on the changes reviewed.

Summary

The PR streams restore inputs through attempt-owned staging files, scopes control-plane and MariaDB dumps to individual attempts, and documents the start-restore 409 responses. The changes since the previous review give Docker uploads a separate, size-dependent window for the daemon’s response.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Stream upload body] --> B{Body exhausted?}
  B -- No --> C[Apply progress-based stall timeout]
  C --> A
  B -- Yes --> D[Wait for daemon response]
  D --> E[Apply size-dependent confirmation timeout]

Loading

_{Reviews (3) · Last reviewed commit: "fix(restore): give the daemon time to an..."}

[dviejokfs]

@greptile-apps review

[dviejokfs]

@greptile-apps review