[RonaldHensbergen]

Closes #669.

Summary

• New experimental object-storage-backup module (modules-experimental/backup/object-storage): scheduled, immutable, UTC-timestamped snapshot copies of an object-storage contract's bucket to a separately configured S3-compatible destination (rclone copy, not sync, with --immutable), so source overwrites and deletions never erase earlier snapshots. Timestamped copies stand in for the S3 object versioning Garage does not implement.
• Follows the backup-module conventions from feat: add generic file/volume backup capability (Dagster IO manager, DuckDB, dlt) #668: production-only (service carries production: true; instantiating outside production fails E143), digest-pinned image, hardened service (non-root, read-only rootfs, cap_drop: ALL, no-new-privileges, pids limit, tmpfs /tmp), separate least-privilege destination credentials, and a source/destination sameness check. Every successful pass ends by writing a .cds-snapshot-complete marker into its timestamped prefix, so a prefix left by a failed pass is recognizable as partial; failed passes retry with a fresh timestamp after a configurable delay (retryDelaySeconds) instead of crash-looping. A destination configured as literally the bound source endpoint and bucket fails planning with a new E147 diagnostic before anything boots; the service's start-time guard remains as a defense-in-depth backstop.
• Explicit productionSuitable: false (plaintext snapshots, operator-managed restore/retention), so the security scan raises CDS-SEC-073 — production use requires a scoped runtime-policy waiver, documented in the README and docs/backup-restore.md. Each pass is a full copy of the current bucket contents, so destination storage grows linearly with the number of retained snapshots.
• Wiring tests cover the config schema, snapshot semantics, hardening, credential placeholder rendering, required contract binding, E143 outside production, E147 for same-source destinations, and the production security pass (CDS-SEC-073 accepted, nothing else unexpected).
• Docker-gated runtime proof (tests/test_object_storage_backup_runtime.py, runs in the Docker Runtime Smoke Test workflow): boots two real Garage instances as source and destination, seeds a marker object, waits for the scheduled snapshot under its immutable timestamped prefix, overwrites the source object, and recovers it byte-for-byte from the snapshot prefix with the documented rclone copy procedure. The same cycle was verified manually against real Garage instances before committing.

Validation

• python -m unittest tests.test_object_storage_backup_module -v (10 tests)
• python -m unittest tests.test_object_storage_backup_runtime -v — skips locally without CDS_RUN_DOCKER_SMOKE=1; the harness phases (render, boot, health, seed) were validated locally, the full proof runs in CI (this daemon cannot exec no-new-privileges containers, the same limitation documented on fix(backup): one restic snapshot per pass so multi-source restores are complete #1018)
• Full offline suite: 1940 tests, 44 Docker-gated skips, 1 pre-existing local-environment SQLAlchemy error (cds-runtime extra not installed locally; skips cleanly in CI)
• yaml.safe_load on the new module.yaml and the workflow; markdownlint on all touched docs
• Manual end-to-end proof against real Garage instances: snapshot layout <prefix>/<bucket>/<timestamp>/, overwrite, byte-for-byte restore

Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe vibe@mistral.ai

[RonaldHensbergen]

Review pass on this PR — verdict: looks good. Conventions match #668, and all checks are green including the Docker-gated runtime proof (seed -> scheduled snapshot -> overwrite -> byte-for-byte recovery). Findings below, none blocking.

What's solid

• rclone copy (not sync) + --immutable + a fresh UTC-timestamped prefix per pass is the right primitive for Garage's missing object versioning: source overwrites and deletions can never touch earlier snapshots.
• Hardening parity with the restic module: digest-pinned image, non-root, read-only rootfs, cap_drop: ALL, no-new-privileges, pids limit, tmpfs /tmp; separate least-privilege destination credentials.
• Honest productionSuitable: false with the CDS-SEC-073 waiver path documented, and the security test pins that no other finding sneaks in for this module.
• Tests assert the rendered output carries only ${CDS_*} placeholders (no secret material), E143 outside production, and the required contract binding.
• The workflow step and cleanup match the restic-proof pattern.

Findings

1. (medium) Partial prefixes after a failed pass. rclone copy failing mid-run (destination outage, network blip beyond rclone's internal --retries) leaves a partial timestamped prefix, and the destination credentials intentionally cannot delete it. On restart the retry writes a new timestamp, so the partial prefix survives as the newest recovery point — an operator restoring "latest" can silently restore a partial bucket. Since the creds can PUT, a cheap fix is a completion-marker object (e.g. <prefix>/<bucket>/<timestamp>/.cds-snapshot-complete) written after a pass succeeds, with the recovery procedure saying to restore only from prefixes carrying the marker. At minimum, document that the newest prefix may be partial.

2. (low) Crash-retry has no backoff. sh -ec makes any failed pass exit, and restart: unless-stopped re-runs the entrypoint immediately — each restart takes a fresh snapshot right away. A persistently flaky source/destination therefore produces one partial prefix per crash cycle (amplifies finding 1). rclone's built-in retries absorb most transient blips, so this is minor; a short sleep before exiting on failure would tame the loop.

3. (low) The sameness guard is lexical, not semantic. It compares literal endpoint strings plus bucket names, so the same Garage reachable under a different hostname/port (or http vs https of the same host) passes the guard. The README already tells operators to prefer a different backend and host, so best-effort is defensible — maybe one README line stating the guard is a string comparison.

4. (low, docs) The documented restore clobbers current data. The recovery procedure copies a timestamped prefix straight back over the source bucket; the runtime test itself proves this silently overwrites newer objects (V2 replaced by V1). The README's recovery section could suggest a --dry-run first or restoring into a scratch bucket/prefix before overwriting live data.

5. (medium, concrete) .gitignore misses the generated proof compose file. tests/test_object_storage_backup_runtime.py writes docker-compose.objstore-proof.yml at the repo root, but .gitignore lists only docker-compose.vault-proof.yml, docker-compose.restore-proof.yml, and docker-compose.restic-proof.yml. If the test is killed before its addCleanup unlink runs, the file lingers as untracked and risks an accidental commit. Add docker-compose.objstore-proof.yml next to the other three.

6. (nit) The finally: pass block at the end of test_snapshot_recovers_overwritten_object is dead code — drop it.

7. (nit) The interval path is never exercised by CI: the proof covers the first immediate snapshot, but the sleep "${SNAPSHOT_INTERVAL_HOURS}h" suffix parsing (busybox) only runs from the second pass on. Fine if the manual end-to-end proof covered a second pass; otherwise a one-off local run with intervalHours: 1 closes the gap.

[RonaldHensbergen]

All seven review findings are addressed in 5c05bfa:

1. Partial prefixes: every successful pass now writes a .cds-snapshot-complete marker object into its timestamped prefix (written last, after the copy); the README's recovery procedure says to restore only from marked prefixes, and the runtime proof only accepts a snapshot once the marker has landed.
2. Backoff: a failed pass logs the error, leaves its marker-less partial prefix in place, and retries with a fresh timestamp after five minutes inside the loop -- no more crash-restart cycling.
3. Sameness guard: the README now states the guard is a literal string comparison and that the destination choice remains operator responsibility.
4. Restore safety: the recovery section recommends --dry-run or restoring into a scratch bucket/prefix before overwriting live data, and mentions excluding the marker on restore.
5. .gitignore now lists docker-compose.objstore-proof.yml alongside the other proof compose files.
6. The dead finally: pass wrapper is gone.
7. The interval sleep is now arithmetic seconds ($((SNAPSHOT_INTERVAL_HOURS * 3600))), removing the busybox duration-suffix path CI never exercised; verified the arithmetic parses and runs in the actual rclone image's busybox ash, and that rclone rcat exists in 1.74.0.

Local validation: module suite 8/8, full offline suite 1938 tests OK, markdownlint/yamllint/ruff clean. The full Docker proof still needs the CI runner (this daemon cannot exec init/no-new-privileges containers, per #1018).

[RonaldHensbergen]

Gap closure follow-up (the "not exercised by any automated test" items from the review): all three are now covered.

Failure branch (in CI, 7fd3564 + e500311): a new runtime proof boots the stack with the destination running but the module's destination credentials wrong -- rclone does not retry auth failures, so passes fail fast and deterministically. It asserts: at least two failed passes with fresh timestamps on the configured retryDelaySeconds, restart count 0 (the service stays up -- no crash-restart loop), no pass ever logs "complete", and the destination listing stays completely empty (the wrong credentials cannot write a completion marker, or anything else). The first attempt used a destination-down outage, which taught us something useful: rclone's internal --retries ride out a transient outage (the copy spanned ~2m20s of downtime and completed into its original timestamp), so the module's in-loop backoff only engages for genuinely non-transient failures. That behavior is correct (blips absorbed, hard failures retried without crashing) and is now pinned by the proof. The marker-less partial prefix after a mid-copy failure remains inspection-covered only -- staging a deterministic mid-copy failure needs a large bucket and a timed kill, which is not CI-suitable.

Sameness guard (in CI, 7fd3564 + e500311): a new runtime proof renders a profile whose destination is literally the source endpoint and bucket, boots it, and asserts the service exits with code 1 and logs the guard's message (it crash-loops under restart: unless-stopped, which is the designed fail-fast). The first CI run failed only because the test's inspect template used .State.RestartCount; RestartCount is a top-level field. Fixed and verified.

Interval path (local simulation, not CI): CI cannot wait out the 1-hour minimum interval, so the loop-after-success path was proven locally against the rendered stack (with only this daemon's blocking no-new-privileges line stripped) with the interval set to 0: ~190 consecutive successful passes, each writing its timestamped prefix and its .cds-snapshot-complete marker, with the arithmetic sleep $((SNAPSHOT_INTERVAL_HOURS * 3600)) executing between every pass, and the seeded content recovered byte-for-byte from a late timestamp. One incidental observation: while the service snapshotted once per second, full-tree rclone lsf --recursive listings against Garage intermittently came back empty until the burst settled (per-prefix listings were always correct). At the module's real cadence this is unreachable, and the restore procedure only ever lists a single timestamp prefix, but it is worth knowing if anyone builds tooling that walks the whole snapshot tree right after a pass.

Validation: full offline suite 1938 tests OK, module suite 8/8, markdownlint/yamllint/ruff clean; CI green on e500311 including the Docker Runtime Smoke Test, which ran all three object-storage proofs in 37s.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
82.8% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud