Skip to content

feat(deploy-backup): verify database backup restores - #365

Draft
Adamskiee wants to merge 5 commits into
developfrom
feature/backup-restore-verification-338
Draft

feat(deploy-backup): verify database backup restores#365
Adamskiee wants to merge 5 commits into
developfrom
feature/backup-restore-verification-338

Conversation

@Adamskiee

Copy link
Copy Markdown
Collaborator

Overview

Closes #338.

Adds scheduled restore verification for the newest finalized SAPOT MariaDB backup. Each run inspects the dump, restores it into disposable isolated storage, validates schema and row counts, publishes sanitized state, and removes run-scoped resources.

What changed

  • adds filename-based finalized-backup selection, a bounded-memory dump inspector, and a fail-closed JSON state reader
  • adds Docker bundle and bare-metal restore verification orchestration without reading production database credentials
  • adds atomic verification status, --dry-run, --status, cleanup handling, and the db-backup-restore doctor check
  • ships daily verification service and timer units in deployment bundles
  • preserves verification timer intent across install, upgrade, and rollback paths
  • updates deployment, maintenance, monitoring, installation, and architecture documentation
  • adds Bash and Python coverage for backup selection and dump inspection

Impact

Operators gain evidence that the newest backup is restorable, separate from the existing backup freshness check. The manual production restore procedure and backup artifact format remain unchanged.

Validation

  • deploy/scripts/tests/run-tests.sh (19 Bash tests and 3 Python tests)
  • bash -n deploy/scripts/verify-db-backup.sh
  • bash -n deploy/scripts/doctor.sh
  • python3 -m py_compile deploy/scripts/lib/dump-inspector.py deploy/scripts/lib/verification-state.py
  • systemd-analyze verify deployment-scripts/sapot-db-backup.service deployment-scripts/sapot-db-backup.timer deployment-scripts/sapot-db-backup-verify.service deployment-scripts/sapot-db-backup-verify.timer
  • git diff --check

systemd-analyze also reported the host unrelated missing nix-daemon.socket and noted that RuntimeMaxSec is ignored for a oneshot service.

@Adamskiee
Adamskiee changed the base branch from main to develop August 11, 2026 07:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(server-ops): automate backup restore verification

1 participant