Skip to content

Add end-to-end tests for every option against PostgreSQL 18 and MinIO - #35

Merged
itbm merged 36 commits into
masterfrom
claude/vibrant-newton-4n6mke
Oct 20, 2026
Merged

Add end-to-end tests for every option against PostgreSQL 18 and MinIO#35
itbm merged 36 commits into
masterfrom
claude/vibrant-newton-4n6mke

Conversation

@itbm

@itbm itbm commented Oct 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

Adds an end-to-end test suite that builds the image and runs it the way users do, against real services on a private Docker network:

  • PostgreSQL 18, matching the image's pg_dump 18.6 (one test also backs up and restores PostgreSQL 16).
  • MinIO over HTTP, and over HTTPS with a throwaway CA.
  • A small server that receives URL hooks.

There are 228 tests in 11 files, one file per feature area. They cover every option in the Dockerfile and README. Restores are checked against a fingerprint of every table, view, function, sequence, index, trigger and large object, so a restore must be exact to pass.

The suite needs only bash and Docker. CI runs each case file as a parallel job, so a full run takes about 4 minutes.

Known bugs

The tests found 41 product bugs. Each is a test for the correct behaviour, marked xfail, so the suite is green on master today. When a fix lands, its test passes and the run fails with XPASS until the marker is removed. That way each fix gets recorded.

Area Bug Severity Fixed by #34
Restore A plain restore whose SQL fails still exits 0 and reports success High Yes
Restore A failed DROP_DATABASE only warns, then restores on top of the existing database High Yes
Restore An unsupported backup file still drops the database when DROP_DATABASE=yes High Yes
Retention A future threshold (DELETE_OLDER_THAN="7 days", missing "ago") deletes the new backup and everything else, and exits 0 High Partly: the new backup is kept
Scheduler A run hangs for ever when its output has a line over 64 KiB (aws progress on a ~475 MB upload) High Yes
Scheduler COMMAND_TIMEOUT kills only the shell: pg_dump, aws and hooks keep running and pile up, cleanup and error hooks never run High Yes
Scheduler The end of each run's output can be lost Low Yes
Empty values POSTGRES_DATABASE="" backs up or restores the wrong database and reports success High Yes
Empty values DELETE_OLDER_THAN="" deletes every backup older than midnight today High Yes
Empty values ENCRYPTION_PASSWORD="" encrypts with an empty password High Yes
Empty values SCHEDULE, POSTGRES_PORT or S3_ENDPOINT set to "" (as in the README's Kubernetes example) breaks every run Medium Yes
Empty values S3_BUCKET, POSTGRES_HOST, POSTGRES_USER or POSTGRES_PASSWORD set to "" pass validation and fail late, after hooks fire Low Yes
Empty values BACKUP_FILE="" given to restore.sh drops the database and restores nothing Medium Yes
Empty values AWS_CA_BUNDLE="" silently turns off certificate checks Medium No
Empty values COMPRESSION_CMD="" or DECOMPRESSION_CMD="" makes every plain backup or restore fail Low No
S3 prefix S3_PREFIX that is empty, or has a leading or trailing slash, writes // keys; a trailing slash fails every backup on MinIO, and retention breaks Medium Yes
Hooks A pre-backup hook (unprivileged) can plant symlinks in /tmp so that root overwrites another file Medium Yes
Hooks HOOK_INHERIT_ENV=yes drops HOOK_EVENT and HOOK_EXIT_CODE Medium No
Encryption AES-CBC has no integrity check, so a wrong key "decrypts" about 1 time in 256: about 1 legacy backup in 256 cannot be restored, and a wrong password with DROP_DATABASE=yes drops the database about 2 times in 256 High No
Backup A database name starting with - is read as pg_dump options, so another database is backed up under that name Medium No
Backup A database name containing =, or a POSTGRES_USER containing a space, breaks the dump Low No
Restore A custom-format restore into PostgreSQL 16 or older exits 1 although the data is restored (pg_restore 18 sends SET transaction_timeout) Medium No
Retention Keys with leading, trailing or repeated spaces are mixed up, and a key containing PRE is never deleted Low No

Results against #34

I ran the whole suite against an image built from #34 (62aad10):

Merging this first and then updating #34 from master would let #34's CI show exactly this.

How the tests were checked

  • Mutation testing: more than 250 deliberate breakages of the scripts, the Dockerfile and go-cron. Every one is caught by at least one test, apart from a few that change nothing observable; those are noted in the commit messages.
  • Stability: each case file was run at least twice on master's image and once on Fix backup bugs, harden runtime and add essential features #34's, and the whole suite twice (187 passed, 41 expected failures, 0 skipped).
  • Outcomes, not wording: tests check exit codes, S3 contents and data, not temporary paths or exact log wording, so a refactor like Fix backup bugs, harden runtime and add essential features #34 doesn't break them for the wrong reasons.

Notes

  • MinIO no longer publishes its own images (the Docker Hub repository is gone, and Quay needs a login), so the suite uses Chainguard's free build, cgr.dev/chainguard/minio.
  • Run it locally with test/e2e/run.sh, optionally with a name filter. Guide: test/e2e/README.md.
  • Make End-to-end tests the required check in branch protection. It passes only when every per-file job passes, and a cancelled run counts as a failure.
  • Things the README doesn't define, so no test pins them:
    • TZ has no effect (the image has no time zone data).
    • HOOK_DROP_PRIVS=no runs hooks as root and is undocumented.
    • A CA mounted in /usr/local/share/ca-certificates stops the container on a read-only root filesystem.
    • S3_S3V4 has no effect with AWS CLI v2.
    • AWS_REGION overrides S3_REGION.
    • The README's scheduling link describes 6-field cron expressions, but only 5 fields are accepted.

🤖 Generated with Claude Code

https://claude.ai/code/session_01UdSRPLaq3mTA6Djdzte7C5

claude added 30 commits October 20, 2026 08:29
@claude
Add end-to-end test harness with PostgreSQL 18 and MinIO
122a0da 
Builds the image and runs it against real PostgreSQL 18, MinIO (HTTP and
HTTPS with a throwaway CA) and a URL hook receiver on a private Docker
network. Needs only bash and Docker. Includes a smoke test that backs up
and restores a seeded database and compares fingerprints of both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UdSRPLaq3mTA6Djdzte7C5
@claude
Run hook unit tests and end-to-end tests in CI
2304604 
Adds a Test workflow for pull requests and pushes to master, a guide to
the end-to-end suite, and a Testing section in the README.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UdSRPLaq3mTA6Djdzte7C5
@claude
Add end-to-end tests for script hooks and scheduled backups
edbb4d7 
Script hooks: event order, arguments, uid, environment scrubbing,
HOOK_INHERIT_ENV, HOOKS_DIR, fail-closed behaviour and error hooks.

Scheduled backups: @every and cron expressions, invalid schedules,
COMMAND_TIMEOUT parsing and enforcement, overlap skipping, failures,
docker stop and BACKUP_FILE precedence.

Known bugs are marked xfail: COMMAND_TIMEOUT leaves child processes
running, a scheduled run hangs on an output line over 64 KiB, the end of
a run's output can be lost, and HOOK_INHERIT_ENV=yes drops HOOK_EVENT
and HOOK_EXIT_CODE.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UdSRPLaq3mTA6Djdzte7C5
@claude
Add end-to-end tests for restore
73fb5c7 
Plain and custom-format round trips checked against fingerprints of the
source database: new, existing and renamed targets, CREATE_DATABASE,
DROP_DATABASE (including with other sessions connected), quoted names,
PARALLEL_JOBS, DECOMPRESSION_CMD, pg_dumpall into a fresh server, deep
BACKUP_FILE prefixes, POSTGRES_EXTRA_OPTS and the error paths.

Marks as xfail a known bug: with DROP_DATABASE=yes, a backup file with
an unsupported extension still drops the target database.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UdSRPLaq3mTA6Djdzte7C5
@claude
Add end-to-end tests for backup
7f2826b 
Plain, custom and pg_dumpall formats; COMPRESSION_CMD with pigz,
arguments and failures; key naming and UTC timestamps; non-default
POSTGRES_PORT; Docker link variables; POSTGRES_EXTRA_OPTS and
POSTGRES_EXTRA_DUMP_OPTS; pipefail; upload failures; database names
needing quoting; and a backup of an older (PostgreSQL 16) server.

Known bugs are marked xfail: a database name starting with a hyphen is
parsed as pg_dump options, a name containing '=' is treated as a
connection string, and a POSTGRES_USER with a space breaks the command.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UdSRPLaq3mTA6Djdzte7C5
@claude
Remove PostgreSQL data volumes when e2e containers are removed
0ef0a1b 
The postgres image declares a volume, so docker rm -f without -v left
one anonymous volume behind per server, filling the disk over repeated
local runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UdSRPLaq3mTA6Djdzte7C5
@claude
Add end-to-end tests for encryption
986930e 
Encrypted plain and custom-format backups (only the .enc object is
uploaded, it decrypts with the README's manual command, and it restores
exactly), wrong and missing passwords, passwords with special
characters, unencrypted backups restored with a password set, and
legacy (pre-PBKDF2) backups.

Known bugs are marked xfail, with fixed salts so they reproduce every
time. AES-CBC has no authentication, so a wrong key passes the padding
check about 1 time in 256: about 1 legacy backup in 256 cannot be
restored, and a wrong password with DROP_DATABASE=yes drops the
database about 2 times in 256.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UdSRPLaq3mTA6Djdzte7C5
@claude
Add end-to-end tests for S3 options
da5ed2e 
S3_PREFIX (default, nested, empty), S3_REGION checked against the
signing region in MinIO's request trace, S3_S3V4, credentials from S3_*,
AWS_* variables, shared credentials files, profiles and session tokens,
precedence between them, wrong or missing credentials and buckets,
unreachable endpoints, AWS_ENDPOINT_URL, checksum settings, a multipart
round trip over 8 MB, and keys containing spaces.

Known bugs are marked xfail: an empty S3_ENDPOINT breaks every aws call,
an empty S3_PREFIX uploads to s3://bucket//file and makes retention fail
with exit 2, and retention mangles keys with repeated spaces.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UdSRPLaq3mTA6Djdzte7C5
@claude
Add end-to-end tests for TLS to S3
1c676f8 
Against MinIO over HTTPS with a throwaway CA: no CA, S3_SSL_VERIFY=yes,
S3_CA_BUNDLE (including multi-certificate bundles and paths with
spaces), CAs mounted in /usr/local/share/ca-certificates (including in
scheduled mode and ignoring non-.crt files), AWS_CA_BUNDLE, the
precedence between them, a wrong or missing CA file, a hostname
mismatch, and S3_SSL_VERIFY=no for backup, retention and restore.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UdSRPLaq3mTA6Djdzte7C5
@claude
Add end-to-end tests for retention (DELETE_OLDER_THAN)
1e78d48 
Old objects are deleted and the new backup kept, the documented
'N days ago' form, an exclusive threshold, only objects directly in the
prefix are touched (not sub-prefixes or sibling prefixes), any file in
the prefix is deleted as the README warns, keys with spaces, nested
prefixes, TZ, scheduled runs, and failures of the upload, listing or
delete. Objects are backdated with MinIO's X-Minio-Source-Mtime header.

Known bugs are marked xfail: a future threshold (for example '7 days'
without 'ago') deletes the new backup and everything else while the run
exits 0, an empty DELETE_OLDER_THAN deletes everything before midnight,
keys differing only in whitespace are mixed up, and a key containing
' PRE ' is never deleted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UdSRPLaq3mTA6Djdzte7C5
@claude
Add end-to-end tests for URL hooks
2306c66 
Request order for every HOOK_*_URL on backup and restore success and
failure, no requests after validation errors, URLs kept out of logs,
HTTP refused unless HOOK_ALLOW_HTTP=yes, refusal of other schemes,
credentials, whitespace, loopback, link-local and metadata hosts before
any request, 403 failing the job, 500 retried three times, redirects not
followed, untrusted certificates failing closed, and a failing error URL
keeping the original exit code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UdSRPLaq3mTA6Djdzte7C5
@claude
Add end-to-end tests for validation, dispatch and empty values
e453879 
Required variables for backup and restore, the S3 key pair, Docker link
variables, run.sh dispatch (one-off backup, BACKUP_FILE winning over
SCHEDULE), the README environment table matching the Dockerfile, and
the image's tools, hook user and /hooks permissions.

Also checks every variable set to an empty string, as the README's
Kubernetes example does. Empty values that do not behave like unset are
marked xfail: POSTGRES_DATABASE (backs up or restores into the wrong
database and exits 0), DELETE_OLDER_THAN, ENCRYPTION_PASSWORD,
AWS_CA_BUNDLE (turns off certificate checks), SCHEDULE, POSTGRES_PORT,
S3_ENDPOINT, COMPRESSION_CMD, DECOMPRESSION_CMD, and BACKUP_FILE given
to restore.sh.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UdSRPLaq3mTA6Djdzte7C5
@claude
Run each end-to-end case file as a parallel CI job
6378eee 
The single job had grown to 15 minutes. Case files are discovered at run
time, so a new file cannot be skipped by accident, and an
"End-to-end tests" job still reports one overall result for branch
protection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UdSRPLaq3mTA6Djdzte7C5
@claude
Strengthen script hook tests found weak by mutation testing
b9cf527 
46 deliberate breakages of hooks.sh, backup.sh, restore.sh, the
Dockerfile and go-cron were run against the file; 43 were caught. The
three survivors are now caught too:

- post-backup must get BACKUP_DEST_FILE and BACKUP_S3_URI for a default
  plain backup, not only for encrypted ones
- the same for an unencrypted custom-format (.dump) backup
- a scheduled run's hook output must reach the container log on both
  stdout and stderr

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UdSRPLaq3mTA6Djdzte7C5
@claude
Strengthen scheduled backup tests after mutation testing
4881681 
19 deliberate breakages of go-cron, run.sh and backup.sh were run
against the file, plus a build with the three known bugs fixed (which
turns their xfails into XPASS). All are caught. Changes:

- the repeated-run test changes the data between runs and restores the
  newest backup, proving scheduled runs dump fresh data
- the timeout test blocks its hook on a marker file and uses an 8 s
  timeout, leaving headroom for slower CI runners
- the 6-field cron case is no longer pinned as invalid: the README links
  to docs that describe 6-field expressions, so which is right is a
  documentation question

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UdSRPLaq3mTA6Djdzte7C5
@claude
Strengthen restore tests after mutation testing
18f6209 
30 deliberate breakages of restore.sh and the image defaults were run
against the file; 24 were caught. The five real survivors are now
caught:

- PARALLEL_JOBS logged but -j not passed to pg_restore, or hard-coded:
  a login event trigger now counts pg_restore sessions (PostgreSQL 17+;
  skipped with a note on older servers)
- errors from a parallel pg_restore swallowed
- the 'all' restore ignoring DECOMPRESSION_CMD, or swallowing pipeline
  errors: a bzip2 'all' script must fail with gunzip and succeed with
  bunzip2

Also lengthens the held session in the DROP ... WITH (FORCE) test so a
slow runner cannot make it vacuous, and fails fast on setup errors.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UdSRPLaq3mTA6Djdzte7C5
@claude
Strengthen backup tests after mutation testing
266631f 
25 deliberate breakages of backup.sh were run against the file; 19 were
caught. The six survivors are now caught:

- custom format and pg_dumpall ignoring POSTGRES_PORT: the non-default
  port test now also takes custom-format and 'all' backups
- custom format ignoring POSTGRES_EXTRA_OPTS, or getting an unquoted
  database name
- custom format written uncompressed: the archive header must show
  compression
- pg_dumpall ignoring COMPRESSION_CMD

The Docker link test also sets POSTGRES_PORT=tcp://... as Docker does,
checking that the link variables override it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UdSRPLaq3mTA6Djdzte7C5
@claude
Strengthen encryption tests after mutation testing
5473499 
27 deliberate breakages of backup.sh, restore.sh and the image's openssl
were run against the file; 25 were caught here and one more by the
script hook tests. The survivor, openssl defaulting to another message
digest, is now caught: the README's manual decrypt and the legacy
fixtures name -md sha256, which is what standard OpenSSL 1.1+ and every
pre-PBKDF2 release used, so a digest change in the image breaks the
tests instead of users' backups.

Adds a check that the fixed-salt PBKDF2 fixtures used by the
wrong-password xfails restore with the right password, so those xfails
can only fail on the bug itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UdSRPLaq3mTA6Djdzte7C5
@claude
Strengthen S3 option tests after mutation testing
1cc6892 
33 deliberate breakages of backup.sh, restore.sh and run.sh were run
against the file; 30 were caught here and one more by the retention
tests. The others are now caught:

- set -x in restore.sh printing the secret key: a failed restore's log
  must not contain it
- an empty S3_ACCESS_KEY_ID or S3_SECRET_ACCESS_KEY treated as set: the
  empty-values test now also restores
- a secret key without a key id accepted on restore

The empty-prefix retention xfail now also checks the key is at the
bucket root, so a wrong fix cannot pass it. The repeated-spaces
retention xfail is removed as a duplicate of the one in retention.sh.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UdSRPLaq3mTA6Djdzte7C5
@claude
Strengthen TLS tests after mutation testing
a718063 
22 deliberate breakages of run.sh, backup.sh, restore.sh and go-cron
were run against the file; 14 were caught. Five more are now caught:

- S3_CA_BUNDLE applied only to one-off runs, not scheduled ones (new
  scheduled S3_CA_BUNDLE test; go-cron stripping AWS_* is also caught)
- the CA directory pointing AWS_CA_BUNDLE at one file, or replacing the
  public roots instead of adding to them: three CAs are mounted and the
  merged bundle must hold them all alongside the public roots
- DROP_DATABASE moved before the download: a restore rejected by TLS
  must leave the existing database untouched

The three survivors are a value the README does not define
(S3_SSL_VERIFY other than yes or no), a cleanup path only the restore
tests can reach, and an export that has no effect on this image.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UdSRPLaq3mTA6Djdzte7C5
@claude
Strengthen retention tests after mutation testing
c2a5619 
23 deliberate breakages of backup.sh, restore.sh and the image default
were run against the file; 19 were caught. Three more are now caught:

- a default DELETE_OLDER_THAN in the image: new test that retention is
  off by default and old objects survive
- read without -r mangling keys with backslashes
- the sub-prefix skip widened to any key containing PRE

The survivor (parsing the threshold in UTC) gives the same result for
every documented threshold form.

Also stops three tests asserting the '<new backup> not older than' log
line, which pinned an implementation detail rather than the outcome
(the new backup is still checked in S3), and gives the TZ test a valid
POSIX zone: musl read 'E2E-14' as UTC-2 plus a DST zone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UdSRPLaq3mTA6Djdzte7C5
@claude
Strengthen validation and image tests after mutation testing
892b0ed 
22 deliberate breakages of the scripts and image were run against the
file; 14 were caught. The other eight are now caught: blank S3 key
pairs on restore, HOOK_DROP_PRIVS defaulting to no or baked into the
image, HOOK_ALLOW_HTTP or DROP_DATABASE defaulting to yes, a setuid
su-exec, and a world-writable /usr/local or /usr/local/bin.

The README table is now checked against the built image's defaults
rather than a parse of the Dockerfile, and the image test no longer
pins the scripts' directory or the PostgreSQL major version beyond
what the README states.

Adds an xfail for a new bug: a pre-backup hook running as the
unprivileged user can plant symlinks at /tmp/dump.sql.gz.enc or
/tmp/s3-listing so that root overwrites another file. It skips where
fs.protected_symlinks is on, as on most Ubuntu hosts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UdSRPLaq3mTA6Djdzte7C5
@claude
Drop each case's databases, log hook request headers
f0f7f18 
- run.sh drops the databases a case created on the shared server when
  the case ends (names starting with its t_<slug> prefix, plus any made
  with new_db_named), so disk use no longer grows with every case
- psql_exec no longer reads stdin, so it is safe inside while-read loops
- the hook receiver also logs each request's headers, for checking that
  URL hooks send no secret headers (hook_request_headers)
- README: call xfail just before the step that exposes the bug, so a
  setup failure shows as FAIL; assert outcomes rather than temporary
  paths or internal wording; use docker rm -v after E2E_KEEP=1

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UdSRPLaq3mTA6Djdzte7C5
@claude
Cover restore-side URL hook failures and secret-free requests
ab90037 
- a failing HOOK_PRE_RESTORE_URL stops the restore before the download
  and before DROP_DATABASE, leaving the database untouched
- a failing HOOK_POST_RESTORE_URL fails the job and fires the
  restore-error URL
- a failing HOOK_RESTORE_ERROR_URL keeps the original exit code
- hook requests are GETs without a body, credentials or secret values
  (README), checked against the receiver's header log
- rename the validation test after the single error it exercises

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UdSRPLaq3mTA6Djdzte7C5
@claude
Extend scheduled backup tests and make them robust to rewording
b5f0159 
- new xfail: a run stopped by COMMAND_TIMEOUT must still run cleanup
  (backup-error script hook and URL, no dump left in /tmp); today
  go-cron SIGKILLs the shell, so its EXIT trap never runs
- '* * * * *' really fires (the only minute-long test)
- @monthly, @Yearly and @annually are accepted
- COMMAND_TIMEOUT does not affect one-off backups, as the README says
- timeout tests match 'timed out after Ns' rather than the full log
  line, so a reworded message still passes
- xfail calls moved to just before the step that exposes each bug

Against PR #34 the file has no failures; its four xfails pass there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UdSRPLaq3mTA6Djdzte7C5
@claude
Extend restore tests: SQL errors, failed drops, older servers
a30915d 
New tests:
- xfail: a plain restore whose SQL fails (target already populated)
  must not exit 0 and report success
- xfail: a DROP_DATABASE that fails must fail the restore rather than
  restore on top of the existing database
- restore with a non-default POSTGRES_USER (password with spaces and
  quotes) and POSTGRES_PORT, using parallel pg_restore
- a backup of a PostgreSQL 16 server restores back into it: plain
  passes; custom is an xfail because pg_restore 18 sends
  'SET transaction_timeout', which PostgreSQL 16 rejects, so the run
  exits 1 although the data is restored
- POSTGRES_EXTRA_DUMP_OPTS is ignored by restores, as the README says

The create-when-exists test now asserts the outcome instead of the
warning text, the unsupported-format test is renamed for what it
checks, and xfail calls sit just before the step that exposes each bug.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UdSRPLaq3mTA6Djdzte7C5
@claude
Reject blank required variables in validation tests
a0d7409 
New xfails: S3_BUCKET, POSTGRES_HOST, POSTGRES_USER and
POSTGRES_PASSWORD set to an empty string (as the README's Kubernetes
example shows them) must be rejected like unset ones, for backup and
restore: exit 1, the usual message, no dump or download, no hooks.
Today they pass validation, hooks fire, and the job fails later with a
misleading error.

The docs check now accepts a README row without an ENV only for a
variable the scripts read (such as TZ), instead of pinning how a
default is provided. The image test checks only that go-cron runs and
mentions itself, not its exact usage text. The dispatch test is removed
as a duplicate of test_schedule_backup_file_restores_once, and xfail
calls sit just before the step that exposes each bug.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UdSRPLaq3mTA6Djdzte7C5
@claude
Test slashed S3_PREFIX values and S3_BUCKET sub-paths
7edd660 
New tests:
- S3_BUCKET with a sub-path (the README calls it a bucket path): backup
  key, BACKUP_S3_URI, retention and restore all work
- xfails for S3_PREFIX with a trailing slash ('backups/'): every backup
  fails against MinIO (XMinioInvalidObjectName) because the key has
  '//', and AWS S3 would store such keys
- xfails for a leading slash ('/backups'): the request path and
  BACKUP_S3_URI contain '//', and retention lists a prefix that matches
  nothing, so the backup exits 2 after uploading
- the custom-region trace now also covers retention's list and delete

Removes four tests duplicated in validation.sh, retention.sh, smoke.sh
and backup.sh. Against PR #34 the file has no failures; its seven
prefix xfails pass there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UdSRPLaq3mTA6Djdzte7C5
@claude
Make encryption tests path-independent; cover encrypted 'all' backups
ad53a53 
- an encrypted POSTGRES_DATABASE=all backup is stored only as
  all_<ts>.sql.gz.enc, decrypts with the README's command, and restores
  both databases, the role and its ownership into a fresh server
- tests no longer pin the temporary file in the 'Encrypting ...' log
  line; the .enc object, its Salted__ header and the round trips prove
  encryption
- the encryption-failure test uses a stand-in openssl that fails 'enc',
  instead of blocking a fixed temporary path
- the legacy xfail is renamed for the behaviour it expects, and xfail
  calls sit just before the step that exposes each bug

All three tests that failed against PR #34 only because of the old
temporary path now pass on both images.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UdSRPLaq3mTA6Djdzte7C5
@claude
Check backup-error scrubbing and directory hooks; fix a stop race
eb0e7d9 
- the error-event scrubbing test now also covers backup-error, with
  ENCRYPTION_PASSWORD, AWS_SESSION_TOKEN and AWS_ROLE_ARN set, and
  HOOK_EXIT_CODE=2
- new test: a directory at $HOOKS_DIR/<event> (which passes a plain -x
  check) fails the job closed with nothing uploaded
- the scheduled-hooks test judges only events recorded before it stops
  the container. On PR #34 docker stop forwards SIGTERM to the running
  backup, which then correctly fires backup-error; the old check failed
  whenever the stop landed mid-run
- the HOOK_INHERIT_ENV xfail moves to just before the step that exposes
  the bug

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UdSRPLaq3mTA6Djdzte7C5
claude added 6 commits October 20, 2026 19:11
@claude
Remove duplicate backup tests and move xfail calls
5c81eed 
- removes three tests covered elsewhere: the Docker link fallback and
  the required-settings checks (validation.sh covers both, with
  restores and hook checks), and the single --exclude-table test
  (test_backup_extra_dump_opts_several_options asserts the same)
- xfail calls move to just before the backup that exposes each bug
- databases made by the quoting test are recorded for clean-up, since
  their names do not share the case prefix

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UdSRPLaq3mTA6Djdzte7C5
@claude
Test retention over TLS with a trusted CA
67f3622 
- new test: with S3_CA_BUNDLE and verification on, DELETE_OLDER_THAN
  deletes an old object on minio-tls and keeps a newer one and the new
  backup (only S3_SSL_VERIFY=no exercised retention over TLS before)
- renames the explicit S3_SSL_VERIFY=yes test for what it proves
- drops per-test database clean-up helpers now that run.sh does it

Passes on both the master and PR #34 images.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UdSRPLaq3mTA6Djdzte7C5
@claude
Remove duplicate retention xfail and move xfail calls
f9e1e85 
The empty-threshold xfail is covered by validation.sh's
test_validation_empty_delete_older_than_deletes_nothing, which keeps
all the empty-value cases together. The remaining xfail calls move to
just before the backup that exposes each bug.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UdSRPLaq3mTA6Djdzte7C5
@claude
Count assertion failures inside $(...); explain cancelled CI runs
908ec70 
- fail() records the failure in the case directory, because inside a
  command substitution its exit ends only that subshell. run.sh reports
  such a case as FAIL, and one that failed before reaching its xfail
  step as FAIL rather than as an expected failure
- the overall End-to-end tests job now says when it failed because the
  per-file jobs were cancelled (usually by a newer push); it still
  fails, so a cancelled run never satisfies a required check
- README: note the $(...) behaviour and when to add '|| exit 1'

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UdSRPLaq3mTA6Djdzte7C5
@claude
Fix a timestamp race that made a validation test skip
b12cd17 
_val_put_aged computed '3 days ago' twice, once for the upload and once
for the expected Last-Modified. When a second boundary fell between the
two, they differed by a second and the test skipped as if MinIO could
not backdate objects. It now computes the timestamp once, as the
retention helper does; five runs on each image gave the same result.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UdSRPLaq3mTA6Djdzte7C5
@claude
Give MinIO a fixed console port in the e2e stack
27a0d88 
Without --console-address MinIO picks a random console port at start-up
and exits if that port is taken by the time it binds ('Specified port is
already in use'), which stopped one CI job before any test ran. Both
MinIO containers now use :9001 inside their own network namespace.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UdSRPLaq3mTA6Djdzte7C5
@itbm itbm changed the title Add end-to-end test harness with PostgreSQL 18 and MinIO Add end-to-end tests for every option against PostgreSQL 18 and MinIO Oct 20, 2026
@itbm
itbm merged commit 529d015 into master Oct 20, 2026
14 checks passed
@itbm
itbm deleted the claude/vibrant-newton-4n6mke branch October 20, 2026 20:32
@itbm itbm mentioned this pull request Oct 20, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@itbm @claude