Backup fails because of a stale Restic lock #8229

When the backup destination (for example a B2 or S3 bucket) is slow to respond, a module backup can leave a "stale lock" file in its Restic repository. Restic never removes such a lock by itself, so later backups of the same application can keep failing until an administrator runs restic unlock by hand.

Steps to reproduce

The problem is timing-dependent and was observed on a production node:

• Configure a scheduled backup to a cloud destination (B2/S3)
• During the backup, the remote storage becomes slow: requests take more than 30 seconds to complete
• Restic fails to create its lock with unexpected HTTP response (504): 504 Gateway Time-out, retries for about 15 minutes, then gives up

Save(<lock/c558447a75>) returned error, retrying after 694.836292ms: unexpected HTTP response (504): 504 Gateway Time-out
[...]
Save(<lock/c558447a75>) failed: unexpected HTTP response (504): 504 Gateway Time-out
{"message_type":"exit_error","code":1,"message":"unable to create lock in backend: unexpected HTTP response (504): 504 Gateway Time-out"}
[ERROR] restic stats failed for <destination>/nethvoice-proxy/<uuid>: Command '[...'stats', '--json', 'latest']' returned non-zero exit status 1.
Save(<lock/a64a42d591>) returned error, retrying after 1.405563479s: unexpected HTTP response (504): 504 Gateway Time-out
[...]
unable to create lock in backend: unexpected HTTP response (504): 504 Gateway Time-out
[ERROR] retention policy failed for <destination>/nethvoice-proxy/<uuid>: Command '[...'forget', '--prune', '--keep-last=5']' returned non-zero exit status 1.
nethvoice-proxy6: backup 1 failed (exit code 1)

Expected behavior

A temporary problem of the remote storage makes that backup run fail, and the next scheduled run works again once the remote is healthy.

Actual behavior

Sometimes the next runs keep failing because the repository is locked (repository is already locked exclusively). The administrator must remove the lock manually.

The 504 error is generated by the HAProxy front end of the rclone-gateway service, after its 30-second timeout server. When HAProxy gives up, the lock file upload may still be completed by rclone on the remote storage. Restic believes the lock was never created, so it never deletes it. The lock file stays in the repository:

• Restic considers a lock "stale" when it is older than 30 minutes, but it does not ignore stale locks when it acquires a new one: only restic unlock removes them, and NS8 never runs it.
• An orphaned exclusive lock (from forget --prune) blocks every later backup of that application. An orphaned shared lock (from backup or stats) blocks the retention policy, so the backup is reported as failed.

The same kind of orphaned lock can also be left when Restic is interrupted (node reboot, out-of-memory, rclone-gateway restart) while it is refreshing or removing its own lock.

Proposed fix: before each module backup, run restic unlock (without --remove-all). It removes only locks older than 30 minutes; a running Restic process refreshes its lock every 5 minutes, so active locks are never removed.

As a separate improvement, the HAProxy timeout server of rclone-gateway could be raised above Restic's own 5-minute --stuck-request-timeout, so that a slow remote does not cut requests at 30 seconds.

Components

NethServer 8 core 3.23.0-dev

See also

• Restic lock implementation: https://github.com/restic/restic/blob/master/internal/repository/lock_file.go