Add monitoring for backup jobs

[nosmo]

In light of recent events and also just as good common sense, every backup host should also create monitoring jobs for all backups run. This check could be a Nagios check, a Sensu check or any other check really.

[tomryanx]

I think I've understood this right, but correct me if not. There's a few things that could conceivably be checked, and a couple of problems.

• did the backup complete without error? Was success/failure reported?
• is the backup the right size?
• it the backup properly signed?
• do the files in the backup match the files that were backed up?

This last point is particularly problematic for two reasons.

1. Files on disk change. A heuristic like "do cksums for X% of files under/over/around size Yb match?" could probably get a decent result. (Taking cksums at the beginning or the end of the backup doesn't completely eliminate this problem)
2. The backups are encrypted. Is there a way to verify the contents without using the decryption key? And if there isn't, do we want a monitoring mechanism that involves using those keys?

The quickest win I can think of is a local script, to be polled by a remote monitoring system (eg nagios-nrpe, but really anything), which checks for job exit status and possibly file size or file size delta (eg "files on disk increased by 15% yesterday, was backup difference also within 5% - 25% increase"). The polling server may need to keep some state.

[nosmo]

Checking for matches is off the table, there's no safe way to implement this - I think a firedrill exercise in a calendar is a better solution for testing backup restoration rather than anything else.
I think point number 2 is also potentially problematic as the backups will be encrypted and so totals can't be calculated properly, and also backups will be differential so increases may be very small for data that is valid.

Checking return codes (and potentially log context if suitable) seems like the safest MVP for this task.