Backups That Fail at 2 AM
7 reasons a restore drill beats a backup check · Hive80 Lab · Sep 2026
Most small teams believe they have backups. What they actually have is a backup job — a process that runs, shows green, and has never once been asked to give the data back. Those are different things, and the gap between them is where companies lose weeks.
The seven failure modes
- The job backs up the wrong volume. Somebody moved the database months ago. The job still runs. It is protecting a directory that is now empty.
- Credentials rotated and the agent kept the old ones. The job errors quietly. Alerting goes to a mailbox nobody owns anymore.
- Retention is a lie. Storage filled up in March. Since then every "successful" backup overwrote the only copy that existed.
- The restore path was never tested. The company discovers that restoring requires a console login held by a contractor who left in 2024.
- RPO is "whenever the job runs" and RTO is "however long it takes". Neither number has ever been written down, so nobody can be angry about them.
- The restore test ran once, from the same disk the backup lives on. It proved nothing except that copies of files exist.
- Offsite copy exists but is encrypted with a key stored in the backup. The password-manager backup that was lost with the laptop.
The fix is boring and cheap: a scheduled restore drill. Not a backup check — a restore. Pick one service, restore last night's copy to a scratch box, time it, write the number down, fix whatever broke. One hour a month. Every failure above is caught by that single routine.
The one-page drill
- Pick the service whose loss would hurt most (not the easiest one).
- Restore yesterday's copy to a scratch machine — never the live one.
- Time: first byte, full restore, application-start. Write all three down.
- Score: pass only if the application actually starts and data verifies.
- Fix one failure mode per month. Re-run.
Run the drill this month. The green dashboard will still be green. That was never the point.