Backup Restore Test Checklist: Untested Backups Are Lost Backups
Every team has backups. Almost no team has restores. The backup job has been green in the dashboard for three years, so everyone believes recovery is a solved problem — until the restore drill runs and the archive turns out to be unreadable, the credentials were rotated, or the only person who knew the restore order left in March. A backup you have never restored is a rumor. Here's the 20-minute quarterly drill that converts the rumor into a capability.
The restore drill — copy this
- Pick the target before the clock starts. One service, one dataset — the one whose loss would hurt most. Announce the drill window, then restore into a sandbox, never into production.
- Pull the backup from the source of truth, not a cached copy. Download the archive from where the job actually writes it. Restoring from a copy on someone's laptop proves nothing about the pipeline.
- Record the wall-clock time before touching anything. The drill's output is a number: minutes from "disaster declared" to "data usable". Everything else in this list feeds that number.
- Restore with the documented procedure only. If the runbook's steps fail, that is a finding — do not improvise past the doc, because the person doing a real 2am restore will only have the doc.
- Verify the data, not just the process. Row counts, checksums, a spot-check of the newest records, and one end-to-end action (log in, place an order, send the query). Green job + corrupt data is the most common failure mode.
- Time the whole thing and write it down. The pass/fail number is the handover-grade fact: "restore drill Sep 11: 23 minutes, 2 defects". No number, no drill.
The one-page drill log
| Field | Record |
|---|---|
| Date / operator | Sep 11 — on-call engineer (quarterly drill) |
| Backup tested | prod-db nightly, Sep 10 archive |
| Declare → usable | 23 min (target < 60 min) |
| Defects found | 1 — creds in runbook were rotated in July; doc now points to vault path |
| Fix owner / due | Ops lead — before next drill (Dec 11) |
That's the whole document. If the log doesn't exist after the drill, the drill didn't happen — and the next audit will agree.
Pass/fail criteria that keep the drill honest
- Time budget is a product decision. Restore-under-60-minutes is a promise to customers; measure against it, not against "it eventually worked".
- The checksum rule: an archive whose checksum can't be verified is a failed backup regardless of size. Checksums are checked during the drill, not in the backup job's own logs.
- Credential decay is finding #1. Most first drills fail on credentials or paths, not on data corruption — rotate-runbook-then-restore is the fix, and it belongs in the offboarding checklist.
- Untested backup retention is a trap. Fourteen copies of an unreadable archive cost more than three verified ones. Prune after each drill, not before.
- Paid kits pay for themselves in one prevented unrecoverable restore. The drill is the cheapest insurance audit you will ever run.
---
The Ops Starter Kit ($14) includes the full disaster-recovery pack — restore drill log, backup policy one-pager, and the incident plan this drill plugs into — launch week: 30% off any paid kit with code HIVE-LAUNCH30 at checkout.