HIVE80lab — Ops notes

Disaster Recovery Plan Template for Small Teams

The one-page DR plan: five fill-in blanks, three numbers, one drill per quarter. Written for teams of 2–30 who cannot afford an enterprise BCP engagement and don't need one.

Most small teams' "disaster recovery plan" is a folder of PDFs from an insurance-driven compliance push, last opened the week it was written. When the region actually goes down, nobody opens the folder — they open a chat window and start improvising. Improvising during a disaster is expensive: the same question ("where are the backups?") gets answered three times, by three people, three different ways, and the two answers that are wrong cost you hours.

The fix isn't a 40-page plan. It's one page that answers, before the outage, the questions everyone will ask during it. Here is the whole template — fill in the blanks, print it, stick it in two places (your password manager and your phone), and rehearse it once a quarter.

The five fill-in blanks

  1. The trigger. What counts as a disaster? Write the test, not the vibes: "primary region unreachable for 15+ minutes from two networks" or "database server dead and restore ETA exceeds 2 hours." If the trigger requires a human to decide whether it's bad enough, you wrote a debate, not a trigger.
  2. The three numbers. RTO — how long until you're back (target: a number you've actually achieved once). RPO — how much data you can afford to lose (this is a statement about backup frequency, nothing else). Who's called — in order, with phone numbers, not Slack handles. Slack is down in the disasters Slack handles were meant for.
  3. The failover path. The exact sequence: restore from where, to where, verified how. This is the same rule as rollbacks: as exact commands, not as descriptions. "Restore from backups" is not a path. "aws s3 cp s3://dr-bucket/latest.sql.gz . → psql < it → run smoke test script" is a path.
  4. The degraded mode. What does the business do while you're restoring? Write-only mode? Static status page + "orders paused"? A DR plan without a degraded mode forces full recovery to be the only acceptable outcome — and full recovery is the slowest one.
  5. The drill. When it's tested (quarterly), who runs it, and where the drill result is written. An unrehearsed DR plan is a rumor.

The one-page plan, filled in

FieldExample
Triggerapi region unreachable 15+ min from 2 networks, OR db restore ETA > 2h
RTO4 hours (achieved in March drill: 3h 41m)
RPO15 minutes (nightly full + 15-min WAL shipping)
Call order1. Sam (mobile) 2. Priya (mobile) 3. Dan (contractor, +44…)
FailoverDR runbook step 1–7 at /ops/dr-runbook.md — ends with smoke-tests.sh green
Degraded modeStatus page live, orders paused, support replies with template #3
DrillQuarterly, first Friday; result logged in ops journal; failures become tickets

The drill is the plan

Untested numbers are wishes. The 20-minute quarterly drill: pick a Tuesday, restore the database to a scratch instance from last night's backup, run the smoke tests, time it, log one line. If the restore takes 3 hours, your RTO of 4 hours just became 3 hours of staring + 1 hour of work — and you learned it on a quiet Tuesday instead of during the real thing. This is the same discipline as the backup restore test, with the added step of timing it against your written RTO.

The rules that keep the plan alive

The numbers that tell the truth

Track three lines per drill and per incident: time-to-detect, time-to-restore, data lost (in minutes). If those three aren't trending down (or already flat-good), the plan is decorative. Everything else — the documents, the tooling, the vendor SLAs — exists only to move those three numbers.

---

The Ops Starter Kit Vol. 2 ($27) includes the full continuity pack — the one-page DR plan, the restore drill card, and the on-call call list template — plus the advanced incident response and comms templates. Launch week: 30% off any paid kit with code HIVE-LAUNCH30 at checkout.