The database outage runbook: protect the data first, fix the second

A database outage is different from every other outage in one way: the thing you are tempted to do to fix it — restart, force-recover, restore — is the thing most likely to destroy the data. An nginx restart that goes wrong costs you five minutes. A database recovery that goes wrong costs you the day's orders. This runbook has one ordering rule above everything else: confirm what you are looking at, stop the writes, degrade honestly, then restore with a plan. Where DNS outages are quiet (see the DNS outage runbook), database outages are loud — your app will tell you, in its own error language, exactly where it hurts. The runbook is about listening before acting.

1. First ten minutes: confirm it is actually the database

Half of "database down" incidents are something else wearing its clothes. One triage table, four checks:

Symptom	Check (2 minutes each)	Verdict
App errors mention `connection refused`, `too many connections`, or timeouts	Read the app logs' error class; connect from a second host with a raw client (`psql`/`mysql`)	Connects from second host = network or pool config. Refused everywhere = the database host or process
Every page 500s, including the marketing pages	Load a static asset directly	Static works = data layer. Static also broken = network, DNS, or disk — wrong runbook, switch to the internet outage runbook
Site works but is seconds-slow	Check for lock waits and long-running queries	Hung or contended, not down — restarting is the wrong move (section 2)
Database host disk near 100%	`df -h` on the database host	It was never a mystery: follow the disk-full runbook — full disks are the most common small-team database outage

The reason for the two-minute checks: the fix for a dead database is a restore, the fix for an exhausted connection pool is a config change, and the fix for a full disk is deleting files. Run the wrong one and you make all three problems.

2. Three failure modes, three different responses

Mode	Looks like	Response
Down	Connections refused; process absent or crash-looping	Read-only degraded mode now (section 4), then the restore path (section 5). Do not restart in a loop — capture state first (section 3)
Hung	App threads stuck waiting; `too many connections`; queries that never finish	Find the blocker query (`pg_stat_activity` / `SHOW PROCESSLIST`), kill selectively, fix the pool config. A restart "fixes" it by evicting every customer mid-write
Slow	Everything works, p95 latency in seconds	Usually a deploy shipped a bad query plan or a missing index; find the new slow queries, roll back the deploy if the cost is high. Slow is a limp, not a fire

The mode matters because the instinct is the same for all three — "restart the database" — and the instinct is only right for one of them. Write the mode verdict into the incident channel before acting; it forces the two minutes of thought.

3. Stop the writes before you touch anything

A database you cannot read from is an outage — painful, recoverable. Data you corrupt while recovering is a catastrophe. The ordering rule for every response:

Flip the stop-write switch before diagnosing further. A maintenance flag, a feature switch, or "pause checkout" — whatever your stack has. Writes flowing into a sick database are how a bad afternoon becomes a bad week.
Capture state before any restart. One dump of whatever is still readable (pg_dump/mysqldump), plus the database logs, copied off the host. This is the evidence trail for the post-mortem — and the only thing that survives if your next step makes it worse.
Queue, don't lose. Orders and form submissions that arrive during the outage go into a durable queue with an honest acknowledgment: "your order is saved and in line — we will email you when it is processed." A queue is not a fallback; it is the product working degraded on purpose.
The payment exception: never let a queued payment auto-execute on recovery without a human eye. Hold the charge, email the customer, process after verification. A double-charge during an outage converts an incident into a refund stampede.

This is where the incident commander earns the title: one person owns the stop-write decision so it is made once, loudly, instead of by whoever is most panicked.

4. Degraded mode: the subset that still works

Customers forgive "part of the site is down" far more than "the site lies." Degraded mode is a decision, not an accident — pre-decide the subset:

Catalog and content stay up — served from cache or static snapshots, no database queries on the path.
Accounts, checkout, and writes pause with a real message — not a 500 page: "Checkout is paused for maintenance; your cart is saved." Link the status page and keep every update carrying a next-update time.
Queue-and-ack for anything a customer must submit — the order queue from section 3, acknowledged in the customer's own words.

The one place degraded mode is wrong: if the fallback would write to a second system of record, you are building a data fork you will pay for at reconciliation time. Queue the writes; serve the reads; never split the truth. Degraded mode pairs with the payment outage playbook when the degraded slice is revenue itself.

5. The restore you rehearsed

The restore path is only fast if it was walked in peacetime. The small-team version, start to finish:

Point-in-time recovery (PITR) is the default answer. Nightly dumps alone mean choosing between "restore to last night, lose today" and "lose nothing, stay down." PITR — a base backup plus the write-ahead log/binlog replayed to just before the disaster — gets you to minutes before the outage. If your managed provider offers it, it is a checkbox, not a project.
The 30-minute path: spin up a fresh instance → restore the base backup → replay the log to the cut-off point → verify with a real query (SELECT max(created_at) FROM orders, not SELECT 1) → flip the app's connection string → stop-write switch off.
"How much can we lose?" is a daylight decision. Write the recovery point objective down — 15 minutes, 1 hour, 1 day — and buy the backup tier that matches. Discovering your RPO is "the whole day" during the outage is how teams lose the day.
Rehearse quarterly. The backup restore drill is the rehearsal: an untested backup is a rumor, and a restore you have never run takes three hours and three mistakes.

6. After it is back: the ten-minute verification pass

Restored does not mean healthy. Five checks before declaring victory:

Check	Healthy looks like
Connection count and pool config	Back to normal levels; pool size matches what the app actually needs (the most common recurrence)
Replication lag (if you have replicas)	Zero before shifting read traffic back — otherwise you serve stale data and call it recovered
Slow query log	Left on for 24 hours; the outage's root cause usually leaves fingerprints here
Disk headroom	Headroom confirmed and an alert set — the full disk comes back otherwise
The queue, reconciled	Every queued order processed and every customer acknowledged — the degraded-mode promise kept to the last one

Then close it the way every incident should close: the post-mortem template for the review, and the action item tracker for the fixes — the pool config change, the disk alert, the PITR checkbox become tracked rows, not meeting memories.

7. Common mistakes

The restart loop that deletes the evidence. Crash, restart, crash, restart — and nobody captured the logs that say why. Capture state (section 3) before the first restart, or the post-mortem will be fiction.
Scaling CPU when the problem is locks. A bigger instance with the same lock contention is the same outage with a bigger bill. Diagnose the mode before buying anything.
Restoring from last night and shrugging at the lost day. If PITR is available and you did not enable it, that decision deserves a post-mortem row of its own.
Degraded mode that forks the data. Two systems of record accepting writes during an outage means a reconciliation project, not a recovery.
App monitoring that never checks the database. A SELECT 1 from the app's own network position, every minute, is the cheapest early warning in ops. The five-signal monitoring checklist covers the rest.

Takeaways

Confirm before you fix. Down, hung, slow, and disk-full get four different responses; the two-minute triage table picks the right one.
Stop the writes before you touch anything. A database you can't read is recoverable; data you corrupt while recovering is not.
The restore you rehearsed is the only fast one. PITR enabled, RPO written down, drill run quarterly — in peacetime, or not at all.

From the HIVE80lab kit

The First 30 Minutes — free incident quick-start
Ops Starter Kit — incident response for small teams — $14
Ops Starter Kit Vol. 2 — advanced incident response, DR plan & evidence log — $27
Custom Incident Runbook — done-for-you, built from your established stack — from $99

Related: the severity matrix decides who gets woken up, the vendor outage runbook covers the managed-database provider's bad day, and the uptime & downtime budget decides how much resilience this runbook deserves.