Disk Full Incident Runbook for Small Teams
A full disk doesn't announce itself — it takes friends down with it: the database stops writing, logs vanish, backups silently fail, and the monitoring agent that would have told you can't write its own data. The runbook has four parts: breathing room first (safely), the usual suspects, the cleanup order that doesn't destroy evidence, and the guardrails that stop the sequel.
1. Buy breathing room — without deleting the evidence
You need space now, but the first deletions decide whether the post-incident review has data. Safe first moves:
- Truncate, don't delete, active logs:
: > big.log (or journalctl --vacuum-size=200M) frees space while keeping the file handle valid — deleting a log a service holds open frees zero space and loses history.
- Clear known-safe caches: package-manager caches, old build artifacts, /tmp older than a week. Not the database directory. Not the backup directory.
- Check the WAL/transaction log: if the disk filled because the database stalled, the fastest fix is usually disk space — the database recovering is the incident ending.
One rule above all: never rm -rf a directory you can't explain. Disk pressure plus improvisation is how backups die with the server.
2. The usual suspects
- Log growth without rotation — a debug level left on, a log shipper retrying into a local file, or rotation configured but never installed.
- Journal and audit growth — systemd journal, auth logs after a brute-force wave, cron mail nobody reads.
- Docker/container layers — dangling images, orphaned volumes, and build caches from every deploy since forever.
- Backups colliding with backups — local snapshots never pruned, a dump job writing into the same disk it's dumping.
- The core-dump surprise — a crash-looping service writing cores on every restart, multiply quickly.
3. The safe cleanup order
- Find the top offenders (
du -x --max-depth=2 or ncdu) — two minutes, and the tree usually explains itself.
- Truncate active logs and vacuum journals. This alone typically buys the day.
- Prune containers/layers with the tool's own commands (
docker system prune and friends) — never by hand in /var/lib.
- Move, don't delete, anything ambiguous to a quarantine directory (same filesystem is fine) — decide its fate in the review, with time pressure off.
- Verify the service recovered before closing: the database writes, the cron runs, the backup completes tonight — not "next week".
4. The guardrails that stop the sequel
- The 80% alert is non-negotiable. One alert, one threshold, outside view. The disk that fills from 85% to 100% in an afternoon was always going to page somebody — the only question is whether it pages you at 80% or the database at 100%.
- Rotation with size caps on every log: rotate + compress + delete after N days. Unbounded logging is a delayed incident.
- Backups land off-box — a backup that fills the production disk isn't a backup, it's a second incident waiting for the first.
- A weekly five-minute disk review (scripted, top-10 largest directories) catches the slow leaks that never trip the 80% line until they do.
5. The mistakes that turn disk-full into data-loss
- Deleting the backup directory to free space. Under pressure, "biggest folder first" logic does this. The backup is the one directory you never touch under pressure.
- Deleting files a running service holds open — space doesn't free until the process restarts, and the log history is gone anyway. Truncate instead.
- Fixing the disk but not the rotation. The next incident is already scheduled; it just needs a busy week to arrive.
- No 80% alert anywhere — meaning the first sign was a database refusing writes. That's the expensive version.
Takeaways
- Truncate active logs; delete only what you can explain. Under pressure, move ambiguous things aside instead of deleting them.
- The usual suspects are boring: unrotated logs, journal growth, Docker layers, colliding backups, core dumps.
- The 80% alert plus off-box backups turn this from a recurring emergency into a five-minute weekly review.
---
The Ops Starter Kit ($14) turns the first-hour chaos of any incident — disk-full included — into a fillable plan, and Automation Starter Pack ($19) turns the disk review, log rotation, and backup sweeps into workflows that run by themselves. Launch week: 20% off any paid kit with code HIVE20 at checkout.
Silent failures love backup jobs: the backup restore test template is the monthly drill that catches the full disk, the expired token and the dead agent while they are still free to fix.
Disk full is the classic missed capacity row: the resource was always going to saturate, it just was not on the page. The capacity planning template gives every service one row — limit, four-week peak, headroom, owner — so this runbook stays a drill instead of a Monday night.
The database migration checklist prevents one classic disk-full cause: an unbatched backfill that grows the database faster than the free space and turns a routine migration into a paging incident.