Disk Full Incident Runbook for Small Teams

A full disk doesn't announce itself — it takes friends down with it: the database stops writing, logs vanish, backups silently fail, and the monitoring agent that would have told you can't write its own data. The runbook has four parts: breathing room first (safely), the usual suspects, the cleanup order that doesn't destroy evidence, and the guardrails that stop the sequel.

1. Buy breathing room — without deleting the evidence

You need space now, but the first deletions decide whether the post-incident review has data. Safe first moves:

Truncate, don't delete, active logs: : > big.log (or journalctl --vacuum-size=200M) frees space while keeping the file handle valid — deleting a log a service holds open frees zero space and loses history.
Clear known-safe caches: package-manager caches, old build artifacts, /tmp older than a week. Not the database directory. Not the backup directory.
Check the WAL/transaction log: if the disk filled because the database stalled, the fastest fix is usually disk space — the database recovering is the incident ending.

One rule above all: never rm -rf a directory you can't explain. Disk pressure plus improvisation is how backups die with the server.

2. The usual suspects

Log growth without rotation — a debug level left on, a log shipper retrying into a local file, or rotation configured but never installed.
Journal and audit growth — systemd journal, auth logs after a brute-force wave, cron mail nobody reads.
Docker/container layers — dangling images, orphaned volumes, and build caches from every deploy since forever.
Backups colliding with backups — local snapshots never pruned, a dump job writing into the same disk it's dumping.
The core-dump surprise — a crash-looping service writing cores on every restart, multiply quickly.

3. The safe cleanup order

Find the top offenders (du -x --max-depth=2 or ncdu) — two minutes, and the tree usually explains itself.
Truncate active logs and vacuum journals. This alone typically buys the day.
Prune containers/layers with the tool's own commands (docker system prune and friends) — never by hand in /var/lib.
Move, don't delete, anything ambiguous to a quarantine directory (same filesystem is fine) — decide its fate in the review, with time pressure off.
Verify the service recovered before closing: the database writes, the cron runs, the backup completes tonight — not "next week".

4. The guardrails that stop the sequel

The 80% alert is non-negotiable. One alert, one threshold, outside view. The disk that fills from 85% to 100% in an afternoon was always going to page somebody — the only question is whether it pages you at 80% or the database at 100%.
Rotation with size caps on every log: rotate + compress + delete after N days. Unbounded logging is a delayed incident.
Backups land off-box — a backup that fills the production disk isn't a backup, it's a second incident waiting for the first.
A weekly five-minute disk review (scripted, top-10 largest directories) catches the slow leaks that never trip the 80% line until they do.

5. The mistakes that turn disk-full into data-loss

Deleting the backup directory to free space. Under pressure, "biggest folder first" logic does this. The backup is the one directory you never touch under pressure.
Deleting files a running service holds open — space doesn't free until the process restarts, and the log history is gone anyway. Truncate instead.
Fixing the disk but not the rotation. The next incident is already scheduled; it just needs a busy week to arrive.
No 80% alert anywhere — meaning the first sign was a database refusing writes. That's the expensive version.

Takeaways

Truncate active logs; delete only what you can explain. Under pressure, move ambiguous things aside instead of deleting them.
The usual suspects are boring: unrotated logs, journal growth, Docker layers, colliding backups, core dumps.
The 80% alert plus off-box backups turn this from a recurring emergency into a five-minute weekly review.

---

The Ops Starter Kit ($14) turns the first-hour chaos of any incident — disk-full included — into a fillable plan, and Automation Starter Pack ($19) turns the disk review, log rotation, and backup sweeps into workflows that run by themselves. Launch week: 30% off any paid kit with code HIVE-LAUNCH30 at checkout.