Alert Fatigue Checklist: Cut the Noise Before Someone Mutes the Pager
Alert fatigue isn't an annoyance — it's an outage waiting for an audience. The first time a tired on-call human mutes a noisy channel, your real next incident pages nobody. Teams that fix alert fatigue don't buy better tools; they audit the noise and delete it. Here's the monthly checklist that keeps the pager loud when it matters and quiet when it doesn't.
The monthly noise audit — copy this
- Count pages, not alerts. Pull last month's pager log and count: total pages, pages outside 9–5, pages that led to an action. If fewer than 1 in 3 pages caused an action, the system is spamming you — that ratio is your fatigue number, and it gets worse on its own.
- Apply the 3-question test to every repeat offender. For each alert that fired more than 3 times last month: Is this actionable right now? Would a human act differently at 3AM than at 3PM? If this alert disappeared, would anything get worse? Two "no"s = the alert is demoted or deleted. Deleting an alert is allowed. Nobody misses the ones that never mattered.
- Tame the flaky three. Every team has 2–3 alerts everyone ignores. For each, write the tell: "the disk alert double-fires after patching, it's real if it persists 15 min." Add that condition to the alert itself (a for/5m clause beats a mental note). The tell goes in the handoff doc until the fix lands.
- Fix thresholds with data, not vibes. An alert at 80% CPU that fires every afternoon is a threshold that describes Tuesday, not danger. Look at the daily curve, set the line above the routine peak, and let the routine stay silent. A threshold you've never tuned is a guess wearing a pager.
- Route by severity, not by curiosity. SEV1 pages a human at any hour. SEV2 pages business hours or queues in chat. SEV3 never pages — it waits in the morning digest. If your SEV3s are waking people, your severity matrix is decorative (see the 3-level matrix below and give the levels teeth).
- Give every alert an owner and a runbook link. An alert that pages "nobody in particular" gets muted in exactly one incident. Each alert rule gets: an owner, a linked runbook, and a last-touched date. No owner + no runbook = the alert is a candidate for deletion, not tuning.
- Report the trend, not the shame. One line in the monthly ops review: "pages: 31 → 14, action rate 28% → 61%." Declining noise is a maintenance win the team can see. Rising noise means the audit didn't run — run it again, not harder.
The flaky-alert taming table
| Alert | Tell | Fix | Done? |
|---|---|---|---|
| Disk 80% nightly spike | Fires 02:00, clears by 02:10 (logrotate) | for 15m condition + move threshold to 85% | yes |
| API 5xx blip after deploys | 30s during rollout only | suppress during deploy window tag | yes |
| Cron heartbeat "late" on weekends | Batch job schedule differs Sat/Sun | per-day schedule in the check | no |
Rules that keep the pager trustworthy
- The pager is sacred; chat is not. Anything that can wait 12 hours belongs in chat or the digest. The moment routine work pages a human, you've spent trust you can't mint back at 3AM.
- Every page gets a disposition. Real, duplicate, tuning-needed, or junk — one tag at close time. The monthly audit is just sorting these tags; without them, tuning is folklore.
- Delete in pairs. Every noisy alert you delete earns the right to add exactly one new, sharper alert. Noise grows back unless the budget stays fixed — 15 pages that all matter beat 40 that don't.
---
The Automation Starter Pack ($19) includes the pick-first alert-routing workflows — severity routing, digest batching, and the flaky-alert tamer this checklist plugs into — launch week: 30% off any paid kit with code HIVE-LAUNCH30 at checkout.