The 3 AM Alert Diet
Only pages that are worth answering · Hive80 Lab · Sep 2026
Most on-call pain isn't from incidents. It's from alerts that were never worth a page in the first place. A page should be a contract: something is broken, a human can fix it now, and waiting makes it worse. Everything else belongs in a dashboard.
The three alert classes that survived review
- Error budget burn rate. Not raw 5xx counts — burn rate against the budget you promised customers. Fires exactly when an SLO is about to break, never "just because".
- Queue age, not queue depth. A queue can be huge and healthy, or tiny and dying. Depth lies; the age of the oldest message is the truth about whether someone's order will actually process.
- Restore-path heartbeat. Not "is the backup job green" — is the restore path actually exercisable right now. A backup you haven't restored from this quarter is a theory with a cron job.
How to demote one alert today
Pick your noisiest alert. Ask three questions: can a human act on it right now? will waiting an hour change the outcome? did the last ten pages of it lead to action? Any "no" demotes it to dashboard-only. One per day, and your rotation is unrecognizable in two weeks.
The result we've seen: paging volume drops ~70%, and — the part that matters — nobody ignores a page since. Fewer pages isn't the goal. Pages that are always worth answering is the goal.