The four golden signals: what to watch when you can only watch four things

Most small teams' monitoring grows the same way: one panel per alert, one alert per scare, until the dashboard is forty panels deep and nobody can say what a healthy service looks like. The four golden signals — latency, traffic, errors, saturation — are the opposite bet: four numbers, one row each, that will notice nearly every real outage before a customer does. This page is the minimum-viable monitoring contract; the SLO definition turns two of the signals into a target, the burn-rate alerts turn the error signal into a pace, and the paging policy decides who gets woken when a row goes red.

The reason four works: every user-visible symptom is one of four sentences — it's slow (latency), nobody's using it (traffic), it's broken (errors), it's about to be (saturation). Panels that don't answer one of those four sentences are decoration.

1. The four signals, one line each

• Latency — how long requests take to serve, with the two-latency rule: successful and failed requests get separate lines. A connection refused returns in two milliseconds and a naive average celebrates it; time-outs get retried and double-counted. Track the latency of successes and of failures separately, and alert on a percentile (p95/p99), never a mean.
• Traffic — demand, in a unit native to the service: requests per second for an API, queries per second for a database, messages per second for a queue, concurrent sessions for a video service. Traffic is the denominator that turns “errors doubled” into “error rate doubled” or “just a busy Tuesday.”
• Errors — the rate of requests that fail: explicit failures (5xx, refused, timeout) and implicit ones — the HTTP 200 whose body says “payment declined,” the job that logged done and did nothing, the queue item dropped on shutdown. Every silent-failure class needs its own counter, or it does not exist.
• Saturation — how full the most constrained resource is: the database connection pool, queue depth, disk, CPU throttling on a container. It is the leading indicator: services degrade at 80% utilization long before they die at 100%, and the saturation row is usually the only place a slow leak is visible.

2. The traps each signal hides

Averaged latency. A mean can sit flat while the p99 is on fire — 1% of users having a terrible time does not move an average, but it fills a support inbox. Percentile lines only.

Status-code error counting. If errors are “HTTP 5xx,” every 200-wrapped failure is invisible: the payment endpoint that returns 200 with {"status":"failed"}, the import that half-completes. Count business-level failures as errors too — one counter per known silent-failure mode.

Reading signals alone. “Errors up 100%” means nothing without traffic: if demand doubled, the error rate is flat. Every alert text should carry the ratio (“error rate 0.8% vs 0.3% baseline”), not the raw count.

Saturating the wrong resource. Teams watch CPU because it is on every dashboard by default, then die by connection-pool exhaustion, disk inode exhaustion, or queue backlog. Ask once, per service: what runs out first? That is the saturation metric — and if you cannot name it, you do not know your own ceiling.

Zero traffic is not quiet health. A service receiving no traffic is a dead service or a broken upstream. Alert on traffic collapse as well as load: traffic < 20% of same-hour baseline pages just like an error spike.

3. Page or ticket: thresholds per signal

Map each of the four rows to a response class using the four-question page test — the defaults that work for most small teams:

• Page: p99 latency > 2× baseline for 10 minutes; the error burn-rate pair from the burn-rate scheme firing; saturation > 90% and latency starting to move; traffic collapse to near zero on a service that should be busy.
• Ticket with a fast SLA: saturation 70–90% but stable; one-AZ latency drift; error rate above target but burn rate under the page pair; a silent-failure counter showing its first trickle.
• Log and trend: everything else — a row that has never fired in 90 days is either over-thresholded or a panel you do not need.

Alert text carries the arithmetic, same rule as everywhere: “DB pool 94% full, p99 1.9s vs 480ms baseline, climbing 6%/min — runbook: db-pool-exhaustion.md.”

4. The wiring checklist (one afternoon)

• One dashboard row per signal — four rows total — before any other panel exists. Every extra panel must have fired at least once to earn its place.
• Define traffic per service in its native unit; write the unit in the panel title (“msgs/s”), or the next reader will compare apples to wall-clock time.
• Add one counter per known silent-failure mode and include it in the error signal. Start with the one that would embarrass you most.
• Every threshold owner is named; every alert links a runbook; non-actionable pages get demoted per the paging policy, not muted in silence.
• Monthly four-row review: which row caught what, which panel never fired, one threshold decision recorded (raise, lower, or delete).

5. Worked example

A nine-person payments SaaS runs exactly four rows on its checkout service. Friday 17:40, the saturation row — DB connection pool — climbs past 90% while latency is still inside its alert line; the pool alert pages with the projection. On-call restarts a wedged worker at 17:52; p99 recovers by 18:05; total customer impact is one minute of slow checkouts that never reached the error line. The post-mortem is four sentences and one dashboard tweak: the pool counter also became a burn-rate input, because the pool had been draining for two hours before anyone looked.

The counter-example is the same Friday at a team with a forty-panel dashboard, all averages: the pool exhaustion is invisible (panel 31, “worker CPU,” reads a comfortable 30%), latency panels show the mean absorbing the tail, and the first error-rate alert waits for hard 5xx — which start at 18:20 when the pool is fully dead. Detection came from a customer's Slack message, recovery took 3 hours, and the Monday retro produces a forty-first panel instead of a threshold decision.

6. Metrics

• Every production service has all four rows, in native units: 100% — a service monitored only where dashboards are easy is unmonitored exactly where it breaks.
• Median time-to-detect via signal < time-to-first-customer-report — the whole point of the four rows.
• ≥95% of fired alerts actionable (page test passed); the rest demoted or retuned within the week.
• Panel count per service trends down or flat: any panel that has not fired in 90 days is deleted, not ignored.

From the HIVE80lab kit

• The First 30 Minutes — free incident quick-start
• Ops Starter Kit — incident response for small teams — $14
• Ops Starter Kit Vol. 2 — advanced incident response & communications — $27
• Ops Mega Bundle — all 5 kits in one download — $49

Related: the SLO definition template turns the latency and error rows into a target, the burn-rate alerts give the error row its pace thresholds, and the paging policy decides which of the four rows is allowed to wake a human.