The error budget policy: what happens when the budget burns

An SLO without a policy is a wish with a decimal point. The team writes 99.9% availability in a wiki page, everyone agrees, and then a bad deploy burns a third of the budget on a Tuesday afternoon and — nothing happens. No state changes, nobody pauses anything, and the next risky deploy ships on schedule, because the number watches you miss it and says nothing. The SLO definition sets the promise; this page is the enforcement: the budget counted in minutes, four states that are computed rather than felt, a named decider for every state, and a fraud list that keeps the number honest. A 99.9% SLO over 28 days is not a percentage — it is 40 minutes of customer-hurting minutes per month that you have chosen to spend on purpose. Spend them deliberately or they get spent for you.

The policy exists to answer one question at any moment: is this the week we ship the risky release, or the week reliability work goes first? Not “how reliable are we” — that is the metrics report's job. The budget converts reliability from a feeling into arithmetic, and arithmetic has states.

1. Three numbers, counted in minutes

The policy needs exactly three numbers, written on one line at the top: the SLO (99.9%), the window (28 days), and the budget in bad minutes (0.1% of 28 days = 40.3 minutes). Always translate to minutes: “0.1%” hides its smallness; “40 minutes per month” is small enough to feel. One bad minute = one minute in the window where the SLI says a real user was hurt. The severity matrix classifies the incident; the budget counts its damage — the two agree on one number per incident, copied by hand into the budget line, and any disagreement gets resolved in the weekly review, not silently.

Cap SLIs at two per service — availability (5xx or failed probes) and one latency threshold (p95 over N ms counts as bad minutes from threshold-cross, not from request start). A third SLI means nobody updates the budget line, and a budget nobody updates is a decoration. The budget line lives in the same place as the action tracker — one table, one owner, updated within 24 hours of every incident and every deploy that counts.

2. Four states, computed not felt

The budget's remaining percentage puts the service in one of four states. A cron (or a human with a Tuesday reminder, for small teams) computes it — the state is read off the number, never argued into existence by whoever is most upset this week:

• GREEN (more than 50% remaining): normal shipping. Nothing changes; the state exists so its absence is loud.
• AMBER (20–50% remaining): risky releases go first. Not a freeze — a re-ordering. The risky deploy you were saving for Thursday ships now, before the budget burns further, because a risky release is cheapest when there is budget left to absorb its mistakes.
• RED (under 20% remaining): feature work touching the affected service pauses; reliability work gets first pick of engineers; no deploys that change the service's behavior without a rollback plan linked. The rollback checklist becomes mandatory pre-merge, not post-mortem reading.
• BLOWN (0% remaining): automatic freeze on the affected service. The freeze ends when the budget recovers to 50% — which happens only as bad minutes age out of the 28-day window, or by design work that demonstrably removes a failure mode. Waiting is a strategy, but a slow one; design work is the fast path.

The states are deliberately boring. The point is not sophistication — it is that nobody has to be brave to declare a freeze. The arithmetic does it, and the policy pre-authorizes the consequences, which removes the argument where arguments actually happen: “should we pause the roadmap for reliability?” becomes “we are in RED, the policy says feature deploys on checkout pause.”

3. Who decides what: the decision table

Every state has one named decider. Committees decide nothing at 4 p.m. on a Friday:

• GREEN — teams decide everything themselves. The policy is silent, on purpose.
• AMBER — the service owner re-orders releases (which risky one goes first). One person, no meeting.
• RED — the engineering or ops lead pauses feature deploys on the affected service. The pause is announced in the team channel with the budget line pasted in — the number speaks, not the person, which is what makes the pause politically survivable.
• BLOWN — the freeze is automatic (nobody decides to start it). Unfreezing is the one decision that needs a named human plus evidence: which failure modes the design work addresses, and the post-mortem actions it closes. The calendar freeze policy handles planned seasonal freezes; the budget freeze handles earned ones, and the two stack.

4. What spends the budget (and the fraud list)

Bad minutes are minutes where real users were hurt by surprise. Three clarifications that kill most budget arguments:

• Planned maintenance in a communicated window does not spend the budget. Customers were told; that is downtime by appointment, not a reliability failure. It still gets counted and reported — just in its own line, not the budget's.
• A failed deploy spends the budget from first customer harm to full recovery — including the minutes it takes to notice (that is why MTTD lives in the metrics report: detection latency is budget spend).
• Partial impact counts fractionally. If 10% of users were hurt, that minute costs 0.1 budget minutes. Budgets that only count total outages teach teams to degrade quietly.

Then the fraud list — the four moves that make a budget meaningless, banned in writing:

• Redefining the SLO mid-window because it is inconvenient. SLO changes happen only at window rollover, with a written reason, at most once per quarter.
• Moving the window so the burned minutes fall out. The window is 28 days; it does not move.
• Post-hoc exclusions (“that one doesn't count, it was DNS”). Everything real users felt counts. If DNS outages are frequent enough to want excluded, that is a design finding, not an accounting problem.
• Counting hypotheticals (“we would have been down if”). The budget counts what happened; the drills count what would have.

5. The weekly line

The budget appears in the weekly ops review as one line: remaining %, state, burn trend, and the one decision the state caused. If the line reads GREEN four weeks running with zero decisions, the SLO is too loose — tighten it until GREEN requires care (a 100% green budget is not a success story; it is a number too far from the edge to steer by, and the tighten rule applies the same honesty as the 70–85% closure band). If the line has been RED for three consecutive weeks, the problem is not the budget — it is that the reliability work keeps losing to features, and that fight belongs in planning, not in the review.

The line also feeds the quarterly metrics report: windows spent in each state, total budget spend, SLO changes made and why. One decision per quarter the budget forced — that is the metric that proves the policy is alive.

6. Worked example: 22 minutes and four days

A twelve-person SaaS, checkout API, SLO 99.9% over 28 days = 40.3 budget minutes. Week one: a connection-pool change ships on Tuesday and burns 22 minutes of harm before rollback — budget drops to 45%, state AMBER before the week is out. Under the old regime (SLO in a wiki, no policy) the next release would have shipped Thursday as planned. Under the policy: the service owner moves the risky queue-rework release to the front of the week, and the connection-pool fix ships with a rollback plan linked in the PR. Week two: a config-drift incident burns another 18 minutes — budget 0.7%, state RED. Feature work on checkout pauses; the two engineers on the roadmap get four days of reliability work instead: the queue-depth alert that cut MTTD from 31 to 9 minutes, and a canary stage for pool changes. Week three: budget recovers past 20% as the burned minutes age; feature work resumes. The counter-example is real too: a team running 99.99% on a tier that cannot fund it blows the budget every month, ignores the state, and concludes budgets are theater. The fix was honesty, not discipline — the SLO moved to 99.5% (a 3.6-hour budget they could actually keep), and the first RED since the change was treated as the signal it always should have been.

7. Metrics for the policy

• Windows with a computed state — 100%. A budget that is not computed weekly is a wiki page, not a policy.
• Decisions logged per RED/BLOWN window — at least one. RED with zero decisions means the state changed nothing, which means the policy is decoration.
• SLO changes — at most one per quarter, at rollover, with a written reason. More means the number is negotiable, and negotiable numbers steer nothing.
• Median freeze length after BLOWN — under 7 days. Longer freezes mean the recovery path is waiting, not design work.

Communicated maintenance windows are the one kind of downtime that does not spend the error budget — but only if the announcement was real: five facts, five beats, and a cap you never overrun silently.

From the HIVE80lab kit

• The First 30 Minutes — free incident quick-start
• Ops Starter Kit Vol. 2 — advanced incident response & communications, including the post-mortem pack — $27
• Ops Mega Bundle — all 5 kits in one download — $29
• Custom Incident Runbook — done-for-you, built from your established stack — from $99

Related: the SLA/SLO definition template sets the number this policy enforces, the incident metrics report reads the trend, the rollback checklist is the fastest way to stop spending, and the change-freeze window policy covers the planned freezes the budget can't.