Degraded mode playbook for small teams: serve something while the dependency is down

When a dependency breaks, a small team really has two options: the reduced service it chose in advance, or a full outage it chose by default. At 2am nobody designs a fallback — they either improvise a risky one or freeze the whole product. Degraded mode is not heroic engineering; it is a one-page decision, made on a calm afternoon, per dependency: if this goes down, we serve X, customers hear it from us, and one named switch flips it. Teams that write the page down survive vendor outages with a banner. Teams that don't survive them with an apology email.

This note gives the four-mode ladder, the kill-switch rule that makes degradation reachable in minutes, the customer conversation, the five traps, and a half-day rollout.

1. The mode ladder: pick one per dependency, in advance

MODE 0  FULL          everything works (the normal state)
MODE 1  STALE READS   serve last-known data + "may be up to N minutes old" banner
MODE 2  QUEUE+ACK     accept the work, queue it, confirm with a timestamp:
                      "received 14:22, will process when X returns"
MODE 3  READ-ONLY     browsing and viewing work; anything that writes is off
MODE 4  DOWN          honest full outage; status page carries the load

The rules that make the ladder real:

• One written mode per tier-1 dependency. "We'd figure something out" is not a mode. The row reads: address-validation vendor down → MODE 2 → queue shipments, validate on recovery → switch: FEATURE_ADDR_QUEUE=on — with the owner's name beside it.
• Mode is a floor, not a ceiling. Start at the pre-chosen mode; escalate up the ladder only with the same care. The pre-choice exists so the first five minutes are execution, not design.
• Every mode change has a customer-visible signal. A banner in the product, a line on the status page, or an email — chosen before the incident, not after the first "is this broken?" ticket.
• The flip-back is planned too. Queued work drains, stale caches refresh, the flag returns to off, and someone owns verifying each step. Degraded modes that have no exit become the new normal.
• Unbounded queues are a disguised outage. MODE 2 without a cap and a drain time is just a slower failure. Bound it, announce the drain estimate, and say what happens to work that won't fit.

2. The kill-switch rule

A degraded mode you cannot reach is a rumor. Every written mode must be reachable by a switch that:

• Flips in under five minutes — a config change, a feature flag, or a cached-response toggle. Not a deploy; not a call to someone on holiday.
• Has been pulled at least once — in a drill or a maintenance window. The switch nobody has ever thrown fails in exactly the way you'd expect: typo'd key, wrong environment, permission nobody has. The flag cleanup pass exists so the inventory stays small enough to actually test.
• Has a named owner and a back-out. Who pulls it, who verifies the mode took effect, and what the switch-back looks like — one line each, in the same table.
• Degrades one dependency, not the product. If flipping the switch takes the whole app down "temporarily," that is not a kill switch; that is a second outage waiting for the first.

3. The customer conversation

Silent degradation is discovered by customers, and discovery without explanation reads as betrayal. The playbook ships with the words, pre-written:

• One product banner line per mode. Stale reads: "Reports may be up to 15 minutes old — your data is safe and syncing." Queue mode: "We're receiving orders and will process them as soon as our payment partner recovers; you'll get a confirmation email either way."
• The status page gets the same sentence plus a next-update time, using the standard status page format — what is reduced, what still works, when the next update lands.
• Recovery gets a closing note, including what happened to queued work. "All 214 queued orders processed by 16:40, three failed and were refunded" is the sentence that turns a degraded hour into a trust deposit. The outage email templates carry the same clocks.

4. The five traps

The 2am design. The mode ladder written during the incident is a wish, not a plan — and it competes with actually putting out the fire. Degraded mode is an afternoon of calm work stored for an uncalm night; if it doesn't exist on paper yet, expect the default (full outage) to win.

The never-pulled switch. A flag added in a sprint six months ago, never flipped since, environment variable name guessed from memory. First real pull: typo, no effect, three wasted minutes that felt like thirty. Drill one switch a month.

Silent degradation. Serving stale data without a banner converts a technical state into a trust incident: customers make decisions on old numbers and find out from each other. If the mode is visible in the data, it must be visible in the product.

Queue without a receipt. Accepting writes into a queue that has no ack, no bound, and no drain announcement means customers don't know whether their order exists. The receipt with a timestamp is the product during degraded mode; the processing is a detail.

No way back. Read-only mode that no one owns exiting becomes permanent read-only, then a migration, then a post-mortem about the post-mortem. Every mode entry has a matching exit checklist: switch off, queue drained, cache refreshed, banner lifted, and someone's name on each line.

5. Worked example

A nine-person logistics SaaS routes shipments and validates addresses through a third-party API. When the vendor had a three-hour outage, their entire product went down with it — checkout included — because validation was on the critical path and nobody had ever decided it didn't have to be. Thirty-one support conversations and two lost customers later, they spent a half-day writing the ladder.

They settled on: address-validation down → MODE 2 (accept shipments, queue validation, confirm with a timestamp) reachable by a config flag they tested that same week in a maintenance window; reporting feeds → MODE 1 (stale reads, 15-minute banner). When the same vendor degraded six weeks later, the flip took four minutes, the banner went up in three, and queued shipments were validated and confirmed by the end of the hour — support conversations: three, all forwarding the same receipt. The counter-example is their own earlier practice, preserved in the post-mortem: the bypass flag existed during the three-hour outage but had never been pulled in production, and the person who knew the variable name was asleep.

6. Metrics (for the mode itself)

• 100% of tier-1 dependencies have a written mode + switch + owner — one table, one screen, reviewed quarterly next to the service catalog.
• One kill-switch drill per month; every switch pulled successfully at least once per quarter.
• Time-to-degrade under 10 minutes from alert to mode active (measured in the drill, not first tried in the incident).
• Zero silent degradations — every mode change has a banner/status/email event attached.
• 100% of queued work gets a receipt, a bound, and a recovery report with counts.

From the HIVE80lab kit

• The First 30 Minutes — free incident quick-start
• Ops Starter Kit — incident response for small teams — $14
• Ops Starter Kit Vol. 2 — advanced incident response & communications — $27
• Ops Mega Bundle — all 5 kits in one download — $49

Related: the vendor outage runbook covers the dependency side, the database outage runbook applies the same read-only and queue-and-ack pattern to your own writes, and the internet outage runbook is the degraded mode for the network itself.