Runaway Automation Runbook for Small Teams

Automation fails differently from people: a confused teammate stops and asks, a confused script keeps going — retries multiplying, emails cascading, an AI agent re-running a task in a loop with your API budget. The runbook has four parts: a kill switch you can reach in one command, bounds that make loops self-limiting, idempotency so re-runs can't double-charge, and the post-incident checklist.

1. The kill switch you can reach in one command

Every scheduled job, queue consumer, and autonomous agent needs a pre-agreed way to stop everything at once — not one process at a time, hunted down by name at 2am:

The test that matters: a new teammate can stop the automation in under a minute using only the runbook. If stopping requires memory, it doesn't exist.

2. Bounds that make loops self-limiting

• Max runs per window — enforced by the scheduler wrapper, not the script. A job that ran 14,000 times in an hour had no ceiling above it.
• Max spend per day on anything that bills per call (APIs, LLMs, egress). The budget alert from the cost runbook is this rule's alarm.
• Backoff with a cap, then dead-letter — after N failures a job must stop trying and park the work somewhere a human sees. Infinite retry is how one outage becomes a retry storm.
• Circuit breakers on dependencies — when the downstream is slow, stop calling it. politeness is also self-preservation: the slow dependency gets slower under load.

3. Idempotency: the reason re-runs don't double-charge

The scariest runaway incidents do real work twice: two invoices per customer, two shipments, two identical customer emails. The fix is boring and architectural:

• Job ID + idempotency key on every external side effect; the downstream deduplicates.
• Check-before-act for stateful steps ("has this already been sent?") — a one-line guard that prevents every duplicate story in this genre.
• Dry-run mode by default for anything destructive — the flag that turns "oops" into "log line".

4. The post-incident checklist (one page, five lines)

1. What guardrail was missing? Every runaway has one absent ceiling: no max-runs, no max-spend, no dead-letter. Install exactly that one this week.
2. What did it cost? Dollars, customer trust, and the data it duplicated — written down while fresh.
3. Who got spammed / double-charged? Outbound apologies beat inbound complaints. Have the list before the customers do.
4. Did the kill switch actually work? If stopping it required tribal knowledge, that's the fix list.
5. Re-test the kill switch quarterly. An unrehearsed stop button is a wish with a label.

5. The mistakes that make runaway automation worse

• Reasoning with the agent. Telling a looping LLM agent "please stop" via prompt is not a control. Kill it from the outside, then fix the loop.
• Retries without dead-letters. Infinite retry is a second automation, and nobody agreed to run it.
• Manual crontab surgery under pressure. Editing schedules one-by-one during an incident is how the second outage is born (see the cron monitoring checklist).
• No cost ceiling anywhere. The runaway that costs money should hit a wall, not a month-end invoice surprise (see the cost spike runbook).

Takeaways

• One-command stop for every layer — flag files, queue pauses, external caps, key revocation.
• Ceilings live outside the automation: max runs, max spend, dead-letters, breakers.
• Idempotency makes re-runs boring — and boring is the goal.

---

The Automation Starter Pack ($19) is built around pick-first workflows with these bounds installed — flag-file wrappers, max-run ceilings, and the weekly automation review. Running autonomous agents? The Agent Ops 24/7 ($19) playbook covers keepalives, watchdogs, and graceful shutdowns for 24/7 operation. Launch week: 30% off any paid kit with code HIVE-LAUNCH30 at checkout.