Unattended Agent Runbook | The 3 AM Pager Problem

You built an agent, gave it a schedule and a budget, and went to sleep. At 3 AM it hit an unexpected state, retried 400 times, burned your quota, and left a note in a log file you would read 8 hours later. That gap — between failure and awareness — is where unattended systems die. Here is the field-tested discipline that closes it.

1. Evidence, not logs

A log line is a claim. Evidence is a verifiable record: every successful step must write an artifact — a file, a status row, an HTTP code captured at write time. A step that claims success without an artifact is a failed step. This single rule eliminates the entire "phantom success" class of failure.

2. Signed probes, not process pings

systemctl status says the process is alive. It says nothing about whether the token, DNS, or quota are dead. Your health check should do one full round trip: authenticate, read one real record, write nothing. That probe catches all three midnight killers in one shot.

3. Ceilings on every retry

A retry loop without a cap is a money pump. Cap retries per step, cap total steps per run, and when a cap trips: stop and escalate with the full attempt history. Flailing is worse than failing.

4. The three-tier escalation ladder

Tier	Meaning	Action
Tier 1	Retryable, known pattern	Self-heal and continue
Tier 2	Degraded capability	Continue the mission with reduced capability; log loudly
Tier 3	Irreversible / repeated	Stop, leave a full attempt report, notify

Most solo setups have tier 1 and tier 3 only. The missing tier 2 is what turns a hiccup into an outage — a degraded-but-live run keeps the mission moving while the failed leg is repaired.

5. The five-minute morning audit

One command, one page: runs started / finished / failed · outward contacts made · money spent vs budget · the single oldest unresolved failure. If your morning review takes 20 minutes, you built a report, not an audit.

6. Someone must watch the watcher

Your monitor can die too: cron killed, laptop asleep, token rotated. Run an independent second probe — different network path, different credentials — whose only job is to confirm it can still see the first one. If it cannot, that is the alert.