API Key Leaked: the First-Hour Runbook for Small Teams
A leaked API key is an incident with a timer — bots scrape public repos and paste sites within minutes, not hours. The order of operations decides whether you get a scare or an outage: scope the blast radius, rotate in the safe order (new key first, revoke second), check for real abuse, then purge the copies. Deleting the file is not the fix; revocation is.
1. Scope the blast radius (minutes 0–10)
Before touching anything, answer four questions — each one changes the play:
| Signal | What it means | First move |
|---|
| Key in a public repo or paste site | Assume scraped already | Rotate immediately (section 2); history cleanup is hygiene, not the fix |
| Key in a shipped client bundle (JS app, mobile app) | It is public by construction | Never ship secret keys client-side; move the call behind your backend, then rotate |
| Key in CI logs, error tracker, or ticket | Every viewer and log archive holds a copy | Purge where you can, rotate regardless — log retention outlives the message |
| Key works and has broad scopes (payments, email, cloud) | Real money and data at risk | Rotate first among all keys; check spend and audit logs for abuse |
Write down: which account, which scopes/permissions, when it was created, where it leaked, who has seen it. That list becomes your rotation checklist and, if abuse occurred, your evidence log (see the first-24-hours breach checklist for the preserve-before-you-act rule).
2. Rotate in the safe order (minutes 10–30)
The classic self-inflicted outage is revoking first and discovering production was using the key. The safe dance:
- Create the replacement key first — most providers allow two live keys during a rotation window.
- Deploy the new key everywhere the old one is used: secrets manager, server config, CI variables, cron jobs, local dotfiles for the two engineers who run things by hand.
- Verify the new key works with a real call in staging or a dry run, then revoke the old key. Revoke — don't just delete the file.
- Can't stage it? For keys with no rotation window (single live key), accept the brief outage or pause the job: revoke, fix, restart. A 10-minute outage beats a drained cloud account.
- Shrink while you're there: if the old key was all-scopes, issue the replacement with least privilege and an expiry or IP allowlist where the provider supports one.
3. Check for real abuse (in parallel, minutes 10–60)
- Usage dashboard: look for calls you didn't make — unfamiliar endpoints, regions, user agents, or a spike since the leak date.
- Billing first, dispute second: if spend occurred, revoke the key, then contact the provider's abuse/fraud desk with your timeline. Providers generally treat scraped-key spend fairly if you reported fast — which is one more reason not to sit on it.
- Cloud keys: check for new IAM keys, new resources in odd regions, and forwarded email rules. Assume persistence until proven otherwise.
- OAuth client secrets: revoking the client secret kills refresh tokens — expect users to re-authenticate; that's correct behavior, not a bug.
- Preserve the logs before you delete anything — export the audit trail first (see the log retention policy for how long to keep it).
4. Purge the copies that matter (minutes 30–60)
- Assume it was scraped. Secret-scanning bots hit public repos in minutes. History cleanup protects you from the next reader, not from whoever already has the string — revocation remains the real fix.
- Rewrite or replace: if the repo is public and you can, purge the blob from history; if you can't, delete-and-rotate is acceptable when the key is confirmed revoked. Weigh a force-push rewrite against your team's reality before doing it.
- Sweep the quiet copies: CI logs, error-tracker events, Slack messages, support tickets, screenshots, laptop dotfiles, and any "quick test" scripts. These outlive the repo.
- Close the hole, not just the instance: the leak happened through a process — a missing
.gitignore, a hardcoded default, a verbose logger. Fix that in the same hour, or next week it happens again.
5. The prevention pass (this week, 30 minutes)
- Secret-scanning in CI (provider-native scanning or an open-source pre-commit hook) — it catches the leak at push time, when it's cheap.
.env files stay out of the repo, with a committed .env.example so onboarding still works.- Least-privilege scopes and expiries on every new key; an expired key can't leak next year.
- CI variable masking so tokens print as
*** in logs — the second most common leak source.
- A quarterly two-minute audit: grep the codebase and home directories for
sk_, AKIA, ghp_ and friends; list every live key with an owner and an expiry.
The mistakes that turn a leak into a crisis
- Revoking before deploying the replacement. You turn a security scare into a production outage and a rollback under pressure.
- Deleting the file and calling it fixed. Git history, CI logs, and pasted messages all still have the key.
- Trusting "private repo" as safety. Forks, CI logs, and misconfigured visibility turn private into public regularly.
- Not checking spend. Crypto-mining on a leaked cloud key is a business model; the bill is the symptom you find last.
- Rotating "everything" in a panic. Mass rotation without a written map breaks production twice in one night — rotate the leaked key's blast radius first, the rest on schedule (see the secrets rotation checklist).
Takeaways
- Scope, rotate in the safe order, check for abuse, purge — in that order, in one hour.
- Revocation is the fix; history cleanup is hygiene.
- Fix the process that leaked it the same week, or the next leak is already scheduled.
---
The Ops Starter Kit ($14) includes the incident templates and evidence-log structure that make key-leak response survivable, and Vol. 2 ($27) adds the full DR plan and evidence log. Launch week: 30% off any paid kit with code HIVE-LAUNCH30 at checkout.