The Hotfix Process for Small Teams

A hotfix is an emergency deploy that skips the release train. The danger is not the speed — speed is the point. The danger is the shortcut trail a hotfix leaves behind: an unreviewed diff, skipped tests, a config changed by hand, a branch that never made it back to main. Next week's regular release steps on that trail and produces the next hotfix. The fix for this is not a committee. It is eight checks that take under ten minutes and keep an emergency deploy fast and survivable.

The bar: what earns a hotfix

Not every bug earns an emergency deploy. A hotfix is justified when production is losing money or trust right now, the normal release train is too slow to stop it, and the fix is smaller than the damage it prevents. Three questions, answered out loud before anyone touches a keyboard:

• What is it costing per hour? A number — orders failing, wrong amounts charged, logins broken. “It's bad” is not a number. If nobody can name the cost, the fix can probably wait for the next release.
• What is the smallest change that stops the bleeding? The hotfix is a tourniquet, not the surgery: disable the feature flag, revert the offending commit, cap the wrong value. The real fix goes through the normal release process.
• Who is the single owner? One person drives; everyone else stays out of the repo. Two people pushing to a hotfix branch is how a 20-line fix becomes a 200-line incident.

The eight checks

1. Name it. One line in the incident channel: symptom, cost, owner, hotfix branch name. Ten seconds. This is the receipt that the shortcut was deliberate — and the thread everyone can find when someone asks “who changed the tax rate on a Friday?”
2. Smallest diff that stops the bleeding. No drive-by cleanups, no refactors “while we're in here.” If the diff is over ~50 lines, split it: tourniquet now, fix through the normal train.
3. Reproduce first, even crudely. One test, one curl, one log line that shows the failure. Without this you are not fixing the bug — you are fixing your theory of the bug, and half of hotfix re-deploys are fixes for guesses.
4. One review, five minutes, async is fine. A second pair of eyes on the diff — not approval theater, just “does this do what you think it does, and what else calls this path?” The review happens while CI runs, so it costs zero minutes of wall time.
5. Rollback note before deploy. One line: “revert = git revert <sha> + flag back on; verify by X.” The rollback plan for a hotfix is a sentence, but it must exist before the deploy, written by someone calm, not reconstructed at 2:40am by someone panicked.
6. Same pipeline, always. The hotfix goes through CI and the normal deploy path. The moment “it's an emergency” justifies pushing straight to prod from a laptop, you have unlearned every discipline the team owns — and the emergency deploy becomes the culture.
7. Verify against the symptom, not the deploy. Green pipeline means the build works, not that customers stopped hurting. Re-run the reproduction from check 3 against production: the failing order flow succeeds, the wrong value is gone. Then watch the error rate for one burn-down window before declaring victory.
8. Close the shortcut trail the same day. Merge or cherry-pick the fix back to main (never leave the hotfix branch diverged), revert any hand-edited config, re-enable anything disabled, write the post-mortem note while it's fresh — two paragraphs is enough this night; the full post-mortem can wait for the morning.

Five traps

• Hotfix as culture. If most deploys are hotfixes, you don't have an emergency process — you have no release process. Two hotfixes a week means something upstream is broken: test coverage, feature-flag hygiene, or the release checklist being skipped. Fix the pipeline, not just the incident.
• The bypass deploy. ssh into the box and restart with the new file “just this once.” Now prod runs code that exists in no repo, and the next deploy overwrites the hotfix in front of the customers it was saving. If it isn't in the pipeline, it didn't happen.
• Fixing a guess. The most common hotfix failure is not a bad deploy — it's the second deploy, shipping because the first one fixed the wrong thing. Reproduction first. Ten minutes of proving the theory saves an hour of re-deploying.
• The forgotten backport. Hotfix branch diverges from main, gets forgotten, and the next scheduled release quietly un-fixes the bug. The hotfix isn't done until main has it. Same day, always.
• Hero mode. The same person always pages themselves, reviews their own diff, and ships alone. It works right up until the night their fix is wrong and nobody knows the context. Rotate the driver; the reviewer is never optional, even at 2am — especially at 2am.

Worked example

An eight-person e-commerce SaaS, Friday 4:40pm: the tax calculation started charging 0% on international orders. The old way: the on-call engineer found the cause (a config change in Thursday's release), pushed a fix straight from her laptop past a red pipeline because “CI is broken anyway,” told no one, and left. Monday, the next deploy overwrote the hand-edited config — the bug came back for six hours, 140 mis-billed orders, eleven support conversations, one chargeback.

The rerun, three months later, same class of bug at the same hour: declare in channel (cost named: ~$2,300/hr in mis-billed orders), four-line diff behind a check already in the code, one failing-order reproduction recorded, five-minute review from the London engineer while CI ran, rollback note written before deploy, deployed through the normal pipeline at 4:58pm, reproduction re-run against prod, error rate watched one burn-down window. Monday's release cherry-picked the fix, config reverted, two-paragraph post-mortem posted that evening. Total: 22 minutes from declare to verified, zero re-deploys, two support emails — and the post-mortem flagged the underlying config gap, which shipped as a proper fix through the normal train the next week.

Metrics (for the hotfix process itself)

• Hotfixes per ten deploys: trending toward 1–2 or fewer. A rising rate means the normal train is dropping the ball — investigate the upstream, not the hotfixes.
• 100% of hotfixes had a written rollback note before deploy. Count it; anything else is aspiration.
• 100% backported to main within 24 hours. Diverged hotfix branches are unexploded ordnance in the repo.
• Time from declare to verified symptom gone — not to “deploy finished.” Track the median; a growing median says the eight checks are decaying.
• Zero repeat hotfixes for the same symptom inside two weeks. A repeat means check 3 (reproduce first) was skipped.

From the HIVE80lab kit

• The First 30 Minutes — free incident quick-start checklist
• Ops Starter Kit — incident response for small teams — $14
• Ops Starter Kit Vol. 2 — advanced incident response & communications — $27
• Ops Mega Bundle — all 5 kits in one download — $49

Related: the deployment rollback checklist is what check 5 points at when the tourniquet needs a real plan; the change freeze window policy decides when “wait for Monday” beats “ship the hotfix”; and the post-mortem template is where the two-paragraph note from check 8 grows up in the morning.check 7 of this list is the post-deploy verification loop run under pressure — same eight checks, ten minutes, no skipping.the configuration drift audit is where forgotten hotfix backports surface — step 8 of the audit reads the break-glass receipts.