Post-Deploy Verification for Small Teams

A green pipeline is the deploy’s opinion of itself. It proves the build compiled and the scripts ran. It proves nothing about whether customers stopped hurting, whether the new code writes rows your reports can read, or whether the queue consumer three services away can still parse its payload. Verification is the ten minutes after every deploy that closes the loop: re-run the reproduction against production, watch the metric the customers feel, and only then say the word “done.” Without it, every deploy ships on faith — and regressions get discovered by support tickets.

The ten-minute loop: eight checks

Re-run the reproduction — in production, from the ticket. The failing order flow succeeds, the wrong value is gone, the timeout stops timing out. Pull the reproduction from the ticket or the incident channel, not from memory: it must be the customer’s failure re-run, not the developer’s theory re-checked. For a feature deploy this is the acceptance path; for a fix it is the exact reproduction recorded before the fix.
Watch the symptom metric for one burn-down window. Ten minutes of the metric the customers feel: error rate, failed-payment rate, queue depth, p95 latency, sign-up conversion. Not CPU, not memory — the thing that was hurting. If nobody names the symptom metric before the deploy, the release checklist box “monitoring row for the change” was skipped.
Read the new-error stream. The last ten minutes of exceptions and error logs, filtered to signatures that did not exist this morning. A deploy routinely fixes the loud bug and quietly ships a quieter one — a new stack trace at 4 per hour is invisible unless you look for new.
Touch the adjacent path. One request through whatever else calls the changed code: the other endpoint sharing the module, the report that reads the table you migrated, the mobile version of the endpoint you reshaped. Most deploy regressions are not in the path that was fixed — they are in the path that shares a wall with it.
Check what the deploy wrote. For migrations and backfills, running is not done: SELECT a sample of the new rows, count them against expectation, spot-check one row’s shape. “The migration completed” and “the data is right” are different facts, and only one of them is visible in the pipeline.
Fire the async paths. Cron jobs, queue consumers, webhook receivers, nightly jobs — anything that runs the changed code later. A deploy can break only the code that executes at 2am, and nobody notices for six days. If the changed code has an async caller, force one run inside the verification window (or verify the last run’s output shape).
Re-validate the rollback note. The one-line revert written before the deploy — still true? Command still correct, flag name unchanged, verification step for the revert still makes sense. If verification fails at minute nine, the revert must be one command, not archaeology. If it fails this check, treat the deploy as un-verified and go run the rollback checklist now, while context is warm.
Declare it done, with a timestamp. In the channel, one line: “Deployed 14:12, verified 14:21 — reproduction passes, error rate flat, queue drained. Watching until 14:45.” The timestamp is not ceremony: it is what the post-mortem, the incident metrics, and the next on-call all read back.

Five traps

Verifying the deploy, not the symptom. Green checks on the pipeline, “shipped!” in the channel — and nobody re-ran the thing the customer was doing. The pipeline proves the build works; only the symptom proves the problem is gone.
curl /healthz theater. The health endpoint ran fine in every deploy, including the three that broke checkout. Health checks prove the process is alive. If the health check doesn’t exercise the changed code, it verified nothing about this deploy.
Deploy-and-walk. The deploy lands at 5:50pm and the window is watched by nobody. The regression is discovered by a customer at 9am — twelve hours of damage that a ten-minute watch would have caught. The verification window is part of the deploy, not an optional add-on; it ends when someone has looked, not when the deploy button turns green.
Staging-only verification. Staging proves the code works somewhere. Production is a different shape: real data volumes, real third parties, real clock skew, real payloads from customers who ignored the docs. Verify in production — carefully, read-first, on the smallest real path.
Verification debt. There is no reproduction, so there is nothing to re-run, so the deploy ships on faith. This is the compound interest version: every un-reproducible deploy makes the next one less verifiable too. The fix is upstream — reproduction happens during the fix, and the release checklist books the ten minutes as a box, not a hope.

Worked example

An eleven-person B2B invoicing SaaS fixed a payment-retry bug on a Tuesday. The old way: merge, green pipeline, “shipped” in the channel at 17:03, everyone logged off. The retry queue consumer couldn’t parse the new payload shape and had been crashing since the deploy; 1,900 retries piled up over six days until a customer asked why their card hadn’t been charged. Cost: one angry renewal call, an apology credit, and a week of “is it fixed yet” tickets.

The rerun, next deploy: reproduction from the ticket re-run against prod — the failed invoice charges and flips to paid (check 1); error stream clean for the window, zero new signatures (check 3); adjacent dunning email path touched — still sending (check 4); retry consumer forced once — drained 214 queued jobs, queue depth flatlined, one poisoned message moved to the dead-letter queue with a ticket (check 6); rollback note re-read at 17:09 and still one command (check 7); declared in channel with both timestamps (check 8). Total: eleven minutes. The same loop, two weeks later, caught a webhook signature change in four minutes — the queue was watched, the poison message was quarantined, and the fix shipped the same evening.

Metrics (for the verification loop itself)

100% of deploys have a written verification — named symptom metric, named reproduction. Count it in the release checklist; anything else is aspiration.
Median deploy-to-verified time. A growing median means the loop is decaying into “green pipeline = done.”
% of fix deploys whose recorded reproduction was re-run in production. If the number is under 100%, the reproduction habit upstream is broken.
Escapes caught in-window vs caught by support. The ratio is the whole point: every escape support finds first is a customer who did your verification for you, unhappily.

From the HIVE80lab kit

The First 30 Minutes — free incident quick-start checklist
Ops Starter Kit — incident response for small teams — $14
Ops Starter Kit Vol. 2 — advanced incident response & communications — $27
Ops Mega Bundle — all 5 kits in one download — $49

Related: the hotfix process is where this loop runs fastest — check 7 lives here; the deployment rollback checklist is what happens when verification fails at minute nine; and the release checklist is where the ten minutes get booked as a box before anyone touches deploy.a week later, run the configuration drift audit — the hand-fixes and break-glass edits a hot week produces are exactly the diffs it clears into PRs.