The certificate expiry outage is the most predictable incident in small-team ops: the tooling existed to prevent it, the renewal "was automatic," and at 11pm on a Friday the browser warning page is standing between your customers and the checkout. The honest fix has four parts: inventory every cert, alert on what is actually served, rehearse the renewal, and write the runbook for when it's already too late.
Most teams monitor one certificate — the public site — and get surprised by the other four. Your inventory is a single table with one row per hostname that terminates TLS:
| Column | Examples |
|---|---|
| Hostname | www, api, mail, vpn, internal admin panel, staging |
| Expiry date | Pulled live, not from memory: openssl s_client -connect host:443 -servername host 2>/dev/null | openssl x509 -noout -enddate |
| Renewal method | ACME bot / cert-manager / CDN-managed / manual (the manual ones are the incidents) |
| Owner | A named person, not "ops" |
Include the unglamorous endpoints: the load balancer's own cert, SMTP/IMAP on the mail host, the VPN gateway, the billing portal someone bought separately in 2023. Internal certs nobody watches are the ones that take down the admin panel at the worst time.
Automation is the right default: ACME/Let's Encrypt for public hosts, cert-manager on Kubernetes, managed certs at the CDN. But automated renewals fail in three silent ways, and all three look like "it's fine, it renews automatically":
| Silent failure | How it bites | The check |
|---|---|---|
| DNS validation broke | DNS moved to a new provider, zone split, or the CNAME the ACME client uses was cleaned up "temporarily" | Quarterly dry-run (below), not a calendar promise |
| Renewed but never reloaded | The bot renews the file; the web server keeps serving the old cert from memory until restart | Probe the served cert's notBefore after each renewal |
| Bot account tied to a person | Renewal email goes to an ex-employee's inbox; nobody notices the failure mails | Bot registered to a shared alias you own |
certbot renew --dry-run, or cert-manager's self-check, or the CDN's "renew now".openssl s_client -connect host:443 -servername host and read the chain, not just the leaf.notBefore moved, not just the file on disk.When the warning page is live, sequence matters less than speed, but do it in this order:
| Control | Effort | Done looks like |
|---|---|---|
| Cert inventory, one table | 1 hr | Every TLS hostname has expiry, renewal method, owner |
| Expiry alerts on every hostname | 30 min | 30/14/7/3/1-day alerts land in the on-call channel |
| ACME automation for public hosts | half a day | No human renews the website cert |
| Quarterly renewal dry-run | 15 min/quarter | Staging renewed by robot; served cert's notBefore moved; chain verified |
| Expired-cert runbook | 30 min | One page: renew, deploy, verify from outside, tell customers, postmortem |
| Manual certs on a named calendar | 15 min | Mail/VPN/panel certs have an owner and a dated reminder that actually fires |
---
The Ops Starter Kit ($14) turns the first-hour chaos of any outage — expired cert included — into a fillable plan, and Vol. 2 ($27) adds the DR plan and evidence log for the review after. Launch week: 30% off any paid kit with code HIVE-LAUNCH30 at checkout (ends Sep 11, 23:30 ACST).