HIVE80lab — Ops notes

Configuration Drift Audit for Small Teams

Every small team has a machine that stopped matching its documentation months ago. A hand-edit during an outage, a cron job added at 2am, a package upgraded “just this once.” None of it went through the repo, so none of it survives a rebuild — and nobody discovers that until the day the rebuild is the emergency. Drift is not a scandal; it is uncommitted work running your business. The audit is a read-only pass, once a quarter, that diffs production against the baseline and turns every difference into a decision: fix forward, revert, or declare. Run it on a quiet Tuesday, not mid-incident.

The audit (read-only until step 5)

  1. Pick the three boxes that hurt most. You are not auditing the fleet; you are auditing the snowflakes — the machines nobody has rebuilt since onboarding, the one with the “temporary” edit, the box only one person can deploy to. If you don’t know which three those are, that fact is the audit’s first finding.
  2. Export the live config, read-only. Installed packages, running services, crontabs, firewall rules, env files (keys redacted — diff the names, never the values), mount points, kernel/sysctl settings. One command per box if you can; the point is a snapshot you can diff, not a change session. Nothing gets modified during the audit — that discipline is the whole game.
  3. Diff against the baseline. Baseline means: the repo, the infrastructure code, or the last audited tag. If none exists, the baseline is today’s export and step one next quarter is comparing against it — a baseline you create now beats a perfect one you never create.
  4. Sort the diffs by blast radius. What breaks customers first if this box dies? A changed backup schedule is bigger than a changed MOTD. Sort before you triage, because step 5 is a budget and you will not clear the whole list in one sitting.
  5. Classify every diff, three buckets only. Fix forward — production is right, the repo is stale: the diff becomes a PR (this is most diffs, and it is the happy path, because the change already shipped and someone depended on it). Revert — production is wrong: put prod back and note why. Declare — production must differ (a CAN bus of licensing quirks, a vendor-required sysctl, a bigger swap on the reporting box): write it down as a documented exception with an owner and a review date. Anything not in one of the three buckets is a diff you are hiding from yourself.
  6. Fix-forward first, as PRs. Every fix-forward diff is a pull request against the baseline, reviewed like any other change. Production is the source of truth until the PR merges; then the repo is. If the PR would be rejected as-is, that is not a merge blocker — it is a finding: the hand-change encoded a decision nobody documented.
  7. Revert every “temporary” edit older than a sprint. If it was temporary in spirit but load-bearing in fact, that is exactly what the declare bucket is for. What you must not do is leave it silently — the silent temporary is the diff that takes the rebuild down with it.
  8. Find the pipeline gap, not the person. For each diff, answer one question: which process should have carried this change and didn’t? A 2am hand-edit means the break-glass path worked but the backport never happened (the hotfix process owns that receipt); a hand-edited crontab means there was no repo for cron at all. The fix is a missing script, a missing repo, or a missing step — never a conversation about who to be mad at.
  9. Close the gap while the audit is warm. Anything done by hand twice becomes a script; anything scripted goes in the repo; anything in the repo gets deployed by the pipeline. Small closes: one crontab dir added to git, one env template added, one break-glass backport check added to the checklist you already run.
  10. Re-baseline and book the next one. Tag the audited state, log the audit’s findings in the decision log, and put the next audit in the calendar — quarterly for the snowflakes, and a cheap monthly hash check (cron a config checksum and diff it) so drift between audits trips an alarm instead of a surprise. A drift audit that happens “when we have time” is a rebuild that happens when you don’t.

Five traps

Worked example

A nine-person analytics company, twelve staff-facing servers, no infra code. The old way: the reporting box’s disk died on a Thursday; rebuild from “the docs” took three days, and six weeks later finance asked why invoices had stopped reconciling — a 4am cron the dead box ran had never been documented, and 410 invoices had silently double-billed. Cost: three days of rebuild, a credits batch, and an auditor’s raised eyebrow.

The rerun: quarterly read-only audit. The export diffed against a baseline tag they created that same morning found seven diffs across the three worst boxes: five fix-forwarded as PRs (including the missing reconcile cron, now in a git-tracked crontab dir deployed by the pipeline), one reverted (a debug logging flag someone left at verbose in prod, quietly eating 14% of disk), one declared (the reporting box’s vendor-required sysctl, documented with owner and review date). The monthly hash alarm caught its first drift eleven weeks later — a package installed during a support call — and the whole loop was one PR, not one surprise. Rebuild-from-repo time for all three boxes: under an hour, measured, because they made it a drill.

Metrics (for the drift loop itself)

From the HIVE80lab kit

Related: the post-deploy verification is the ten minutes after a deploy where fresh drift is cheapest to catch; the change management checklist is what keeps changes riding the pipeline that the audit then diffs against; and the break-glass account checklist is where emergency hand-changes come from — the audit is how their backports stop getting forgotten. and when the drift audit surfaces an auth-shaped emergency, the SSO outage runbook is what runs while the break-glass edits the audit later reconciles