HIVE80lab — Ops notes

On-Call Escalation Slack Structure

Purpose: Define Slack channels, roles, and messaging conventions for on-call escalation. Move from chaotic chat to structured incident communication.

Incident Channels

Channel NamePurposeMembersAuto-InviteUrgency
#incident-sreTier 1 SLO breachSREs, Infra TeamYes (PagerDuty)P1
#incident-leadTier 2 business impactSenior Leads, ProductNo (human)P2
#incident-criticalTier 3 major incidentsC-level, Legal, CommsNo (human)P3
#oncall-updatesNon-critical updatesAll Ops, WeeklyNoP4

Channel Rules

Alert Message Format

P1 / Tier 1 Alert

`` 🚨 P1 - SLO BREACH DETECTED Service: [service-name] SLA: [SLA name] ([SLA %]) ([downtime allowed]) Metric: [metric-name] ([value]) Window: [last X minutes] Impact: [business impact] Responder: @sre-oncall Status: [investigating / observed / detected] ``

P2 / Tier 2 Update

`` 🏥 INCIDENT UPDATE (Tier 2 - [time elapsed]) Status: [resolved / under control / escalating] Root cause: [one sentence summary] Progress: [what was tried] Next action: [what will happen next] Est. resolution: [time or "ongoing"] ``

P3 / Tier 3 Update

`` 🚨 MAJOR INCIDENT - ACTION REQUIRED Service: [service-name] Impact: [revenue / customers / compliance] Responder: [all-critical-team] Plan: [high-level plan] Customer comms: [plan] Next update: [time] ``

P4 / Tier 4 Update

`` 📝 Incident Update (Tier 4 - [time elapsed]) Status: [resolved / paused] Root cause: [summary] Resolution: [what was fixed] Lessons learned: [summary] ``

Escalation Workflow

Step 1: Initial Alert (Tier 1)

1. Automated alert fires → posted to #incident-sre 2. @sre-oncall notified → must respond within 15 min 3. Message format: Use 🚨 P1 - SLO BREACH DETECTED format 4. First responder links incident timeline

Step 2: Tier 2 Escalation

1. If unresolved after 15 min → escalate to #incident-lead 2. Senior lead notified → must respond within 30 min 3. Message format: Use 🏥 INCIDENT UPDATE (Tier 2 - X min) format 4. SRE leads: Keep #incident-sre updated on progress

Step 3: Tier 3 Escalation

1. If unresolved after 30 min → escalate to #incident-critical 2. CTO / Director notified → stand-up call triggered 3. Message format: Use 🚨 MAJOR INCIDENT - ACTION REQUIRED format 4. Plan finalized: Customer comms, resources allocated

Step 4: Tier 4 Follow-up

1. After 1 hour or resolution → move to #oncall-updates 2. Post-mortem drafted → linked in incident timeline 3. Lessons learned shared → team weekly sync

On-Call Message Templates

Acknowledgement Template

`` 👀 Received - investigating now ``

Update Template (Progress)

`` Update: Testing hypothesis X Timeline: Est. 15 min to verify ``

Escalation Template

`` Escalating to Tier 2 - no resolution after 15 min Please join: #incident-lead ``

Resolution Template

`` ✅ Resolved - check incident timeline for details RCA: [link to post-mortem] Lessons learned: [one sentence] ``

Escalation Timer Setup

Slack App Integration

- Tier 1: 15 min timeout - Tier 2: 30 min timeout - Tier 3: 1 hour timeout

- Tier 1 → #incident-sre only - Tier 2 → #incident-lead only - Tier 3 → #incident-critical only

Manual Escalation

Channel Naming Conventions

Active Incidents

Historical Incidents

Archive Rule

Documentation Fields

Best Practices

For First Responders

For Senior Leads

For C-Level

For Everyone

Usage: Use for every incident. Follow the channel structure for Tier 1-4 escalation. Works with incident-timeline-template.html for structured tracking.

Product links: /l/ops-starter-kit-vol-2 | /l/ops-starter-kit | /l/automation-starter-pack

From the HIVE80lab kit

Part of the five-pillar incident-response set: see the pillars overview and the blameless post-incident review template.