TOPIC DEVOPS
Incident Management the structured process of detecting, responding to, and learning from outages
IN 10 SECONDS
When a production incident happens — site down, errors spiking, data corrupted — you need a process, not panic. Incident management means: declare severity, assemble a response team (commander, comms, responders), communicate in a dedicated channel, mitigate (rollback, scale up, feature flag off), and follow up with a postmortem. Blameless culture is non-negotiable.
GOTCHA The goal of the postmortem is not to find who caused the incident — it's to find what system failures allowed it. Blameless postmortems lead to better actions and a culture where people report issues, not hide them. If the action item is 'be more careful', you're doing it wrong.
HOW AN INCIDENT UNFOLDS
01 Alert PagerDuty/OpsGenie pages the on-call engineer because an SLO burn rate alert is firing.
02 Triage the engineer acknowledges, assesses severity (SEV1 = customer-impacting outage, SEV2 = degraded, SEV3 = minor).
03 Response if SEV1/SEV2, the incident commander is assigned, a war room (Slack/Discord) is created, and responders jump in.
04 Mitigation priority is to restore service — rollback the deploy, scale out, enable a feature flag, or run a mitigation script.
05 Postmortem within 48h, a blameless root cause analysis is written with action items to prevent recurrence.
POKE IT YOURSELF
pd ack INCIDENT_ID — acknowledge a PagerDuty incident
curl -X POST -d '{"status":"resolved"}' https://api.pagerduty.com/incidents/INCIDENT_ID — resolve an incident via API
Drill this topic →
~85 sec read