OPS07-BP01 Perform post-incident analysis
A structured post-incident analysis examines the contributing factors behind a customer-impacting event and converts those findings into changes that reduce the chance of recurrence.
Desired outcome:
-
You run a consistent post-incident analysis for every customer-impacting event, triggered by your incident management process.
-
Your analysis identifies the multiple contributing factors behind an incident rather than a single root cause.
-
You capture the telemetry and timeline data needed to reconstruct what happened and when, and you automate its assembly so responders spend their time interpreting evidence rather than collecting it.
-
Your findings become tracked corrective actions with named owners and due dates, and a person remains accountable for each decision even when automation proposed it.
-
You share learnings across teams, so other workloads benefit from each incident.
Common anti-patterns:
-
Teams treat post-incident analysis as a search for who to blame, which discourages honest disclosure and hides the real contributing factors.
-
Responders resolve an incident and move on without recording the timeline, so the same failure recurs later with wider impact.
-
Organizations document a single root cause and stop, missing the chain of conditions that allowed the incident to happen.
-
Corrective actions are identified but never tracked to completion, so the analysis produces no lasting change.
-
Analysis happens only for the largest incidents, so smaller recurring incidents accumulate unaddressed.
-
Teams accept an automated root cause summary as the finished analysis, leaving the remaining contributing factors unexamined and the corrective actions unowned.
Benefits of establishing this best practice:
-
You identify the components, conditions, and actions that contributed to an incident, which reveals concrete opportunities for improvement.
-
You reduce recurrence because corrective actions address contributing factors instead of symptoms.
-
You shorten future detection and recovery times by improving alarms, runbooks, and escalation paths based on evidence.
-
You build trust because a blameless approach encourages people to share what they observed.
-
You spread knowledge across teams, so improvements from one incident protect other workloads.
-
You complete analyses sooner, and for more incidents, because automation absorbs the evidence gathering while your team's effort goes to judgment and deciding what to change.
Level of risk exposed if this best practice is not established: High
Implementation guidance
A deployment, a latent configuration error, a missing alarm, and a slow escalation can combine into an event that no single fix could avoid. Look for the contributing factors — the set of conditions and actions that together allowed the incident.
When people expect blame, they withhold detail, and the analysis lacks the information it needs. A blameless approach assumes responders acted reasonably on what they knew at the time and focuses on the system and process gaps that allowed the incident. Amazon uses a structured Correction of Error (COE) process built on this principle.
Reconstruct the timeline and record key measurements. Detection time is the gap between when the incident began and when an alarm or person noticed. Time to engage is how long it took to get the right responders involved. Time to mitigate and time to resolve measure how long customers felt the impact. These intervals let you ask whether detection was slow or the response was slow once detected. Your observability and logging must capture the underlying timestamps.
| Timeline point | What it marks |
|---|---|
| Incident start | When customer impact actually began |
| Detection | When an alarm or person first noticed the impact |
| Engagement | When the right responders were involved |
| Mitigation | When customer impact was halted or reduced |
| Resolution | When the incident was fully resolved |
Decide who facilitates the analysis, on what cadence you review findings, and which incidents warrant a full analysis. Analyzing every minor incident is expensive and dilutes attention. Ignoring small recurring incidents is dangerous, because a pattern of small failures often precedes a large one. By default, analyze every customer-impacting incident and any near miss that could have been severe. Tailor what you share to the audience: give engineering teams the technical detail, and give business stakeholders the impact and remediation.
Split the work: automation collects evidence, people apply judgment. Assembling the timeline, correlating deployments and configuration changes with the incident window, and pulling logs and metrics is mechanical, time-consuming work suited to automation. Judgment and accountability stay with named people. They decide which conditions contributed, judge what is worth changing, weigh a fix's cost against the risk it removes, and commit to own that fix.
Treat an agent's probable-cause summary as a starting hypothesis backed by evidence. Keep a human review step that asks what else had to be true for this incident to happen, and record the corrective actions under the names of the people who will complete them.
Use your team's existing incident tracking system. Trigger the analysis consistently, reconstruct the timeline from evidence, and track corrective actions to completion in that system. AWS Systems Manager OpsCenter can hold corrective actions as OpsItems alongside your other operational work. Many teams use a third-party incident management or ticketing system instead.
Use the Operational Readiness Review (ORR) approach to feed incident findings back into your readiness criteria. Creating ORR guidance from an incident walks through turning a single incident into a reusable readiness question. AWS couples its COE process to its ORR checklists so each incident produces new readiness checks.
Implementation steps
-
Trigger the analysis from your incident process: Make post-incident analysis a defined step in incident management, so it starts once a customer-impacting incident is closed rather than depending on manual follow-up. Confirm the scope criteria that decide which incidents and near misses receive a full analysis, and name who facilitates.
-
Reconstruct the timeline with metrics: Collect the deployment changes, configuration changes, incident start time, alarm time, engagement time, mitigation start time, and resolution time. Correlating deployment and infrastructure change events with the incident window cuts the manual effort of evaluating potential causes. AWS DevOps Agent
can do much of this correlation for you, investigating an incident across metrics, logs, and API change history and maintaining an investigation timeline you can carry into the analysis. Place these points on a single timeline so that detection, engagement, mitigation, and resolution intervals are visible and comparable. -
Identify contributing factors without blame: Work through the timeline to find the conditions and actions that combined to produce the incident. Assume responders acted reasonably on the information available, and document system and process gaps rather than individual fault. Amazon's Correction of Error (COE)
process is a working example of this blameless approach. Bring in all parties connected to the incident, including the people who support the system, customer advocates, and those affected internally, and anonymize the findings so the record cannot be read as assigning blame. Where an agent has already investigated, open the discussion with its output rather than a blank page. DevOps Agent publishes a root cause summary with the observations supporting it and the investigation gaps it encountered. Read the gaps as carefully as the conclusion, and have the group work outward from that hypothesis to the other conditions that had to hold for the incident to occur.
-
Ask targeted improvement questions: Once the timeline is settled, work through four areas of questioning, following the structure Amazon uses in its COE process
. For detection, ask when and how you learned there was customer impact, and what would halve the time to detect. For diagnosis, ask what underlying condition caused the impact and what would halve the time to diagnose. For mitigation, ask how you determined where to intervene, how you knew service was restored, and what would halve the time to mitigate. For prevention, ask not just what happened but why the underlying causes were possible. -
Drive prevention questions past the first answer: Apply the 5 Whys
to each candidate cause, asking why at each level until you have the full chain of conditions rather than the first plausible explanation. Keep it blame-free by pursuing why rather than who. Where an answer is "human error," treat that as a signal to ask why the error was possible, since it usually indicates a missing validation or fail-safe. Continue past five whys if the chain demands it, and plan a remediation for each answer, not only the last one. -
Create and track corrective actions: Convert each finding into a specific action stating its priority, the person responsible, and a due date, and track it to completion in your normal work-tracking system so it competes for priority alongside other work. Amazon's COEs average six to eight action items against an internal SLA measured in weeks, which is a useful sanity check on whether your analysis produced a realistic amount of work. Treat unfinished actions as open risk, because an analysis that produces no completed change delivers no value.
-
Communicate findings to the right audiences: Publish the analysis where teams can find it, and tailor the message to each audience. Give engineering teams the technical detail and contributing factors, and give business stakeholders the customer impact and the remediation plan. The resilience lifecycle "Respond and learn" stage describes sharing these reports widely across the business so other teams benefit.
Resources
Related best practices:
Related documents:
Related examples:
Related services: