View a markdown version of this page

OPS06-BP02 Automate responses to events - AWS Well-Architected Framework

OPS06-BP02 Automate responses to events

Manual handling of routine operational events is slow, inconsistent, and hard to scale. Automate the response to well-understood events to act immediately, apply the same remediation every time, and free operators for problems that need judgment.

Desired outcome:

  • Your workload responds automatically to well-understood events without waiting for an operator.

  • Your automated responses apply the same remediation every time, removing variation between operators.

  • Your operators spend their attention on novel problems, not on repetitive, mechanical tasks.

  • Your automation is version-controlled and tested, so you trust it to run unattended.

Common anti-patterns:

  • Teams handle repetitive, well-defined events manually, which delays resolution and introduces mistakes.

  • Operators run the same manual remediation for a known event many times a week.

  • Critical but repetitive tasks stay manual, so alert fatigue causes teams to miss real issues.

  • Automation runs without testing or version control, so a faulty script causes more damage than the event it addresses.

Benefits of establishing this best practice:

  • Event responses accelerate, which reduces downtime.

  • Operations stay reliable because automated handling is consistent and repeatable.

  • Operators reclaim time for work that requires human judgment.

  • Configuration drift and compliance gaps close automatically as they appear.

Level of risk exposed if this best practice is not established: Medium

Implementation guidance

Not every event is a good candidate for automation. Automate events that occur often enough to justify the effort, have a well-understood cause, and have a remediation safe to run unattended. Leave rare events, or events whose response depends on context, to an operator with a runbook.

Event-driven automation connects a trigger, such as a metric crossing a threshold or a log entry, to a remediation action. With detection separated from response, one event can drive notification, remediation, and record-keeping. Define the trigger precisely so that it fires on the precise event you want to catch.

Unattended automation is risky. A buggy remediation, or one triggered by a mismatched condition, can cause more damage than the original event and spread across many resources before anyone notices. Treat automation as production code: keep it in version control, test it against realistic scenarios, and constrain its impact scope so a single faulty run cannot cascade. Prefer idempotent actions (those that produce the same result whether they run once or many times) over actions that accumulate side effects.

Automation pays off for remediation of known failures, capacity and scaling adjustments, deployment and release tasks, security responses, and drift correction. Start where the manual toil is highest and the risk of a mistake is lowest, then expand as your confidence and test coverage grow.

Implementation steps

  1. Identify automation opportunities: List the repetitive tasks your teams perform, such as issue remediation, ticket enrichment, capacity management, scaling, deployments, and testing. Rank them by how often they occur and how well their remediation is understood, and automate the safest, highest-toil tasks first.

  2. Define the triggers: Set the conditions that initiate each automated response. Use Amazon CloudWatch alarm actions for metric thresholds, Amazon EventBridge to match events from AWS services and applications, and metric filters on log events for conditions found in logs.

  3. Automate remediation and maintenance with runbooks: Use AWS Systems Manager Automation runbooks to run maintenance, deployment, and remediation tasks without manual steps. When a work item opens in AWS Systems Manager OpsCenter, attach and run automation runbooks against it so common remediation runs as soon as the item opens and responders start with context.

  4. Automate capacity and scaling: Monitor quotas with the Quota Monitor for AWS solution so you act before you hit a limit, and adjust capacity automatically with AWS Auto Scaling to hold availability and performance.

  5. Automate delivery and validation: Automate build and release work with AWS CodePipeline, and continually validate endpoints and APIs with synthetic monitoring canaries.

  6. Automate risk mitigation and compliance: Apply automated security responses to address findings quickly, hold configuration steady with AWS Systems Manager State Manager, and remediate noncompliant resources with AWS Config rules.

  7. Augment automated response with agentic AI: For events whose remediation is not fully predetermined, use AWS DevOps Agent to investigate autonomously, correlate telemetry and deployment history, and recommend or apply targeted remediation, so investigation and recovery keep pace with events that a fixed script alone cannot handle.

Resources

Related best practices:

Related documents:

Related examples:

Related workshops:

Related services: