REL06-BP04 Automate responses (Real-time processing and alarming)
Use automation to take action when an event is detected, for example, to replace failed components.
Automated real-time processing of alarms is implemented so that systems can take quick corrective action and attempt to prevent failures or degraded service when alarms are triggered. Automated responses to alarms could include the replacement of failing components, the adjustment of compute capacity, the redirection of traffic to healthy hosts, availability zones, or other regions, and the notification of operators.
Desired outcome: Real-time alarms are identified, and automated processing of alarms is set up to invoke the appropriate actions taken to maintain service level objectives and service-level agreements (SLAs). Automation can range from self-healing activities of single components to full-site failover.
Common anti-patterns:
-
Not having a clear inventory or catalog of key real-time alarms.
-
No automated responses on critical alarms (for example, when compute is nearing exhaustion, autoscaling occurs).
-
Contradictory alarm response actions.
-
No standard operating procedures (SOPs) for operators to follow when they receive alert notifications.
-
Not monitoring configuration changes, as undetected configuration changes can cause downtime for workloads.
-
Not having a strategy to undo unintended configuration changes.
Benefits of establishing this best practice: Automating alarm processing can improve system resiliency. The system takes corrective actions automatically, reducing manual activities that allow for human, error-prone interventions. Workload operates meet availability goals, and reduces service disruption.
Level of risk exposed if this best practice is not established: Medium
Implementation guidance
To effectively manage alerts and automate their response, categorize alerts based on their criticality and impact, document response procedures, and plan responses before ranking tasks.
Identify tasks requiring specific actions (often detailed in runbooks), and examine all runbooks and playbooks to determine which tasks can be automated. If actions can be defined, often they can be automated. If actions cannot be automated, document manual steps in an SOP and train operators on them. Continually challenge manual processes for automation opportunities where you can establish and maintain a plan to automate alert responses.
Implementation steps
-
Create an inventory of alarms: To obtain a list of all alarms, you can use the AWS CLI
using the Amazon CloudWatch command describe-alarms. Depending upon how many alarms you have set up, you might have to use pagination to retrieve a subset of alarms for each call, or alternatively you can use the AWS SDK to obtain the alarms using an API call. -
Document all alarm actions: Update a runbook with all alarms and their actions, irrespective if they are manual or automated. AWS Systems Manager provides predefined runbooks. For more information about runbooks, see Working with runbooks. For detail on how to view runbook content, see View runbook content.
-
Set up and manage alarm actions: For any of the alarms that require an action, specify the automated action using the CloudWatch SDK. For example, you can change the state of your Amazon EC2 instances automatically based on a CloudWatch alarm by creating and enabling actions on an alarm or disabling actions on an alarm.
You can also use Amazon EventBridge
to respond automatically to system events, such as application availability issues or resource changes. You can create rules to indicate which events you're interested in, and the actions to take when an event matches a rule. The actions that can be automatically initiated include invoking an AWS Lambda function, invoking Amazon EC2 Run Command, relaying the event to Amazon Kinesis Data Streams, and seeing Automate Amazon EC2 using EventBridge. -
Standard Operating Procedures (SOPs): Based on your application components, AWS Resilience Hub recommends multiple SOP templates. You can use these SOPs to document all the processes an operator should follow in case an alert is raised. You can also construct a SOP based on Resilience Hub recommendations, where you need an Resilience Hub application with an associated resiliency policy, as well as a historic resiliency assessment against that application. The recommendations for your SOP are produced by the resiliency assessment.
Resilience Hub works with Systems Manager to automate the steps of your SOPs by providing a number of SSM documents you can use as the basis for those SOPs. For example, Resilience Hub may recommend an SOP for adding disk space based on an existing SSM automation document.
-
Perform automated actions using Amazon DevOps Guru: You can use Amazon DevOps Guru
to automatically monitor application resources for anomalous behavior and deliver targeted recommendations to speed up problem identification and remediation times. With DevOps Guru, you can monitor streams of operational data in near real time from multiple sources including Amazon CloudWatch metrics, AWS Config , AWS CloudFormation , and AWS X-Ray . You can also use DevOps Guru to automatically create OpsItems in OpsCenter and send events to EventBridge for additional automation.
Resources
Related best practices:
Related documents:
Related videos:
Related examples:
