PERF05-BP05 Use automation to proactively remediate performance-related issues
Use key performance indicators (KPIs), combined with monitoring and alerting systems, to proactively address performance-related issues.
Common anti-patterns:
-
You only allow operations staff the ability to make operational changes to the workload.
-
You let all alarms filter to the operations team with no proactive remediation.
Benefits of establishing this best practice: Proactive remediation of alarm actions allows support staff to concentrate on those items that are not automatically actionable. This helps operations staff handle all alarms without being overwhelmed and instead focus only on critical alarms.
Level of risk exposed if this best practice is not established: Low
Implementation guidance
Use alarms to trigger automated actions to remediate issues where possible. Escalate the alarm to those able to respond if automated response is not possible. For example, you may have a system that can predict expected key performance indicator (KPI) values and alarm when they breach certain thresholds, or a tool that can automatically halt or roll back deployments if KPIs are outside of expected values.
Implement processes that provide visibility into performance as your workload is running. Build monitoring dashboards and establish baseline norms for performance expectations to determine if the workload is performing optimally.
Implementation steps
-
Identify and understand the performance issue that can be remediated automatically. Use AWS monitoring solutions such as Amazon CloudWatch or AWS X-Ray to help you better understand the root cause of the issue.
-
Create a step-by-step remediation plan and process that can be used to automatically fix the issue.
-
Configure the trigger to automatically initiate the remediation process. For example, you can define a trigger to automatically restart an instance when it reaches a certain threshold of CPU utilization.
-
Use AWS services and technologies to automate the remediation process. For example, AWS Systems Manager Automation provides a secure and scalable way to automate the remediation process.
-
Test the automated remediation process in a pre-production environment.
-
After testing, implement the remediation process in production environment and continually monitor to identify areas for improvement.
Resources
Related documents:
Related videos:
Related examples:
