REL11-BP01 Monitor all components of the workload to detect failures
Continually monitor the health of your workload so that you and your automated systems are aware of failures or degradations as soon as they occur. Monitor for key performance indicators (KPIs) based on business value.
All recovery and healing mechanisms must start with the ability to detect problems quickly. Technical failures should be detected first so that they can be resolved. However, availability is based on the ability of your workload to deliver business value, so key performance indicators (KPIs) that measure this need to be a part of your detection and remediation strategy.
Desired outcome: Essential components of a workload are monitored independently to detect and alert on failures when and where they happen.
Common anti-patterns:
-
No alarms have been configured, so outages occur without notification.
-
Alarms exist, but at thresholds that don't provide adequate time to react.
-
Metrics are not collected often enough to meet the recovery time objective (RTO).
-
Only the customer facing interfaces of the workload are actively monitored.
-
Only collecting technical metrics, no business function metrics.
-
No metrics measuring the user experience of the workload.
-
Too many monitors are created.
Benefits of establishing this best practice: Having appropriate monitoring at all layers allows you to reduce recovery time by reducing time to detection.
Level of risk exposed if this best practice is not established: High
Implementation guidance
Identify all workloads that will be reviewed for monitoring. Once you have identified all components of the workload that will need to monitored, you will now need to determine the monitoring interval. The monitoring interval will have a direct impact on how fast recovery can be initiated based on the time it takes to detect a failure. The mean time to detection (MTTD) is the amount of time between a failure occurring and when repair operations begin. The list of services should be extensive and complete.
Monitoring must cover all layers of the application stack including application, platform, infrastructure, and network.
Your monitoring strategy should consider the impact of gray failures. For more detail on gray failures, see Gray failures in the Advanced Multi-AZ Resilience Patterns whitepaper.
Implementation steps
-
Your monitoring interval is dependent on how quickly you must recover. Your recovery time is driven by the time it takes to recover, so you must determine the frequency of collection by accounting for this time and your recovery time objective (RTO).
-
Configure detailed monitoring for components and managed services.
-
Determine if detailed monitoring for EC2 instances and Auto Scaling is necessary. Detailed monitoring provides one minute interval metrics, and default monitoring provides five minute interval metrics.
-
Determine if enhanced monitoring for RDS is necessary. Enhanced monitoring uses an agent on RDS instances to get useful information about different process or threads.
-
Determine the monitoring requirements of critical serverless components for Lambda, API Gateway, Amazon EKS, Amazon ECS
, and all types of load balancers. -
Determine the monitoring requirements of storage components for Amazon S3, Amazon FSx, Amazon EFS, and Amazon EBS.
-
-
Create custom metrics to measure business key performance indicators (KPIs). Workloads implement key business functions, which should be used as KPIs that help identify when an indirect problem happens.
-
Monitor the user experience for failures using user canaries. Synthetic transaction testing (also known as canary testing, but not to be confused with canary deployments) that can run and simulate customer behavior is among the most important testing processes. Run these tests constantly against your workload endpoints from diverse remote locations.
-
Create custom metrics that track the user's experience. If you can instrument the experience of the customer, you can determine when the consumer experience degrades.
-
Set alarms to detect when any part of your workload is not working properly and to indicate when to automatically scale resources. Alarms can be visually displayed on dashboards, send alerts through Amazon SNS or email, and work with Auto Scaling to scale workload resources up or down.
-
Create dashboards to visualize your metrics. Dashboards can be used to visually see trends, outliers, and other indicators of potential problems or to provide an indication of problems you may want to investigate.
-
Create distributed tracing monitoring
for your services. With distributed monitoring, you can understand how your application and its underlying services are performing to identify and troubleshoot the root cause of performance issues and errors. -
Create monitoring systems (using CloudWatch or X-Ray
) dashboards and data collection in a separate Region and account. -
Create integration for Amazon Health Aware
monitoring to allow for monitoring visibility to AWS resources that might have degradations. For business essential workloads, this solution provides access to proactive and real-time alerts for AWS services.
Resources
Related best practices:
Related documents:
Related videos:
Related examples:
Related tools:
