REL06-BP03 Send notifications with real-time processing and alarming
Real-time notifications alert responders and trigger automation the moment a metric crosses a threshold or a significant event occurs. The design challenge is setting thresholds that catch real problems without burying responders in noise.
Desired outcome:
-
Your alarms are based on service and application metrics that reflect workload health.
-
When an alarm threshold is breached, the appropriate personnel and systems are notified promptly.
-
Your notification routing sends automatable events to automation and human-judgment events to people.
-
Your alarm thresholds are reviewed and adjusted as usage changes.
Common anti-patterns:
-
Teams set alarm thresholds too high, so vital notifications never fire.
-
Teams set thresholds too low, so important alerts are ignored amid a flood of noise.
-
Teams leave alarms and thresholds unchanged as usage patterns shift.
-
Teams route alarms best handled by automation to people, which generates excessive notifications.
Benefits of establishing this best practice:
-
You detect issues early and respond to operational incidents quickly.
-
You reduce alert fatigue by tuning thresholds to real conditions.
-
You shorten mean time to detection for events that affect availability.
-
You free responders to focus on the events that need human judgment.
Level of risk exposed if this best practice is not established: High
Implementation guidance
Alarms are useful when the thresholds behind them are chosen with care. To avoid alert fatigue, treat thresholds as values you revisit as traffic grows and access patterns change.
Not every alarm needs to page a person. A useful distinction separates alarms that a machine can resolve from alarms that need human judgment. For example, automatic scaling can handle compute exhaustion without paging anyone. When a business metric drops for reasons the system cannot diagnose, a person needs to investigate. Decide the response path for each alarm before you create it, and send automatable events to automation.
Amazon EventBridge suits event-driven applications that react to events from your own applications, SaaS providers, and AWS services. It ingests events from more than 200 AWS services with no resources to create and uses a structured JSON event format. Use rules to select events and forward them to targets such as AWS Lambda, Amazon SQS, and Amazon SNS. Amazon Simple Notification Service (Amazon SNS) suits high fan-out delivery to thousands or millions of endpoints, with typical latency under 30 milliseconds. It supports Lambda, Amazon SQS, HTTP and HTTPS endpoints, SMS, mobile push, and email. A common pattern uses an EventBridge rule to filter events and Amazon SNS to fan them out to many subscribers.
Notifications need to cover both your own components and the health of the AWS services your workload depends on. AWS Health is the authoritative source for the state of your AWS resources. Route Health events to email and chat channels through AWS User Notifications, and integrate them programmatically through Amazon EventBridge. If you use AWS Organizations, aggregate Health events across accounts. Amazon CloudWatch generates the underlying alarms, supports both metric and composite alarms, and emits EventBridge events whenever an alarm changes state.
Implementation steps
-
Create metric and composite alarms: Use Amazon CloudWatch alarms to watch a single metric or an expression over a number of intervals. Use a composite alarm to combine the conditions of several alarms, so it enters alarm state only when the combined rule is met, which reduces noise from correlated signals.
-
Configure Amazon SNS notifications: Attach an Amazon SNS topic to each alarm so it notifies the right subscribers when the alarm changes state. Choose subscriber protocols that match how each audience works, such as email, chat, or SMS.
-
Create EventBridge rules for event-driven responses: Create rules in EventBridge that match specific CloudWatch alarm state changes and route them to targets. For example, an alarm for low disk space can trigger a Lambda function through a rule to reclaim space. See EventBridge targets for the supported destinations.
-
Add AWS Health notifications: Configure AWS User Notifications to deliver AWS Health events to email and chat, and integrate Health with EventBridge for programmatic handling. If you use AWS Organizations, aggregate Health events across accounts.
Resources
Related best practices:
Related documents:
Related videos:
Related examples:
Related services: