OPS10-BP05 Define a customer communication plan for outages
Define and test a communication plan for system outages that you can rely on to keep your customers and stakeholders informed during outages. Communicate directly with your users both when the services they use are impacted and when services return to normal.
Desired outcome:
-
You have a communication plan for situations ranging from scheduled maintenance to large unexpected failures, including invocation of disaster recovery plans.
-
In your communications, you provide clear and transparent information about systems issues to help customers avoid second guessing the performance of their systems.
-
You use custom error messages and status pages to reduce the spike in help desk requests and keep users informed.
-
The communication plan is regularly tested to verify that it will perform as intended when a real outage occurs.
Common anti-patterns:
-
A workload outage occurs but you have no communication plan. Users overwhelm your trouble ticket system with requests because they have no information on the outage.
-
You send an email notification to your users during an outage. It doesn’t contain a timeline for restoration of service so users cannot plan around the outage.
-
There is a communication plan for outages but it has never been tested. An outage occurs and the communication plan fails because a critical step was missed that could have been caught in testing.
-
During an outage, you send a notification to users with too many technical details and information under your AWS NDA.
Benefits of establishing this best practice:
-
Maintaining communication during outages ensures that customers are provided with visibility of progress on issues and estimated time to resolution.
-
Developing a well-defined communications plan verifies that your customers and end users are well informed so they can take required additional steps to mitigate the impact of outages.
-
With proper communications and increased awareness of planned and unplanned outages, you can improve customer satisfaction, limit unintended reactions, and drive customer retention.
-
Timely and transparent system outage communication builds confidence and establishes trust needed to maintain relationships between you and your customers.
-
A proven communication strategy during an outage or crisis reduces speculation and gossip that could hinder your ability to recover.
Level of risk exposed if this best practice is not established: Medium
Implementation guidance
Communication plans that keep your customers informed during outages are holistic and cover multiple interfaces including customer facing error pages, custom API error messages, system status banners, and health status pages. If your system includes registered users, you can communicate over messaging channels such as email, SMS or push notifications to send personalized message content to your customers.
Customer communication tools
As a first line of defense, web and mobile applications should provide friendly and informative error messages during an outage as well as have the ability to redirect traffic to a status page. Amazon CloudFront
Custom API error messages can help detect and reduce impact when outages are isolated to discrete services. Amazon API Gateway
Direct messaging is the most personalized type of customer messaging. Amazon Pinpoint
Customer example
When the workload is impaired, AnyCompany Retail sends out an email notification to their users. The email describes what business functionality is impaired and provides a realistic estimate of when service will be restored. In addition, they have a status page that shows real-time information about the health of their workload. The communication plan is tested in a development environment twice per year to validate that it is effective.
Implementation steps
-
Determine the communication channels for your messaging strategy. Consider the architectural aspects of your application and determine the best strategy for delivering feedback to your customers. This could include one or more of the guidance strategies outlined including error and status pages, custom API error responses, or direct messaging.
-
Design status pages for your application. If you’ve determined that status or custom error pages are suitable for your customers, you’ll need to design your content and messaging for those pages. Error pages explain to users why an application is not available, when it may become available again, and what they can do in the meantime. If your application uses Amazon CloudFront you can serve custom error responses or use Lambda at Edge to translate errors and rewrite page content. CloudFront also makes it possible to swap destinations from your application content to a static Amazon S3
content origin containing your maintenance or outage status page . -
Design the correct set of API error statuses for your service. Error messages produced by API Gateway when it can’t reach backend services, as well as service tier exceptions, may not contain friendly messages suitable for display to end users. Without having to make code changes to your backend services, you can configure API Gateway custom error responses to map HTTP response codes to curated API error messages.
-
Design messaging from a business perspective so that it is relevant to end users for your system and does not contain technical details. Consider your audience and align your messaging. For example, you may steer internal users towards a workaround or manual process that leverages alternate systems. External users may be asked to wait until the system is restored, or subscribe to updates to receive a notification once the system is restored. Define approved messaging for multiple scenarios including unexpected outages, planned maintenance, and partial system failures where a particular feature may be degraded or unavailable.
-
Templatize and automate your customer messaging. Once you have established your message content, you can use Amazon Pinpoint or other tools to automate your messaging campaign. With Amazon Pinpoint you can create customer target segments for specific affected users and transform messages into templates. Review the Amazon Pinpoint tutorial to get an understanding of how-to setup a messaging campaign.
-
Avoiding tightly coupling messaging capabilities to your customer facing system. Your messaging strategy should not have hard dependencies on system data stores or services to verify that you can successfully send messages when you experience outages. Consider building the ability to send messages from more than one Availability Zone or Region for messaging availability. If you are using AWS services to send messages, leverage data plane operations over control plane operation to invoke your messaging.
Level of effort for the implementation plan: High. Developing a communication plan, and the mechanisms to send it, can require a significant effort.
Resources
Related best practices:
-
OPS07-BP03 Use runbooks to perform procedures - Your communication plan should have a runbook associated with it so that your personnel know how to respond.
-
OPS11-BP02 Perform post-incident analysis - After an outage, conduct post-incident analysis to identify mechanisms to prevent another outage.
Related documents:
Related examples:
Related services:
