Partitional services best practices
This section provides best practices for using partitional services in your recovery operations.
This section covers the following topics:
AWS Identity and Access Management
The IAM control plane consists of all public IAM APIs
(including Access Advisor but not IAM Access Analyzer or IAM
Roles Anywhere). This includes actions like CreateRole,
AttachRolePolicy, ChangePassword,
UpdateSAMLProvider, and UpdateLoginProfile.
The IAM data plane provides authentication and authorization for
IAM principals in each AWS Region. During a control plane
impairment, CRUDL type operations for IAM might not work, but
authentication and authorization for existing principals will
continue to work.
When you’re planning for dependencies on IAM, you shouldn’t rely on the IAM control plane in your recovery path. For example, a statically stable design for a break-glass admin user would be to create a user with the appropriate permissions attached, have the password set and the access key and secret access key provisioned, and then lock those credentials in a physical or virtual vault. When required during an emergency, retrieve the user credentials from the vault and use them as needed.
During a failover, you should not rely on the IAM control plane.
Instead, pre-provision IAM roles to use with resources that you
provision as part of your recovery. For example, use an existing
role with a Lambda
A non-statically stable approach would be to provision IAM users during a failure, or having the user pre-provisioned, but only attaching the admin policy when required. Similarly, provisioning IAM resources during a failover process, for example a backup and restore approach where you launch IaC, would also not be statically stable. Both approaches depend on the IAM control plane to succeed.
AWS Organizations
The AWS OrganizationsAcceptHandshake, AttachPolicy,
CreateAccount, CreatePolicy, and
ListAccounts. There’s no data plane for AWS Organizations. The
service orchestrates the data plane for other services like IAM.
During a control plane impairment, CRUDL type operations for
Organizations might not work, but the policies, like service
control policies (SCP), resource control policies (RCP), and tag
policies, will continue to work and be evaluated as part of the
IAM authorization process. Delegated admin capabilities and
multi-account features in other AWS services that are supported
by Organizations will also continue to work.
When you’re planning for dependencies on AWS Organizations, you shouldn’t
rely on the Organizations control plane in your recovery path.
Instead, implement static stability in your recovery plan. For
example, a non-statically stable approach might be to update SCPs
to remove restrictions on allowed AWS Regions using the
aws:RequestedRegion condition, or to enable admin
permissions for specific IAM roles. This relies on the
Organizations control plane to make these updates. A statically
stable approach would be to use
session
tags to grant the use of admin permissions. Your identity
provider (IdP) can include session tags that can be evaluated
against the aws:PrincipalTag condition, which helps
you to dynamically configure permissions for certain principals
while allowing your SCPs and RCPs to remain static. This removes
dependencies on the Organizations control plane.
AWS Account Management
The AWS Account Management control plane consists of all
public
APIs for managing an AWS account, such as
GetContactInformation and
PutContactInformation. It also includes creating or
closing a new AWS account through the management console. The APIs
for CloseAccount, CreateAccount,
CreateGovCloudAccount, and DescribeAccount
are part of the AWS Organizations control plane.
Creating
an AWS GovCloud (US) account relies on the AWS Account
Management control plane. Also, AWS GovCloud (US) accounts
must
be 1:1 linked to an AWS account in the aws
partition. Creating accounts in the aws-cn and
aws-eusc partitions doesn’t rely on resources in the
aws partition. The data plane for AWS accounts is the
accounts themselves. During a control plane impairment, CRUDL-type
operations (like creating a new account or getting and updating
contact information) for AWS accounts might not work. References to
the account in IAM policies will continue to work.
When you’re planning for dependencies on AWS Account Management, you shouldn’t rely on the account management control plane in your recovery path. Although the account management control plane doesn’t provide functionality that you would typically use in a recovery situation, there might be times when you would. For example, a statically stable design would be to pre-provision all the AWS accounts you need for failover. A non-statically stable design would be to create new AWS accounts during a failure event to host your failover resources.
Amazon Application Recovery Controller (ARC)
Amazon Application Recovery Controller (ARC) provides three separate functions: routing control, Region switch, and zonal shift. Zonal shift is a regional service. The global control plane for ARC routing control consists of the Recovery Readiness API and the Recovery Control Configuration API. The control plane for the Region switch API is also global. These are documented in the Amazon Application Recovery Controller endpoints and quotas. You manage readiness checks, routing control setup, cluster operations, and Region switch plans by using these global control planes.
The recovery control data plane of ARC is your recovery cluster,
which manages the routing control values that are queried by Route 53
health checks and implements the safety rules. The
data
plane functionality of ARC is accessed through your
recovery cluster APIs such as
https://aaaaaaaa.route53-recovery-cluster.eu-west-1.amazonaws.com.
Region switch has independent data planes in each AWS Region. You should use the data plane for recovery actions, that is, for executing Region switch plans. For a list of the data plane operations, see the Region switch API operations.
You shouldn’t rely on the ARC global control plane in your recovery
path. For ARC recovery control, there are two
best
practices that help implement this guidance. First,
bookmark or hard code the five regional cluster endpoints. This
removes the need to use the DescribeCluster control
plane operation during a failover scenario to discover the endpoint
values. Second, use the ARC cluster APIs by using the AWS CLI or
AWS SDK to perform updates to routing controls instead of the
console. This removes the console as a dependency for your failover
plan and helps make sure that it depends only on data plane
actions.
AWS Network Manager
The AWS Network Manager service is primarily a control plane-only
system. Its purpose is to centrally manage the configuration of
your AWS Cloud wide area networking (WAN) core network and your
AWS Transit Gatewayus-west-2, which can also be accessed through the CloudWatch
data plane. If Network Manager is impaired, the data plane of the
services it orchestrates will not be impacted. The CloudWatch metrics for
Cloud WAN are also available in us-west-2. If you want
historical metric data, like bytes in and out per Region, to
understand how much traffic might shift to other Regions during a
failure impacting us-west-2, or for other operational
purposes, you can export those metrics as CSV data directly from
the CloudWatch console or using this method:
Publish
Amazon CloudWatch metrics to a CSV file. The data can be found under
the AWS/Network Manager namespace and you can perform
this on a schedule you choose and store it in Amazon S3 or in another
data store you select. To implement a statically stable recovery
plan, don’t use AWS Network Manager to make updates to your
network or rely on data from its control plane operations for
failover input.
Amazon Route 53
The Route 53 control plane consists of all public Route 53 APIs covering functionality for
hosted zones, records, health checks, DNS query logs, reusable delegation sets, traffic
policies, and cost allocation tags. It’s hosted in us-east-1 with an
optional accelerated recovery option for
public hosted zones in us-west-2. Domain registration is only hosted in
us-east-1.
The data plane for Route 53 consists of three parts, the public
authoritative DNS resolution service, the private DNS service, and
the health check service. In the aws partition, the
data plane for public hosted zones runs across hundreds of PoP
locations and in each AWS Region. In other partitions like
aws-us-gov and aws-cn, the data plane for
public hosted zones runs in multiple AWS Regions in the partition.
The data plane for private hosted zones runs in each AWS Region in
the partition. The data plane for the health check service runs
across multiple AWS Regions where it performs the health checks,
aggregates the results, and delivers them to the data plane of Route 53
public and private DNS and AGA. During a control plane impairment,
CRUDL-type operations for Route 53 might not work, but DNS resolution
and health checks, and updates to routing resulting from changes in
health checks, will continue to work.
The accelerated recovery option is best suited for situations where you must provision new resources that have new IP addresses that can’t be pre-provisioned and failed over with Route 53 health checks. For example, if you’re using service discovery and you need to replace a failed Amazon EC2 instance with a new one. When you’re planning for dependencies on Route 53, you shouldn’t rely on the Route 53 control plane in your recovery path for failover and recovery between Regions or to shift away from an Availability Zone. Instead, you should implement a statically stable design that uses the status of health checks to perform multi-Region failover or use zonal shift for Multi-AZ recovery. Route 53 health checks can monitor:
Amazon Application Recovery Controller (ARC) routing controls
HTTP/S and TCP endpoints
CloudWatch alarms
Calculated health checks
When configuring Route 53 health checks, you should have a manual way
to force failover and optionally, an automated approach as well.
Amazon ARC routing controls and the
Amazon ARC Region Switch health check execution block options
provide this capability. You can also
perform manual failover using an HTTP/S or TCP endpoint where you
control the response, like an Amazon S3 bucket as described in
Creating
Disaster Recovery Mechanisms using Route 53
When using the CloudWatch alarm approach there are several important things to consider. First, the health checks for CloudWatch alarms are performed in two Availability Zones in three Regions to gather six signals (as opposed to the eight Regions that are used for HTTP/S and TCP endpoints). The health checkers must still consider at least 18% of their results to be healthy to consider the health check to pass, which requires two or more data points for CloudWatch alarms.
Next, the health checkers must collect data from CloudWatch metrics to make their health
evaluation. If there are no data points in the metric data, Route 53 relies on the
InsufficientDataHealthStatus, which might be healthy, unhealthy, or last
known status. If the metric will be unavailable frequently or for long periods (longer than
a few hours), we recommend that you not use the last known status. If the metric is unavailable, meaning it
can’t be retrieved by the health checker rather than only missing data, the health
checks always use the last known status.
If the alarm is in the same Region that’s impaired, for either
automatic or manual failover, the health check system might not be
able to retrieve CloudWatch metrics and will continue to use the last
known status. To help make sure correlated impairments to CloudWatch in
the Region you’re trying to fail away from don’t prevent you from
failing over, you should place the alarm in an alternate Region. For
example, if you primarily operate in us-east-1, you can
place the alarm in us-west-2. If the alarm is
monitoring application metrics, they would also need to exist in the
other Region, which might require updates to your observability
solution.
To implement a pattern that allows for both automated failover and
manual failover, you can use a calculated health check in Route 53
where one of the checks is based on your alarm in the primary Region
and another monitors an option where you control the response to the
health check. You would configure the health check to use Report
healthy when all health checks are healthy (AND). This makes sure
that if either target is unhealthy, the overall record is considered
unhealthy.
A non-statically stable design would be to update the value of a
Route 53 resource record using the ChangeResourceRecordSets
API, change the weight of a weighted record, or create new records
to perform failover. These approaches depend on the Route 53 control
plane.
Amazon CloudFront
The Amazon CloudFront control plane consists of all public CloudFront APIs for managing distributions. The data plane is the distribution itself served from the PoPs in the edge network. It performs the request handling, routing, and caching of your origin content. During a control plane impairment, CRUDL-type operations for CloudFront (including invalidation requests) might not work, but your content will continue to be cached and served, origin failovers will continue to work, and any associated CloudFront functions will continue to execute.
When you’re planning for dependencies on CloudFront, you shouldn’t rely on the
CloudFront control plane in your recovery path. For example, a statically stable design would be
to use automated origin failovers to mitigate the impact from an impairment to one of your
origins. Alternatively, you can use approaches described in the preceding Amazon Route 53 section to failover to a different CloudFront origin using DNS. You
can also combine these approaches as described in Improve web application availability with CloudFront and Route 53 hybrid origin failover
Lambda@Edge
The
Lambda@Edge
service runs its data plane in the CloudFront Regional Edge Caches
(REC). The service’s control plane is hosted in
us-east-1. The PoP location that receives the request
forwards it to a pre-determined REC to handle the function
invocation. You can’t control which Region handles Lambda@Edge
invocations from specific PoPs, but CloudFront can shift invocations to
other RECs when they observe a service disruption in one Region.
Don’t rely on deploying or updating Lambda@Edge functions as part
of your recovery path. Instead, a statically stable design would
be to use dynamic lookups to the data plane of other services. For
example, instead of changing the function code to rewrite the
origin domain of a request, use a
DynamoDB
Although very unlikely, internet routing protocols might send a
user request to a PoP that isn’t the closest geographically to
the user. CloudFront doesn’t have control over this routing and as a
result, an unexpected REC might process a user request. If
Lambda@Edge invocations are impaired in a Region, users from other
geographic areas might see impact. If the event is impacting
services in us-east-1, both Lambda@Edge invocations
and the control plane for Lambda@Edge or CloudFront might be impacted
simultaneously.
As a result, you may not be able to recover from the service disruption by disabling Lambda@Edge processing or modifying your CloudFront distribution. Lambda@Edge can shift invocations away from a Region on your behalf in response to a service disruption. However, depending on your use case, you should consider whether CloudFront Functions, which execute in the PoP rather than in a Region, provides a more resilient alternative.
Amazon CloudFront Functions
CloudFront
Functions is a global service whose control plane is
hosted in us-east-1 and consists of all public APIs
for the service. The service’s data plane runs in each PoP
location alongside your distribution. Don’t rely on deploying or
updating CloudFront Functions as part of your recovery path. Instead, a
statically stable approach for recovery from an origin disruption
would be to use automatic origin failover or origin failover with
DNS as described in the preceding section.
Amazon CloudFront KeyValueStore
The
CloudFront
KeyValueStore is a global service hosted in
us-east-1 that consists of the APIs for creating,
updating, reading, and deleting key stores and values. The
contents of each KeyValueStore are distributed to all edge
locations for low latency reads and can be performed without
dependency on us-east-1.
CloudFront functions can be used to
rewrite
the request origin to perform dynamic origin failover. If
your function looks up the health status of each origin from a
KeyValueStore to perform dynamic failover, updates to that health
status key value depends on the CloudFront KeyValueStore API in
us-east-1, which might not succeed if the API in
us-east-1 is impaired. A statically stable approach
for recovery from an origin disruption would be to use automatic
origin failover or origin failover with DNS as described in the
preceding section.
AWS Certificate Manager (ACM) for CloudFront
If you’re using custom certificates with your CloudFront distribution,
you also have a dependency on ACM. Using custom certificates with
your CloudFront distribution relies on the ACM control plane in
us-east-1. During a control plane impairment, your
existing certificates configured in your distribution will
continue to work as will automatic certificate renewals. Don’t
rely on changing the distribution’s configuration or creating new
certificates as part of your recovery path.
AWS Web Application Firewall (WAF) and WAF Classic
If you’re using AWS WAFus-east-1. During a control plane impairment, the
configured web access control lists (ACLs) and their associated
rules continue to function. Don’t rely on updating your WAF web
ACLs as part of your recovery path.
AWS Global Accelerator
The Global Accelerator control plane consists of all public Global Accelerator APIs. The data plane is the network routing of the anycast IP addresses provided by Global Accelerator to your registered endpoints. Global Accelerator also uses Route 53 health checks to determine the health of your Global Accelerator endpoints, which is part of the Route 53 data plane. During a control plane impairment, CRUDL-type operations for Global Accelerator might not work. Routing to your existing endpoints, along with existing health checks, traffic dials, and endpoint weight configurations used to route or shift traffic to other endpoints and endpoint groups will continue to work.
When you’re planning for dependencies on AGA, you shouldn’t rely on
the Global Accelerator control plane in your recovery path. For
example, a statically stable design would be to use the status of
the configured health checks to fail away from unhealthy endpoints.
See
Deploying
multi-Region applications in AWS using AWS Global Accelerator
AWS Shield Advanced
The AWS Shield Advanced control plane consists of all public
Shield Advanced APIs. This includes functionality like
CreateProtection, CreateProtectionGroup,
AssociateHealthCheck, DescribeDRTAccess,
and ListProtections. The data plane is the distributed
denial of service (DDoS) protection provided by Shield Advanced and
the creation of Shield Advanced metrics. Shield Advanced also uses
Route 53 health checks (which are part of the Route 53 data plane) if
you’ve configured them. During a control plane impairment,
CRUDL-type operations for Shield Advanced might not work, but the
DDoS protection configured for your resources, in addition to
responses to changes in health checks, will continue to function.
When planning for dependencies on Shield Advanced, you shouldn’t rely on its control plane in your recovery path. Although the Shield Advanced control plane doesn’t provide direct functionality that you would typically use in a recovery situation, there might be certain situations when you would. For example, a statically stable design would be to have your DR resources already configured to be part of a protection group and have health checks associated with them as opposed to configuring that protection after the failure occurs. This prevents depending on the Shield Advanced control plane for recovery.
