View a markdown version of this page

REL11-BP03 Automate healing on all layers - AWS Well-Architected Framework

REL11-BP03 Automate healing on all layers

When a component fails, automated actions should remediate it. Some degradations heal through a service's own internal mechanisms, while others need a resource to be restarted, replaced, or removed. Automate healing on all layers to improve recovery time and reduce manual intervention.

Desired outcome:

  • Automated actions remediate failures at each layer of the workload without manual intervention.

  • Stateless components are replaced rather than repaired, so a restart is a safe and reliable recovery action.

  • Databases fail over or promote replicas through automation, not a manual runbook.

  • Clusters that do not sit behind a load balancer are sized to survive the loss of at least one node while they recover.

  • Event-driven automation detects a failure signal and runs remediation logic in response.

Common anti-patterns:

  • Teams provision resources without autoscaling.

  • Teams deploy applications on individual instances or containers with no group to replace them.

  • Applications that could run in multiple locations are deployed without automatic recovery.

  • Teams manually heal the failures that autoscaling and automatic recovery were meant to handle.

  • Databases have no automated failover.

  • Traffic has no automated way to reach new endpoints after a failure.

Benefits of establishing this best practice:

  • Mean time to recovery drops because remediation runs the moment a failure is detected.

  • Recovery is consistent because automation performs the same steps every time.

  • Operators are freed from routine restarts and can focus on root causes.

  • Stateless, replaceable components turn most failures into a simple resource swap.

  • Event-driven remediation extends healing to failures that autoscaling alone cannot address.

Level of risk exposed if this best practice is not established: High

Implementation guidance

Automated healing depends on an early design choice: statelessness. When a component holds no local state, you can replace it instead of repairing it. Replacement is far more reliable than repair because it returns the component to a known-good starting point. In the cloud, the cleanest recovery is often to discard the failed compute instance or function and launch a replacement. Where state is unavoidable, isolate it in a managed data store so the compute tier stays disposable.

Handle network calls like component failures. A timeout and an error response from a dependency have the same effect on your system. Apply one strategy to both: a limited number of retries with exponential backoff and jitter. The limit prevents retry traffic from overwhelming the dependency, the backoff spreads attempts over time, and the jitter prevents many clients from retrying at the same time.

Stateless tiers behind a load balancer heal through an Auto Scaling group that replaces unhealthy members. Clustered systems that manage their own membership, such as search or streaming clusters, must be sized to keep running after losing a node. They can then serve at reduced capacity while a replacement joins. For large-scale loss, such as an entire Availability Zone, provisioning replacements on demand is the least reliable option. It assumes both a working control plane and spare capacity exactly when both are least certain. Static stability, which holds the spare capacity before the event, is the more resilient choice for that scale.

Not every failure fits a built-in recovery feature. When autoscaling and automatic recovery do not apply or fail, use an event-driven pattern. A failure signal such as an alarm or a state change invokes remediation logic that you define. This pattern extends healing beyond compute to almost any resource. Because you write the recovery logic, test it as carefully as the workload it protects.

Implementation steps

  1. Heal stateless tiers with Auto Scaling groups: Deploy stateless application tiers into Auto Scaling groups so unhealthy members are replaced and capacity adjusts automatically.

  2. Front compute with the right load balancer: Place compute behind Elastic Load Balancing and choose the load balancer type that matches the traffic, so requests route only to healthy targets.

  3. Automate database healing: Configure Amazon RDS automatic failover to a standby instance, and build automation to promote an Amazon RDS read replica to primary where a standby is not available.

  4. Enable automatic recovery for single-location instances: For applications that cannot run in multiple locations but tolerate a reboot, enable automatic recovery on Amazon EC2 instances. Recovery preserves instance metadata, IP addresses, Amazon EBS volumes, and mount points to Amazon EFS or Amazon FSx.

  5. Size self-managed clusters for node loss: For clustered systems that do not use a load balancer, provision enough capacity to keep serving after losing at least one node, and trigger a workflow with AWS Systems Manager to recreate the lost node.

  6. Script remediation with event-driven automation: Use Amazon EventBridge to match failure events such as CloudWatch alarms and invoke AWS Lambda or AWS Step Functions to run custom remediation when built-in recovery does not apply.

Resources

Related best practices:

Related documents:

Related videos:

Related examples:

Related services: