Design interactions in a distributed system to mitigate or withstand failures

Distributed systems rely on communications networks to interconnect components (such as servers or services). Your workload must operate reliably despite data loss or latency over these networks. Components of the distributed system must operate in a way that does not negatively impact other components or the workload. These best practices allow workloads to withstand stresses or failures, more quickly recover from them, and mitigate the impact of such impairments. The result is improved mean time to recovery (MTTR).

These best practices prevent failures and improve mean time between failures (MTBF).

Best practices

• REL05-BP01 Implement graceful degradation to transform applicable hard dependencies into soft dependencies
• REL05-BP02 Throttle requests
• REL05-BP03 Control and limit retry calls
• REL05-BP04 Fail fast and limit queues
• REL05-BP05 Set client timeouts
• REL05-BP06 Make services stateless where possible
• REL05-BP07 Implement emergency levers