REL05-BP05 Set client timeouts
Set timeouts appropriately on connections and requests, verify them systematically, and do not rely on default values as they are not aware of workload specifics.
Desired outcome: Client timeouts should consider the cost to the client, server, and workload associated with waiting for requests that take abnormal amounts of time to complete. Since it is not possible to know the exact cause of any timeout, clients must use knowledge of services to develop expectations of probable causes and appropriate timeouts
Client connections time out based on configured values. After encountering a timeout, clients make decisions to back off and retry or open a circuit breaker
Common anti-patterns:
-
Not being aware of system timeouts or default timeouts.
-
Not being aware of normal request completion timing.
-
Not being aware of possible causes for requests to take abnormally long to complete, or the costs to client, service, or workload performance associated with waiting on these completions.
-
Not being aware of the probability of impaired network causing a request to fail only once timeout is reached, and the costs to client and workload performance for not adopting a shorter timeout.
-
Not testing timeout scenarios both for connections and requests.
-
Setting timeouts too high, which can result in long wait times and increase resource utilization.
-
Setting timeouts too low, resulting in artificial failures.
-
Overlooking patterns to deal with timeout errors for remote calls like circuit breakers and retries.
-
Not considering monitoring for service call error rates, service level objectives for latency, and latency outliers. These metrics can provide insight to aggressive or permissive timeouts
Benefits of establishing this best practice: Remote call timeouts are configured and systems are designed to handle timeouts gracefully so that resources are conserved when remote calls respond abnormally slow and timeout errors are handled gracefully by service clients.
Level of risk exposed if this best practice is not established: High
Implementation guidance
Set both a connection timeout and a request timeout on any service dependency call and generally on any call across processes. Many frameworks offer built-in timeout capabilities, but be careful, as some have default values that are infinite or higher than acceptable for your service goals. A value that is too high reduces the usefulness of the timeout because resources continue to be consumed while the client waits for the timeout to occur. A value that is too low can generate increased traffic on the backend and increased latency because too many requests are retried. In some cases, this can lead to complete outages because all requests are being retried.
Consider the following when determining timeout strategies:
-
Requests may take longer than normal to process because of their content, impairments in a target service, or a networking partition failure.
-
Requests with abnormally expensive content could consume unnecessary server and client resources. In this case, timing out these requests and not retrying can preserve resources. Services should also protect themselves from abnormally expensive content with throttles and server-side timeouts.
-
Requests that take abnormally long due to a service impairment can be timed out and retried. Consideration should be given to service costs for the request and retry, but if the cause is a localized impairment, a retry is not likely to be expensive and will reduce client resource consumption. The timeout may also release server resources depending on the nature of the impairment.
-
Requests that take a long time to complete because the request or response has failed to be delivered by the network can be timed out and retried. Because the request or response was not delivered, failure would have been the outcome regardless of the length of timeout. Timing out in this case will not release server resources, but it will release client resources and improve workload performance.
Take advantage of well-established design patterns like retries and circuit breakers to handle timeouts gracefully and support fail-fast approaches. AWS SDKs and AWS CLI
Implementation steps
-
Configure timeouts on remote service calls and take advantage of built-in language timeout features or open source timeout libraries.
-
When your workload makes calls with an AWS SDK, review the documentation for language specific timeout configuration.
-
When using AWS SDKs or AWS CLI commands in your workload, configure default timeout values by setting the AWS configuration defaults for
connectTimeoutInMillisandtlsNegotiationTimeoutInMillis. -
Apply command line options
cli-connect-timeoutandcli-read-timeoutto control one-off AWS CLI commands to AWS services. -
Monitor remote service calls for timeouts, and set alarms on persistent errors so that you can proactively handle error scenarios.
-
Implement CloudWatch Metrics and CloudWatch anomaly detection on call error rates, service level objectives for latency, and latency outliers to provide insight into managing overly aggressive or permissive timeouts.
-
Configure timeouts on Lambda functions.
-
API Gateway clients must implement their own retries when handling timeouts. API Gateway supports a 50 millisecond to 29 second integration timeout for downstream integrations and does not retry when integration requests timeout.
-
Implement the circuit breaker
pattern to avoid making remote calls when they are timing out. Open the circuit to avoid failing calls and close the circuit when calls are responding normally. -
For container based workloads, review App Mesh Envoy features to leverage built in timeouts and circuit breakers.
-
Use AWS Step Functions to build low code circuit breakers for remote service calls, especially where calling AWS native SDKs and supported Step Functions integrations to simplify your workload.
Resources
Related best practices:
Related documents:
Related examples:
Related tools:
