REL05-BP02 Throttle requests
Throttle requests to mitigate resource exhaustion due to unexpected increases in demand. Requests below throttling rates are processed while those over the defined limit are rejected with a return message indicating the request was throttled.
Desired outcome: Large volume spikes either from sudden customer traffic increases, flooding attacks, or retry storms are mitigated by request throttling, allowing workloads to continue normal processing of supported request volume.
Common anti-patterns:
-
API endpoint throttles are not implemented or are left at default values without considering expected volumes.
-
API endpoints are not load tested or throttling limits are not tested.
-
Throttling request rates without considering request size or complexity.
-
Testing maximum request rates or maximum request size, but not testing both together.
-
Resources are not provisioned to the same limits established in testing.
-
Usage plans have not been configured or considered for application to application (A2A) API consumers.
-
Queue consumers that horizontally scale do not have maximum concurrency settings configured.
-
Rate limiting on a per IP address basis has not been implemented.
Benefits of establishing this best practice: Workloads that set throttle limits are able to operate normally and process accepted request load successfully under unexpected volume spikes. Sudden or sustained spikes of requests to APIs and queues are throttled and do not exhaust request processing resources. Rate limits throttle individual requestors so that high volumes of traffic from a single IP address or API consumer will not exhaust resources impact other consumers.
Level of risk exposed if this best practice is not established: High
Implementation guidance
Services should be designed to process a known capacity of requests; this capacity can be established through load testing. If request arrival rates exceed limits, the appropriate response signals that a request has been throttled. This allows the consumer to handle the error and retry later.
When your service requires a throttling implementation, consider implementing the token bucket algorithm, where a token counts for a request. Tokens are refilled at a throttle rate per second and emptied asynchronously by one token per request.
The token bucket algorithm.
Amazon API Gateway
Implementation steps
You can configure API Gateway with throttling limits for your APIs and return 429 Too Many Requests errors when limits are exceeded. You can use AWS WAF with your AWS AppSync and API Gateway endpoints to enable rate limiting on a per IP address basis. Additionally, where your system can tolerate asynchronous processing, you can put messages into a queue or stream to speed up responses to service clients, which allows you to burst to higher throttle rates.
With asynchronous processing, when you’ve configured Amazon SQS as an event source for AWS Lambda, you can configure maximum concurrency to avoid high event rates from consuming available account concurrent execution quota needed for other services in your workload or account.
While API Gateway provides a managed implementation of the token bucket, in cases where you cannot use API Gateway, you can take advantage of language specific open-source implementations (see related examples in Resources) of the token bucket for your services.
-
Understand and configure API Gateway throttling limits at the account level per region, API per stage, and API key per usage plan levels.
-
Apply AWS WAF rate limiting rules
to API Gateway and AWS AppSync endpoints to protect against floods and block malicious IPs. Rate limiting rules can also be configured on AWS AppSync API keys for A2A consumers. -
Consider whether you require more throttling control than rate limiting for AWS AppSync APIs, and if so, configure an API Gateway in front of your AWS AppSync endpoint.
-
When Amazon SQS queues are set up as triggers for Lambda queue consumers, set maximum concurrency to a value that processes enough to meet your service level objectives but does not consume concurrency limits impacting other Lambda functions. Consider setting reserved concurrency on other Lambda functions in the same account and region when you consume queues with Lambda.
-
Use API Gateway with native service integrations to Amazon SQS or Kinesis to buffer requests.
-
If you cannot use API Gateway, look at language specific libraries to implement the token bucket algorithm for your workload. Check the examples section and do your own research to find a suitable library.
-
Test limits that you plan to set, or that you plan to allow to be increased, and document the tested limits.
-
Do not increase limits beyond what you establish in testing. When increasing a limit, verify that provisioned resources are already equivalent to or greater than those in test scenarios before applying the increase.
Resources
Related best practices:
Related documents:
Related examples:
Related videos:
Related tools:
