Troubleshooting latency issues in Amazon DynamoDB - Amazon DynamoDB

Troubleshooting latency issues in Amazon DynamoDB

If your workload appears to experience high latency, you can analyze the CloudWatch SuccessfulRequestLatency metric, and check the average latency to see if it’s related to DynamoDB. Some variability in the reported SuccessfulRequestLatency is normal, and occasional spikes (particularly in the Maximum statistic) should not be cause for concern. However, if the Average statistic shows a sharp increase and persists, you should check the AWS Service Health Dashboard and your Personal Health Dashboard for more information. Some possible causes include the size of the item in your table (a 1kb item and a 400kb item will vary in latency) or the size of the query (10 items versus 100 items).

If necessary, consider opening a support case with AWS Support, and continue to assess any available fall-back options for your application (such as evacuation of a Region if you have a multi-Region architecture) according to your runbooks. You should log request IDs for slow requests for providing these IDs to AWS Support when you open a support case.

The SuccessfulRequestLatency metric only measures latency which is internal to the DynamoDB service - client side activity and network trip times are not included. To learn more about overall latency for calls from your client to the DynamoDB service, you can enable latency metric logging in your AWS SDK.

Note

For most singleton operations (operations which apply to a single item by fully specifying the primary key's value), DynamoDB delivers single-digit millisecond Average SuccessfulRequestLatency. This value does not include the transport overhead for the caller code accessing the DynamoDB endpoint. For multi-item data operations, the latency will vary based on factors such as size of the result set, the complexity of the data structures returned, and any condition expressions and filter expressions applied. For repeated multi-item operations to the same data set with the same parameters, DynamoDB will provide highly consistent Average SuccessfulRequestLatency.

Consider one or more of the following strategies to reduce latency:

  • Adjust request timeout and retry behavior: The path from your client to DynamoDB traverses many components, each of which is designed with redundancy in mind. Think about the scope of network resiliency, TCP packet timeouts, and the distributed architecture of DynamoDB itself. The default SDK behaviors are designed to find the right balance for most applications. If the best possible latency is your highest priority, you should consider adjusting the default request timeout and retry settings for your SDK to closely track to the typical latency for a successful request measured by your client. A request which is taking significantly longer than normal is less likely to ultimately succeed - if you fail fast and make a new request, this is likely to take a different path and may quickly succeed. Keep in mind that there can be downsides to being too aggressive in these settings. A helpful discussion on this topic can be found in Tuning AWS Java SDK HTTP request settings for latency-aware Amazon DynamoDB applications.

  • Reduce the distance between the client and DynamoDB endpoint: If you have globally dispersed users, consider using Global tables - multi-Region replication for DynamoDB. With global tables, you can specify the AWS Regions where you want the table to be available. Reading data from a local global tables replica can significantly reduce latency for your users. Also, consider using a DynamoDB gateway endpoint to keep your client traffic within your VPC.

  • Use caching: If your traffic is read heavy, consider using a caching service, such as In-memory acceleration with DynamoDB Accelerator (DAX). DAX is a fully managed, highly available, in-memory cache for DynamoDB that delivers up to a 10x performance improvement, from milliseconds to microseconds, even at millions of requests per second.

  • Reuse connections: DynamoDB requests are made via an authenticated session which defaults to HTTPS. Initiating the connection takes time so the latency of the first request is higher than typical. Requests over an already initialized connection deliver DynamoDB's consistent low latency. For this reason, you may wish to make a "keep-alive" GetItem request every 30 seconds if no other requests are made, to avoid the latency of establishing a new connection.

  • Use eventually consistent reads: If your application doesn't require strongly consistent reads, consider using the default eventually consistent reads. Eventually consistent reads are lower cost and are also less likely to experience transient increases in latency. For more information, see DynamoDB read consistency.