max_queue_size - AWS SDKs and Tools

max_queue_size

This setting applies only to AWS CLI commands in the s3 namespace.

Specifies the maximum number of transfer tasks in the Amazon S3 task queue.

Details

The AWS CLI internally uses a model where it queues up Amazon S3 tasks that are then executed by consumers whose numbers are limited by max_concurrent_requests. A task generally maps to a single Amazon S3 operation. For example, a task could be a PutObjectTask, GetObjectTask, or UploadPartTask. The rate at which tasks are added to the queue can be much faster than the rate at which consumers finish the tasks. To avoid unbounded growth, the task queue size is capped to a specific size. This setting changes the value of that maximum number.

Default value: 1000

You generally don't need to change this setting. This setting also corresponds to the number of tasks that the AWS CLI is aware of that need to run. This means that by default the AWS CLI can only see 1000 tasks ahead. Increasing this value means that the CLI can more quickly know the total number of tasks needed, assuming that the queuing rate is quicker than the rate of task completion. The tradeoff is that a larger max_queue_size requires more memory.

Ways to set this value

Location Supported Example
config file Yes
s3 = max_queue_size = 2000
credentials file -
Environment variable -
AWS CLI parameter -

Compatibility with AWS SDKS and tools