Considerations - Distributed Load Testing on AWS

Considerations

Supported applications

Distributed Load Testing on AWS supports cloud-based applications, and on-prem applications as long as you have a network connection from your AWS account to your application. The solution supports APIs that use either HTTP or HTTPS. You also have control over the HTTP request headers, so you can add authorization or custom headers to pass tokens or API keys.

JMeter script support

When creating a test scenario using this solution’s user interface (UI), you can use a JMeter test script. After selecting the JMeter script file, it is uploaded to the <stack-name>-scenariosbucket Amazon Simple Storage Service (Amazon S3) bucket. When Amazon Elastic Container Service (Amazon ECS) tasks are running, the JMeter script downloads from the <stack-name>-scenariosbucket Amazon S3 bucket and the test runs.

If you have JMeter input files, you can zip the input files together with the JMeter script. You can choose the zip file when you create a test scenario.

Note

If you include JMeter input files with your JMeter script file, you must include the relative path of the input files in your JMeter script file. In addition, the input files must be at the relative path. For example, when your JMeter input files and script file are in the /home/user directory and you refer to the input files in the JMeter script file, the path of input files must be ./INPUT_FILES. If you use /home/user/INPUT_FILES instead, the test will fail because it will not be able to find the input files.

Scheduling tests

You can schedule tests to run at a future date or use the Run Now option. You can schedule a test as a one-time run in the future or set up a recurring test in which you specify a first run date, and planned recurrence. The options for recurrence include: daily, weekly, bi-weekly, and monthly. For more information on how scheduling works, refer to the Test scheduling workflow section of this guide.

Load Testing Limits

The maximum number of tasks that can be running in Amazon ECS using the AWS Fargate launch type is 1,000 per AWS Region, per account. Not all accounts support this limit by default, check the specific service quota for your account. For more information, refer to Amazon ECS Service Limits. For instructions on how to request an increase, refer to AWS Service Limits in the AWS General Reference Guide.

The Taurus load testing container image does not limit concurrent connections per task, but that does not mean that it can support an unlimited number of users. To determine the number of concurrent users the containers can generate for a test, refer to Determine the number of users section of this guide.

Note

The recommended limit for concurrent users based on default settings is 200 users.

Concurrent Tests

This solution includes an Amazon CloudWatch dashboard for each test and displays the combined output of all tasks running for that test in the Amazon ECS cluster in real-time. The CloudWatch dashboard displays the average response time, the number of concurrent users, the number of successful requests, and the number of failed requests. Each metric is aggregated by the second, and the dashboard is updated every minute.

Docker Hub rate limits

For anonymous usage, Docker Hub limits the number of container image requests allowed to 100 every six hours. If the limit is reached, this solution cannot download the Taurus image or build the container image. The limit for Docker Hub Pro accounts is 200 container image requests every six hours. For Docker Team accounts, the limit is 50,000 requests every 24 hours. For more information, refer to Understanding Docker Hub Rate Limiting in the Docker documentation. If you have a Docker Hub account, you can provide your credentials when you launch this solution. First, create an AWS Secrets Manager secret with your credentials and provide the following key value pairs:

Key Value description
username The user name for your Docker Hub account.
password The password for your Docker Hub account.

Once the secret is created, you can pass the name of your secret to the DockerHubSecret parameter in the CloudFormation template. For more information about creating a Secrets Manager secret, refer to Tutorial: Creating and retrieving a secret in the AWS Secrets Manager User Guide.

Amazon EC2 testing policy

You do not need approval from AWS to run load tests using this solution as long as your network traffic stays below 1 Gbps. If your test will generate more than 1 Gbps, contact AWS. For more information, refer to the Amazon EC2 Testing Policy.

User management

During initial configuration, you provide a username and email address that Amazon Cognito uses to grant you access to the solution’s web console. The console does not provide user administration. To add additional users, you must use the Amazon Cognito console. For more information, refer to Managing Users in User Pools in the Amazon Cognito Developer Guide.

Regional Deployment

This solution uses Amazon Cognito which is available in specific AWS Regions only. Therefore, you must deploy this solution in a region where Amazon Cognito is available. For the most current service availability by Region, refer to the AWS Regional Services List.

Solution updates

If you have previously deployed the solution, you must update the solution's CloudFormation stack to get the latest version of the solution's framework. For details, refer to Update the stack.