Design considerations - Distributed Load Testing on AWS

Design considerations

Supported applications

This solution supports cloud-based applications, and on-premises applications as long as you have a network connection from your AWS account to your application. The solution supports APIs that use either HTTP or HTTPS. You also have control over the HTTP request headers, so you can add authorization or custom headers to pass tokens or API keys.

JMeter script support

When creating a test scenario using this solution’s user interface (UI), you can use a JMeter test script. After selecting the JMeter script file, it is uploaded to the <stack-name>-scenariosbucket Amazon Simple Storage Service (Amazon S3) bucket. When Amazon Elastic Container Service (Amazon ECS) tasks are running, the JMeter script downloads from the <stack-name>-scenariosbucket Amazon S3 bucket and the test runs.

If you have JMeter input files, you can zip the input files together with the JMeter script. You can choose the zip file when you create a test scenario.

Note

If you include JMeter input files with your JMeter script file, you must include the relative path of the input files in your JMeter script file. In addition, the input files must be at the relative path. For example, when your JMeter input files and script file are in the /home/user directory and you refer to the input files in the JMeter script file, the path of input files must be ./INPUT_FILES. If you use /home/user/INPUT_FILES instead, the test will fail because it will not be able to find the input files.

For more information about how to use JMeter scripts, refer to JMeter User's Manual.

Scheduling tests

You can schedule tests to run at a future date or use the Run Now option. You can schedule a test as a one-time run in the future or set up a recurring test in which you specify a first run date, and planned recurrence. The options for recurrence include: daily, weekly, bi-weekly, and monthly. For more information on how scheduling works, refer to the Test scheduling workflow section of this guide.

Load testing limits

The maximum number of tasks that can be running in Amazon ECS using the AWS Fargate launch type is 1,000 per AWS Region, per account. Not all accounts support this limit by default, check the specific service quota for your account. For more information, refer to Amazon ECS Service Limits. For instructions on how to request an increase, refer to AWS Service Limits in the AWS General Reference Guide.

The Taurus load testing container image does not limit concurrent connections per task, but that does not mean that it can support an unlimited number of users. To determine the number of concurrent users the containers can generate for a test, refer to Determine the number of users section of this guide.

Note

The recommended limit for concurrent users based on default settings is 200 users.

Concurrent tests

This solution includes an Amazon CloudWatch dashboard for each test and displays the combined output of all tasks running for that test in the Amazon ECS cluster in real-time. The CloudWatch dashboard displays the average response time, the number of concurrent users, the number of successful requests, and the number of failed requests. Each metric is aggregated by the second, and the dashboard is updated every minute.

Amazon EC2 testing policy

You do not need approval from AWS to run load tests using this solution as long as your network traffic stays below 1 Gbps. If your test will generate more than 1 Gbps, contact AWS. For more information, refer to the Amazon EC2 Testing Policy.

User management

During initial configuration, you provide a username and email address that Amazon Cognito uses to grant you access to the solution’s web console. The console does not provide user administration. To add additional users, you must use the Amazon Cognito console. For more information, refer to Managing Users in User Pools in the Amazon Cognito Developer Guide.

Regional deployment

This solution uses Amazon Cognito which is available in specific AWS Regions only. Therefore, you must deploy this solution in a region where Amazon Cognito is available. For the most current service availability by Region, refer to the AWS Regional Services List.

Solution updates

You can update the AWS CloudFormation stack from version 2.0.0 to version 2.0.1. If you choose to update the stack, be sure to clear the CloudFront and browser cache to avoid image issues.

You cannot update version 1.3.0 or earlier versions to version 2.0.0 using the AWS CloudFormation console because of changes to resources deployments. To use version 2.0.0, you must create a new stack using version 2.0.0 of the AWS CloudFormation template.

We recommend deploying version 2.0.0 and transitioning your testing over to this latest version, instead of migrating the data from earlier versions. Once you transition your tests, you can uninstall the earlier version of this solution.