Test workflow - Distributed Load Testing on AWS

Test workflow

The following detailed breakdown shows the steps involved in running a test scenario.


      Test workflow

Figure 3: Test workflow

  1. You use the web console to submit a test scenario that includes the configuration details to the solution’s API.

  2. The test scenario configuration is uploaded to the Amazon Simple Storage Service (Amazon S3) as a JSON file (s3://<bucket-name>/test-scenarios/<$TEST_ID>/<$TEST_ID>.json).

  3. An AWS Step Functions state machine runs using the test ID, task count, test type, and file type as the AWS Step Functions state machine input. If the test is scheduled, it will first create a CloudWatch Events rule, which triggers AWS Step Functions on the specified date. For more details on the scheduling workflow, refer to the Test scheduling workflow section of this guide.

  4. Configuration details are stored in the scenarios Amazon DynamoDB table.

  5. In the AWS Step Functions task runner workflow, the ecr-checker AWS Lambda function checks whether the ContainerCodeBuild AWS CodeBuild project has succeeded. If the CodeBuild project remains in progress, the AWS Lambda function continues checking in one minute intervals. When the CodeBuild project is completed, the AWS Lambda function checks whether the latest tag is applied to the newest image in the Amazon Elastic Container Registry (Amazon ECR) image repository. If this tag has not been applied, the AWS Lambda function creates an error message.

  6. The task-status-checker AWS Lambda function checks if Amazon Elastic Container Service (Amazon ECS) tasks are already running for the same test ID. If tasks with the same test ID are found running, it causes an error. If there are no Amazon ECS tasks running in the AWS Fargate cluster, the function returns the test ID, task count, and test type.

  7. The task-runner AWS Lambda function gets the task details from the previous step and runs the Amazon ECS worker tasks in the AWS Fargate cluster. The Amazon ECS API uses the RunTask action to run the worker tasks. These worker tasks are launched and then wait for a start message from the leader task in order to begin the test. The RunTask action is limited to 10 tasks per definition. If your task count is more than 10, the task definition will run multiple times until all worker tasks have been started. The function also generates a prefix to distinguish the current test in the results-parser AWS Lambda function.

  8. The task-status-checker AWS Lambda function checks if all the Amazon ECS worker tasks are running with the same test ID. If tasks are still provisioning, it waits for one minute and checks again. Once all Amazon ECS tasks are running, it returns the test ID, task count, test type, all task IDs and prefix and passes it to the task-runner function.

  9. The task-runner AWS Lambda function runs again, this time launching a single Amazon ECS task to act as the leader node. This ECS task sends a start test message to each of the worker tasks in order to start the tests simultaneously.

  10. The task-status-checker AWS Lambda function again checks if Amazon ECS tasks are running with the same test ID. If tasks are still running, it waits for one minute and checks again. Once there are no running Amazon ECS tasks, it returns the test ID, task count, test type, and prefix.

  11. When the task-runner AWS Lambda function runs the Amazon ECS tasks in the AWS Fargate cluster, each task downloads the test configuration from Amazon S3 and starts the test.

  12. Once the tests are running the average response time, number of concurrent users, number of successful requests, and number of failed requests for each task is logged in Amazon CloudWatch and can be viewed in a CloudWatch dashboard.

  13. When the test is complete, the container images export a detail report as an XML file to Amazon S3. Each file is given a UUID for the filename. For example, s3://dlte-bucket/test-scenarios/<$TEST_ID>/results/<$UUID>.json.

  14. When the XML files are uploaded to Amazon S3, the results-parser AWS Lambda function reads the results in the XML files starting with the prefix and parses and aggregates all the results into one summarized result.

  15. The results-parser AWS Lambda function writes the aggregate result to an Amazon DynamoDB table.