Tests for CI/CD pipelines

The two types of automated tests that are commonly referred to in deployment pipelines are unit tests and integration tests. However, there are many types of tests that you can run on a code base and the development environment. The AWS Deployment Pipeline Reference Architecture defines the following types of tests:

• Unit test – These tests build and run application code to verify that it is performing according to expectations. They simulate all external dependencies that are used in the code base. Examples of unit test tools include JUnit, Jest, and pytest.

• Integration test – These tests verify that the application satisfies technical requirements by testing against a provisioned test environment. Examples of integration test tools include Cucumber, vRest NG, and integ-tests (for AWS CDK).

• Acceptance test – These tests verify that the application satisfies user requirements by testing against a provisioned test environment. Examples of acceptance test tools include Cypress and Selenium.

• Synthetic test – These tests run continuously in the background to generate traffic and verify that the system is healthy. Examples of synthetic test tools include Amazon CloudWatch Synthetics and Dynatrace Synthetic Monitoring.

• Performance test – These tests simulate production capacity. They determine if the application meets performance requirements and compare metrics to past performance. Examples of performance test tools include Apache JMeter, Locust, and Gatling.

• Resilience test – Also known as chaos testing, these tests inject failures into environments in order to identify risk areas. Periods when the failures are injected are then compared to periods without the failures. Examples of resilience test tools include AWS Fault Injection Service and Gremlin.

• Static application security test (SAST) – These tests analyze code for security violations, such as SQL injection or cross-site scripting (XSS). Examples of SAST tools include Amazon CodeGuru, SonarQube, and Checkmarx.

• Dynamic application security test (DAST) – These tests are also known as penetration testing or pen testing. They identify vulnerabilities, such as SQL injection or XSS in a provisioned test environment. Examples of DAST tools include Zed Attack Proxy (ZAP) and HCL AppScan. For more information, see Penetration Testing.

Not all fully CI/CD pipelines run all of these tests. However, at a minimum, a pipeline should run unit tests and SAST tests on the code base as well as integration and acceptance tests on a test environment.