OPS05-BP02 Test and validate changes - AWS Well-Architected Framework (2023-04-10)
Implementation guidanceResources

OPS05-BP02 Test and validate changes

Every change deployed must be tested to avoid errors in production. This best practice is focused on testing changes from version control to artifact build. Besides application code changes, testing should include infrastructure, configuration, security controls, and operations procedures. Testing takes many forms, from unit tests to software component analysis (SCA). Move tests further to the left in the software integration and delivery process results in higher certainty of artifact quality.

Your organization must develop testing standards for all software artifacts. Automated tests reduce toil and avoid manual test errors. Manual tests may be necessary in some cases. Developers must have access to automated test results to create feedback loops that improve software quality.

Desired outcome:

  • All software changes are tested before they are delivered.

  • Developers have access to test results.

  • Your organization has a testing standard that applies to all software changes.

Common anti-patterns:

  • You deploy a new software change without any tests. It fails to run in production, which leads to an outage.

  • New security groups are deployed with AWS CloudFormation without being tested in a pre-production environment. The security groups make your app unreachable for your customers.

  • A method is modified but there are no unit tests. The software fails when it is deployed to production.

Benefits of establishing this best practice:

  • The change fail rate of software deployments is reduced.

  • Software quality is improved.

  • Developers have increased awareness on the viability of their code.

  • Security policies can be rolled out with confidence to support organization's compliance

  • Infrastructure changes such as automatic scaling policy updates are tested in advance to meet traffic needs.

Level of risk exposed if this best practice is not established: High

Implementation guidance

Testing is done on all changes, from application code to infrastructure, as part of your continuous integration practice. Test results are published so that developers have fast feedback. Your organization has a testing standard that all changes must pass.

Customer example

As part of their continuous integration pipeline, AnyCompany Retail conducts several types of tests on all software artifacts. They practice test driven development so all software has unit tests. Once the artifact is built, they run end-to-end tests. After this first round of tests is complete, they run a static application security scan, which looks for known vulnerabilities. Developers receive messages as each testing gate is passed. Once all tests are complete, the software artifact is stored in an artifact repository.

Implementation steps

  1. Work with stakeholders in your organization to develop a testing standard for software artifacts. What standard tests should all artifacts pass? Are there compliance or governance requirements that must be included in the test coverage? Do you need to conduct code quality tests? When tests complete, who needs to know?

    1. The AWS Deployment Pipeline Reference Architecture contains an authoritative list of types of tests that can be conducted on software artifacts as part of an integration pipeline.

  2. Instrument your application with the necessary tests based on your software testing standard. Each set of tests should complete in under ten minutes. Tests should run as part of an integration pipeline.

    1. Amazon CodeGuru Reviewer can test your application code for defects.

    2. You can use AWS CodeBuild to conduct tests on software artifacts.

    3. AWS CodePipeline can orchestrate your software tests into a pipeline.

Resources

Related best practices:

Related documents:

Related videos:

Related resources:

Related services: