View a markdown version of this page

REL08-BP05 Deploy changes with automation - AWS Well-Architected Framework

REL08-BP05 Deploy changes with automation

Changes to production are one of the largest sources of operational risk. Automate deployments and patching to remove manual steps where human error exists so that every change follows the same tested path.

Desired outcome:

  • Your release process runs through automation with pre-production testing, automatic rollback, and staggered production deployments.

  • Developers no longer watch deployments manually, because the pipeline catches and reverses failures.

  • You lint and test your infrastructure as code to catch configuration failures before they reach production.

  • When a production failure occurs, you add a test that catches the same problem in future releases.

Common anti-patterns:

  • Teams make manual changes to production.

  • Teams skip automated steps through manual emergency workflows.

  • Teams abandon established plans and processes to meet accelerated timelines.

  • Teams run rapid follow-on deployments without allowing bake time.

  • Teams do not lint their infrastructure as code, so known failure patterns reach production.

  • Teams do not update their tests after a failure occurs.

Benefits of establishing this best practice:

  • Human error drops because automation replaces manual deployment steps.

  • Problems are caught before production, because every change is tested on the way there.

  • Automatic rollback returns the workload to its last working state when a production issue appears.

  • Infrastructure misconfigurations are caught in the pipeline through linting rather than in production.

Level of risk exposed if this best practice is not established: Medium

Implementation guidance

Automated deployment builds on a continuous integration and continuous delivery (CI/CD) pipeline. A commit passes through automated stage gates for build, test, and deployment, and each gate can halt the pipeline or trigger a rollback. The pipeline runs tests, detects anomalies, and reverses changes automatically.

Automate your hardest procedures, because they are the most likely to fail when done by humans under pressure. A rarely-used procedure run manually during an incident is a common source of new failures. Encode the procedure once, test it, and run it automatically to remove that risk.

Apply the same quality controls to IaC as to application code. Lint your infrastructure templates to catch common misconfigurations, such as insecure settings or missing guardrails, before deployment. New failure patterns emerge over time, so update your linting rules as you discover them and confirm each is addressed before deployment. Enforce reliability thresholds, such as a minimum instance count for availability, early in the pipeline.

Bake time is how long you wait after a deployment before marking it healthy and continuing. Without bake time, rapid follow-on deployments stack unverified changes, so when something breaks you cannot tell which change caused it. Configuration parameters have the same risk. Templates use variables that differ between environments, and passing those variables on the command line invites human error. Store configuration in a repository, deploy it through a pipeline, and have deployments read only approved values.

Implementation steps

  1. Store code and configuration in a repository: Use a hosted source control system based on a widely adopted technology such as Git to store application source and infrastructure as code (IaC) configuration securely.

  2. Configure a continuous integration service: Compile source, run tests, and produce deployment artifacts automatically. To set up a build project, see Getting started with AWS CodeBuild using the console.

  3. Configure an automated deployment service: Use AWS CodeDeploy to deploy to compute services such as Amazon EC2, AWS Fargate, and AWS Lambda, as well as on-premises servers, without relying on error-prone manual steps.

  4. Automate the release pipeline: Use AWS CodePipeline to orchestrate build, test, and deployment stages for faster, more reliable updates. For examples, see the CodePipeline tutorials.

  5. Lint your infrastructure as code: Scan templates for common failure scenarios with a tool such as cfn_nag, and update the rules as new misconfigurations are discovered. Enforce availability and performance thresholds early in the pipeline.

  6. Manage environmental configuration through change control: Store template variables in a repository and deploy them through a pipeline rather than passing them on the command line. Use a service such as AWS AppConfig so deployments read only approved settings, and monitor changes to these parameters.

Resources

Related best practices:

Related documents:

Related videos:

Related tools: