View a markdown version of this page

SEC11-BP07 Deploy software programmatically - AWS Well-Architected Framework

SEC11-BP07 Deploy software programmatically

Manual deployments introduce human error and make it hard to prove that what you tested is what you released. Programmatic deployment runs the same steps every time, removes people from the production path, and gives you a way to verify the integrity of each release. Externalize configuration so one artifact runs in any environment, and sign packages so you can prove they did not change.

Desired outcome:

  • You deploy the exact version you tested, and deployment runs the same way every time.

  • You externalize configuration, so the same artifact deploys to different environments without change.

  • You cryptographically sign your packages and verify the signatures during deployment.

  • You remove persistent human access to production environments in favor of automated pipelines.

Common anti-patterns:

  • Teams deploy software to production manually.

  • Teams modify software manually to suit different environments.

Benefits of establishing this best practice:

  • Confidence in the release process increases because deployments are consistent and repeatable.

  • The risk of a failed change affecting business functionality drops.

  • Release cadence increases because each change carries less risk.

  • Rollback becomes automatic when a deployment encounters an unexpected event.

  • You can cryptographically prove that the software tested is the software deployed.

Level of risk exposed if this best practice is not established: High

Implementation guidance

Manual deployments fail through forgotten flags, steps that were run in the wrong order, or config values that were edited on the production host. Such mistakes are hard to reproduce and hard to audit, and they explain why a change that worked in testing fails in production. Encode the deployment once and run it identically every time.

Build one artifact and move that exact artifact through your environments. If you rebuild or hand-modify for each environment, production runs a different version of the artifact than the one you tested. Keep environment-specific settings out of the artifact and read them at deploy or run time from a store such as AWS Systems Manager Parameter Store. Sign the artifact after testing with AWS Signer or AWS Key Management Service (AWS KMS), and verify the signature before deployment. Signing algorithms will change as post-quantum algorithms replace traditional ones, and not every environment supports them. Choose signing mechanisms you can move to a new algorithm without re-architecting, and prefer managed ones. The more managed the mechanism, the more of the transition sits on AWS' side of the shared responsibility model.

Remove persistent human access from production so every change reaches it through the pipeline, where it is tested, reviewed, and recorded. Separate environments into distinct accounts so a mistake in development cannot reach production (for more detail, see Organizing your AWS environment using multiple accounts). Run build, test, and deployment through continuous integration and delivery tools such as AWS CodeBuild and AWS CodePipeline.

Apply the same repeatability to infrastructure. Define infrastructure with AWS CloudFormation or the AWS Cloud Development Kit (AWS CDK) (AWS CDK) so environments are created the same way and drift is visible. Deploy progressively to limit the scope of impact. A canary releases the change to a small subset of instances or users first. Watch its effect and roll back before a bad change reaches your entire user base. AWS CodeDeploy supports canaries for Amazon EC2 and AWS Lambda. Amazon ECS provides blue/green service deployments, with bake time, hooks for automated validation, and rollback to the previous revision. AWS CodePipeline can also roll back a stage automatically when it fails.

Implementation steps

  1. Separate environments into accounts: Follow Organizing your AWS environment using multiple accounts to place development, staging, and production in distinct accounts. Configure access controls that remove direct human access to production.

  2. Build a CI/CD pipeline: Use AWS CodeBuild and AWS CodePipeline to build, test, and deploy automatically. Connect your code repositories to the pipeline for version control.

  3. Sign and verify packages: Sign artifacts after testing with AWS Signer or AWS KMS, and verify the signature before deploying so only validated code reaches each environment. For container workloads, verify the signature again when the cluster admits the image.

  4. Externalize configuration: Store environment-specific settings in AWS Systems Manager Parameter Store, and read them at deploy or run time so one artifact serves every environment.

  5. Define infrastructure as code: Provision resources with AWS CloudFormation or the AWS CDK, and deploy infrastructure changes through the same pipeline as application changes.

  6. Deploy progressively with rollback: Release changes to a subset first, monitor the effect, and roll back automatically when issues appear. Use AWS CodeDeploy for Amazon EC2 and AWS Lambda, and Amazon ECS blue/green service deployments for containers. Stage rollback in AWS CodePipeline reverses a failed stage.

  7. Monitor and audit deployments: Track deployments, performance, and infrastructure changes with Amazon CloudWatch and AWS CloudTrail, and use the records to confirm adherence to your security and compliance requirements.

Resources

Related best practices:

Related documents:

Related videos:

Related services: