Set up a CI/CD pipeline for hybrid workloads on Amazon ECS Anywhere by using AWS CDK and GitLab

Created by Dr. Rahul Sharad Gaikwad (AWS)

Code repository: amazon-ecs-anywhere-cicd-pipeline-cdk-sample	Environment: PoC or pilot	Technologies: Hybrid cloud; Containers & microservices; Infrastructure; DevOps
Workload: Open-source	AWS services: AWS CDK; AWS CodePipeline; Amazon ECS; AWS Systems Manager; AWS CodeCommit

Summary

Amazon ECS Anywhere is an extension of the Amazon Elastic Container Service (Amazon ECS). It provides support for registering an external instance, such as an on-premises server or virtual machine (VM), to your Amazon ECS cluster. is feature helps reduce costs and mitigate complex local container orchestration and operations. You can use ECS Anywhere to deploy and run container applications in both on-premises and cloud environments. It removes the need for your team to learn multiple domains and skill sets, or to manage complex software on their own.

This pattern describes a step-by-step approach to provision an Amazon ECS cluster with Amazon ECS Anywhere instances by using Amazon Web Services (AWS) Cloud Development Kit (AWS CDK) stacks. You then use AWS CodePipeline to set up a continuous integration and continuous deployment (CI/CD) pipeline. Then, you replicate your GitLab code repository to AWS CodeCommit and deploy your containerized application on the Amazon ECS cluster.

This pattern is designed to help those who use on-premises infrastructure to run container applications and use GitLab to manage the application code base. You can manage those workloads by using AWS Cloud services, without disturbing your existing, on-premises infrastructure.

Prerequisites and limitations

Prerequisites

• An active AWS account.

• A container application running on on-premises infrastructure.

• A GitLab repository where you manage your application code base. For more information, see Repository (GitLab).

• AWS Command Line Interface (AWS CLI), installed and configured. For more information, see Installing or updating the latest version of the AWS CLI (AWS CLI documentation).

• AWS CDK Toolkit, installed and configured globally. For more information, see Install the AWS CDK (AWS CDK documentation).

• npm, installed and configured for the AWS CDK in TypeScript. For more information, see Downloading and installing Node.js and npm (npm documentation).

Limitations

• For limitations and considerations, see External instances (Amazon ECS Anywhere) in the Amazon ECS documentation.

Product versions

• AWS CDK Toolkit version 2.27.0 or later

• npm version 7.20.3 or later

• Node.js version 16.6.1 or later

Architecture

Target technology stack

• AWS CDK

• AWS CloudFormation

• AWS CodeBuild

• AWS CodeCommit

• AWS CodePipeline

• Amazon ECS Anywhere

• Amazon Elastic Container Registry (Amazon ECR)

• AWS Identity and Access Management (IAM)

• AWS System Manager

• GitLab repository

Target architecture

This diagram represents two primary workflows described in this pattern, provisioning the Amazon ECS cluster and setting up the CI/CD pipeline that sets up and deploys the CI/CD pipeline, as follows:

1. Provisioning the Amazon ECS cluster

   1. When you deploy the first AWS CDK stack, it creates a CloudFormation stack on AWS.

   2. This CloudFormation stack provisions an Amazon ECS cluster and related AWS resources.

   3. To register an external instance with an Amazon ECS cluster, you must install AWS Systems Manager Agent (SSM Agent) on your VM and register the VM as an AWS Systems Manager managed instance. 

   4. You must also install the Amazon ECS container agent and Docker on your VM to register it as an external instance with the Amazon ECS cluster.

   5. When the external instance is registered and configured with the Amazon ECS cluster, it can run multiple containers on your VM, which is registered as an external instance.

   6. The Amazon ECS cluster is active and can run the application workloads through containers. The Amazon ECS Anywhere container instance runs in on-premises environment but is associated with the Amazon ECS cluster in the cloud.

2. Setting up and deploying the CI/CD pipeline

   1. When you deploy the second AWS CDK stack, it creates another CloudFormation stack on AWS.

   2. This CloudFormation stack provisions a pipeline in CodePipeline and related AWS resources.

   3. You push and merge application code changes to an on-premises GitLab repository. 

   4. The GitLab repository is automatically replicated to the CodeCommit repository.

   5. The updates to the CodeCommit repo automatically starts CodePipeline. 

   6. CodePipeline copies code from CodeCommit and creates the deployable application build in CodeBuild.

   7. CodePipeline creates a Docker image of the CodeBuild build environment and pushes it to the Amazon ECR repo.

   8. CodePipeline initiates CodeDeploy actions that pull the container image from the Amazon ECR repo.

   9. CodePipeline deploys the container image on the Amazon ECS cluster.

Automation and scale

This pattern uses the AWS CDK as an infrastructure as code (IaC) tool to configure and deploy this architecture. AWS CDK helps you orchestrate the AWS resources and set up Amazon ECS Anywhere and the CI/CD pipeline.

Tools

AWS services

• AWS Cloud Development Kit (AWS CDK) is a software development framework that helps you define and provision AWS Cloud infrastructure in code.

• AWS CodeCommit is a version control service that helps you privately store and manage Git repositories, without needing to manage your own source control system.

• AWS CodePipeline helps you quickly model and configure the different stages of a software release and automate the steps required to release software changes continuously.

• AWS Command Line Interface (AWS CLI) is an open-source tool that helps you interact with AWS services through commands in your command-line shell.

• Amazon Elastic Container Registry (Amazon ECR) is a managed container image registry service that’s secure, scalable, and reliable.

• Amazon Elastic Container Service (Amazon ECS) is a fast and scalable container management service that helps you run, stop, and manage containers on a cluster. This pattern also uses Amazon ECS Anywhere, which provides support for registering an on-premises server or VM to your Amazon ECS cluster.

Other tools

• Node.js is an event-driven JavaScript runtime environment designed for building scalable network applications.

• npm is a software registry that runs in a Node.js environment and is used to share or borrow packages and manage deployment of private packages.

• Vagrant is an open-source utility for building and maintaining portable virtual software development environments. For demonstration purposes, this pattern uses Vagrant to create an on-premises VM.

Code repository

The code for this pattern is available in the GitHub CI/CD pipeline for Amazon ECS Anywhere using AWS CDK repository.

Best practices

Consider the following best practices when deploying this pattern:

• Best practices for developing and deploying cloud infrastructure with the AWS CDK

• Best practices for developing cloud applications with AWS CDK (AWS blog post)

Epics

Verify the AWS CDK configuration

Task	Description	Skills required
Verify the AWS CDK version.	Verify the version of the AWS CDK Toolkit by entering the following command. cdk --version This pattern requires version 2.27.0 or later. If you have an earlier version, follow the instructions in the AWS CDK documentation to update it.	DevOps engineer
Verify the npm version.	Verify the version of npm by entering the following command. npm --version This pattern requires version 7.20.3 or later. If you have an earlier version, follow the instructions in the npm documentation to update it.	DevOps engineer
Set up AWS credentials.	Set up AWS credentials by entering the aws configure command and following the prompts. $aws configure AWS Access Key ID [None]: <your-access-key-ID> AWS Secret Access Key [None]: <your-secret-access-key> Default region name [None]: <your-Region-name> Default output format [None]:	DevOps engineer

Bootstrap the AWS CDK environment

Task	Description	Skills required
Clone the AWS CDK code repository.	Clone the CI/CD pipeline for Amazon ECS Anywhere using AWS CDK repository for this pattern by entering the following command. git clone https://github.com/aws-samples/amazon-ecs-anywhere-cicd-pipeline-cdk-sample.git Navigate into the cloned directory by entering the following command. cd amazon-ecs-anywhere-cicd-pipeline-cdk-sample	DevOps engineer
Bootstrap the environment.	Deploy the CloudFormation template to the account and AWS Region that you want to use by entering the following command. cdk bootstrap <account-number>/<Region> For more information, see Bootstrapping in the AWS CDK documentation.	DevOps engineer

Build and deploy the infrastructure for Amazon ECS Anywhere

Task	Description	Skills required
Install the package dependencies and compile the TypeScript files.	Install the package dependencies and compile the TypeScript files by entering the following commands. $cd EcsAnywhereCdk $npm install $npm fund  These commands install all the packages from the sample repository. For more information, see npm ci and npm install in the npm documentation. If you get any errors about missing packages when you enter these commands, see the Troubleshooting section of this pattern.	DevOps engineer
Build the project.	To build the project code, enter the following command. npm run build For more information about building and deploying the project, see Your first AWS CDK app in the AWS CDK documentation.	DevOps engineer
Deploy the Amazon ECS Anywhere infrastructure stack.	List the stacks by entering the following command. $cdk list Confirm that the output returns the EcsAnywhereInfraStack and ECSAnywherePipelineStack stacks. Deploy the EcsAnywhereInfraStack stack by entering the following command. $cdk  deploy EcsAnywhereInfraStack	DevOps engineer
Verify stack creation and output.	Sign in to the AWS Management Console and open the CloudFormation console at https://console.aws.amazon.com/cloudformation/. On the Stacks page, select the EcsAnywhereInfraStack stack. Confirm that the stack status is either CREATE_IN_PROGRESS or CREATE_COMPLETE. Setting up the Amazon ECS cluster can take some time. Do not proceed until the stack creation is complete.	DevOps engineer

Set up an on-premises VM

Task	Description	Skills required
Set up your VM.	Create a Vagrant VM by entering the vagrant up command from the root directory where Vagrantfile is located. For more information, see the Vagrant documentation.	DevOps engineer
Register your VM as an external instance.	Log in to the Vagrant VM by using the vagrant ssh command. For more information, see the Vagrant documentation. Install AWS CLI on the VM by following AWS CLI installation instructions and entering the following commands.  $ curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" \ > -o "awscliv2.zip" $sudo apt install unzip $unzip awscliv2.zip $sudo ./aws/install $aws configure AWS Access Key ID [None]: <your-access-key-ID> AWS Secret Access Key [None]: <your-secret-access-key> Default region name [None]: <your-Region-name> Default output format [None]: Create an activation code and ID that you can use to register your VM with AWS Systems Manager and to activate your external instance. The output from this command includes the activation ID and activation code values. aws ssm create-activation \ > --iam-role EcsAnywhereInstanceRole \ > | tee ssm-activation.json If you receive an error when you run this command, see the Troubleshooting section. Export the activation ID and code values. export ACTIVATION_ID=<activation-ID> export ACTIVATION_CODE=<activation-code> Download the installation script to your VM. curl --proto "https" -o "ecs-anywhere-install.sh" \ > "https://amazon-ecs-agent.s3.amazonaws.com/ecs-anywhere-install-latest.sh" Run the installation script on your VM. sudo bash ecs-anywhere-install.sh \ --cluster EcsAnywhereCluster \ --activation-id $ACTIVATION_ID \ --activation-code $ACTIVATION_CODE \ --region <region-name> This sets up your VM was an Amazon ECS Anywhere external instance and registers the instance in the Amazon ECS cluster. For more information, see Registering an external instance to a cluster in the Amazon ECS documentation. If you experience any issues, see the Troubleshooting section.	DevOps engineer
Verify the status of Amazon ECS Anywhere and the external VM.	To verify whether your VM is connected to the Amazon ECS control plane and running, use the following commands. $aws ssm describe-instance-information $aws ecs list-container-instances --cluster $CLUSTER_NAME	DevOps engineer

Deploy the CI/CD pipeline

Task	Description	Skills required
Create a branch in the CodeCommit repo.	Create a branch named main in the CodeCommit repo by creating the first commit for the repository. You can follow AWS documentation to Create a commit in CodeCommit. The following command is an example. aws codecommit put-file \ --repository-name EcsAnywhereRepo \ --branch-name main \ --file-path README.md \ --file-content "Test" \ --name "Dev Ops" \ --email "[e-mail]" \ --commit-message "Adding README."	DevOps engineer
Set up repo mirroring.	You can mirror a GitLab repository to and from external sources. You can select which repository serves as the source. Branches, tags, and commits are synced automatically. Set up a push mirror between the GitLab repository that hosts your application and the CodeCommit repository. For instructions, see Set up a push mirror from GitLab to CodeCommit (GitLab documentation). Note: By default, mirroring automatically syncs the repository. If you want to manually update the repositories, see Update a mirror (GitLab documentation).	DevOps engineer
Deploy the CI/CD pipeline stack.	Deploy the EcsAnywherePipelineStack stack by entering the following command. $cdk  deploy EcsAnywherePipelineStack	DevOps engineer
Test the CI/CD pipeline.	Make application code changes and push it to the source, on-premises GitLab repo. For more information, see Push Options (GitLab documentation). For example, edit the ../application/index.html file to update the application version value. When the code is replicated to the CodeCommit repo, this starts the CI/CD pipeline. Do one of the following: If you are using automatic mirroring to synchronize the GitLab repo with the CodeCommit repo, continue to the next step. If you are using manual mirroring, push the application code changes to the CodeCommit repo by following the instructions in Update a mirror (GitLab documentation). On your local machine, in a web browser, enter http://localhost:80. This opens the NGINX web page because port 80 is forwarded to localhost in Vagrantfile. Confirm that you can view the updated application version value. This validates the pipeline and image deployment. (Optional) If you want to verify the deployment in the AWS Management Console, do the following: Open the Amazon ECS console at https://console.aws.amazon.com/ecs/. From the navigation bar, select the Region to use. In the navigation pane, choose Clusters. On the Clusters page, select the EcsAnywhereCluster cluster. Choose Task Definitions. Confirm the container is running.	DevOps engineer

Clean up

Task	Description	Skills required
Clean up and delete the resources.	After you walk through this pattern, you should remove the proof-of-concept resources you created. To clean up, enter the following commands. $cdk destroy EcsAnywherePipelineStack $cdk destroy EcsAnywhereInfraStack	DevOps engineer

Troubleshooting

Issue	Solution
Errors about missing packages when installing package dependencies.	Enter one of the following commands to resolve missing packages. $npm ci or $npm install -g @aws-cdk/<package_name>
When you run the aws ssm create-activation command on the VM, you receive the following error. An error occurred (ValidationException) when calling the CreateActivation operation: Nonexistent role or missing ssm service principal in trust policy: arn:aws:iam::000000000000:role/EcsAnywhereInstanceRole	The EcsAnywhereInfraStack stack isn’t fully deployed, and the IAM role necessary to run this command hasn’t been created yet. Check the stack status in the CloudFormation console. Retry the command after the status changes to CREATE_COMPLETE.
An Amazon ECS health check returns UNHEALTHY, and you see the following error in the Services section of the cluster in the Amazon ECS console. service EcsAnywhereService was unable to place a task because no container instance met all of its requirements. Reason: No Container Instances were found in your cluster.	Restart the Amazon ECS agent on your Vagrant VM by entering the following commands. $vagrant ssh $sudo systemctl restart ecs $sudo systemctl status ecs

Related resources

• Amazon ECS Anywhere marketing page

• Amazon ECS Anywhere documentation

• Amazon ECS Anywhere demo (video)

• Amazon ECS Anywhere workshop samples (GitHub)

• Repository mirroring (GitLab documentation)