Set up a CI/CD pipeline for hybrid workloads on Amazon ECS Anywhere by using AWS CDK and GitLab - AWS Prescriptive Guidance
SummaryPrerequisites and limitationsArchitectureToolsBest practicesEpicsTroubleshootingRelated resources

Set up a CI/CD pipeline for hybrid workloads on Amazon ECS Anywhere by using AWS CDK and GitLab

Created by Dr. Rahul Sharad Gaikwad (AWS)

Summary

Notice: AWS CodeCommit is no longer available to new customers. Existing customers of AWS CodeCommit can continue to use the service as normal. Learn more.

Amazon ECS Anywhere is an extension of the Amazon Elastic Container Service (Amazon ECS). It provides support for registering an external instance, such as an on-premises server or virtual machine (VM), to your Amazon ECS cluster. is feature helps reduce costs and mitigate complex local container orchestration and operations. You can use ECS Anywhere to deploy and run container applications in both on-premises and cloud environments. It removes the need for your team to learn multiple domains and skill sets, or to manage complex software on their own.

This pattern describes a step-by-step approach to provision an Amazon ECS cluster with Amazon ECS Anywhere instances by using Amazon Web Services (AWS) Cloud Development Kit (AWS CDK) stacks. You then use AWS CodePipeline to set up a continuous integration and continuous deployment (CI/CD) pipeline. Then, you replicate your GitLab code repository to AWS CodeCommit and deploy your containerized application on the Amazon ECS cluster.

This pattern is designed to help those who use on-premises infrastructure to run container applications and use GitLab to manage the application code base. You can manage those workloads by using AWS Cloud services, without disturbing your existing, on-premises infrastructure.

Prerequisites and limitations

Prerequisites

  • An active AWS account.

  • A container application running on on-premises infrastructure.

  • A GitLab repository where you manage your application code base. For more information, see Repository (GitLab).

  • AWS Command Line Interface (AWS CLI), installed and configured. For more information, see Installing or updating the latest version of the AWS CLI (AWS CLI documentation).

  • AWS CDK Toolkit, installed and configured globally. For more information, see Install the AWS CDK (AWS CDK documentation).

  • npm, installed and configured for the AWS CDK in TypeScript. For more information, see Downloading and installing Node.js and npm (npm documentation).

Limitations

Product versions

  • AWS CDK Toolkit version 2.27.0 or later

  • npm version 7.20.3 or later

  • Node.js version 16.6.1 or later

Architecture

Target technology stack

  • AWS CDK

  • AWS CloudFormation

  • AWS CodeBuild

  • AWS CodeCommit

  • AWS CodePipeline

  • Amazon ECS Anywhere

  • Amazon Elastic Container Registry (Amazon ECR)

  • AWS Identity and Access Management (IAM)

  • AWS System Manager

  • GitLab repository

Target architecture

Architecture diagram of setting up the Amazon ECS cluster and CI/CD pipeline.

This diagram represents two primary workflows described in this pattern, provisioning the Amazon ECS cluster and setting up the CI/CD pipeline that sets up and deploys the CI/CD pipeline, as follows:

  1. Provisioning the Amazon ECS cluster

    1. When you deploy the first AWS CDK stack, it creates a CloudFormation stack on AWS.

    2. This CloudFormation stack provisions an Amazon ECS cluster and related AWS resources.

    3. To register an external instance with an Amazon ECS cluster, you must install AWS Systems Manager Agent (SSM Agent) on your VM and register the VM as an AWS Systems Manager managed instance. 

    4. You must also install the Amazon ECS container agent and Docker on your VM to register it as an external instance with the Amazon ECS cluster.

    5. When the external instance is registered and configured with the Amazon ECS cluster, it can run multiple containers on your VM, which is registered as an external instance.

    6. The Amazon ECS cluster is active and can run the application workloads through containers. The Amazon ECS Anywhere container instance runs in on-premises environment but is associated with the Amazon ECS cluster in the cloud.

  2. Setting up and deploying the CI/CD pipeline

    1. When you deploy the second AWS CDK stack, it creates another CloudFormation stack on AWS.

    2. This CloudFormation stack provisions a pipeline in CodePipeline and related AWS resources.

    3. You push and merge application code changes to an on-premises GitLab repository. 

    4. The GitLab repository is automatically replicated to the CodeCommit repository.

    5. The updates to the CodeCommit repo automatically starts CodePipeline. 

    6. CodePipeline copies code from CodeCommit and creates the deployable application build in CodeBuild.

    7. CodePipeline creates a Docker image of the CodeBuild build environment and pushes it to the Amazon ECR repo.

    8. CodePipeline initiates CodeDeploy actions that pull the container image from the Amazon ECR repo.

    9. CodePipeline deploys the container image on the Amazon ECS cluster.

Automation and scale

This pattern uses the AWS CDK as an infrastructure as code (IaC) tool to configure and deploy this architecture. AWS CDK helps you orchestrate the AWS resources and set up Amazon ECS Anywhere and the CI/CD pipeline.

Tools

AWS services

Other tools

  • Node.js is an event-driven JavaScript runtime environment designed for building scalable network applications.

  • npm is a software registry that runs in a Node.js environment and is used to share or borrow packages and manage deployment of private packages.

  • Vagrant is an open-source utility for building and maintaining portable virtual software development environments. For demonstration purposes, this pattern uses Vagrant to create an on-premises VM.

Code repository

The code for this pattern is available in the GitHub CI/CD pipeline for Amazon ECS Anywhere using AWS CDK repository.

Best practices

Consider the following best practices when deploying this pattern:

Epics

TaskDescriptionSkills required

Verify the AWS CDK version.

Verify the version of the AWS CDK Toolkit by entering the following command.

cdk --version

This pattern requires version 2.27.0 or later. If you have an earlier version, follow the instructions in the AWS CDK documentation to update it.

DevOps engineer

Verify the npm version.

Verify the version of npm by entering the following command.

npm --version

This pattern requires version 7.20.3 or later. If you have an earlier version, follow the instructions in the npm documentation to update it.

DevOps engineer

Set up AWS credentials.

Set up AWS credentials by entering the aws configure command and following the prompts.

$aws configure
AWS Access Key ID [None]: <your-access-key-ID>
AWS Secret Access Key [None]: <your-secret-access-key>
Default region name [None]: <your-Region-name>
Default output format [None]:
DevOps engineer
TaskDescriptionSkills required

Clone the AWS CDK code repository.

  1. Clone the CI/CD pipeline for Amazon ECS Anywhere using AWS CDK repository for this pattern by entering the following command.

    git clone https://github.com/aws-samples/amazon-ecs-anywhere-cicd-pipeline-cdk-sample.git

  2. Navigate into the cloned directory by entering the following command.

    cd amazon-ecs-anywhere-cicd-pipeline-cdk-sample
DevOps engineer

Bootstrap the environment.

Deploy the CloudFormation template to the account and AWS Region that you want to use by entering the following command.

cdk bootstrap <account-number>/<Region>

For more information, see Bootstrapping in the AWS CDK documentation.

DevOps engineer
TaskDescriptionSkills required

Install the package dependencies and compile the TypeScript files.

Install the package dependencies and compile the TypeScript files by entering the following commands.

$cd EcsAnywhereCdk
$npm install
$npm fund

These commands install all the packages from the sample repository. For more information, see npm ci and npm install in the npm documentation. If you get any errors about missing packages when you enter these commands, see the Troubleshooting section of this pattern.

DevOps engineer

Build the project.

To build the project code, enter the following command.

npm run build

For more information about building and deploying the project, see Your first AWS CDK app in the AWS CDK documentation.

DevOps engineer

Deploy the Amazon ECS Anywhere infrastructure stack.

  1. List the stacks by entering the following command.

    $cdk list

  2. Confirm that the output returns the EcsAnywhereInfraStack and ECSAnywherePipelineStack stacks.

  3. Deploy the EcsAnywhereInfraStack stack by entering the following command.

    $cdk  deploy EcsAnywhereInfraStack
DevOps engineer

Verify stack creation and output.

  1. Sign in to the AWS Management Console and open the CloudFormation console at https://console.aws.amazon.com/cloudformation/.

  2. On the Stacks page, select the EcsAnywhereInfraStack stack.

  3. Confirm that the stack status is either CREATE_IN_PROGRESS or CREATE_COMPLETE.

    Setting up the Amazon ECS cluster can take some time. Do not proceed until the stack creation is complete.

DevOps engineer
TaskDescriptionSkills required

Set up your VM.

Create a Vagrant VM by entering the vagrant up command from the root directory where Vagrantfile is located. For more information, see the Vagrant documentation.

DevOps engineer

Register your VM as an external instance.

  1. Log in to the Vagrant VM by using the vagrant ssh command. For more information, see the Vagrant documentation.

  2. Install AWS CLI on the VM by following AWS CLI installation instructions and entering the following commands. 

    $ curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" \
> -o "awscliv2.zip"
$sudo apt install unzip
$unzip awscliv2.zip
$sudo ./aws/install
$aws configure
AWS Access Key ID [None]: <your-access-key-ID>
AWS Secret Access Key [None]: <your-secret-access-key>
Default region name [None]: <your-Region-name>
Default output format [None]:

  1. Create an activation code and ID that you can use to register your VM with AWS Systems Manager and to activate your external instance. The output from this command includes the activation ID and activation code values.

    aws ssm create-activation \
> --iam-role EcsAnywhereInstanceRole \
> | tee ssm-activation.json

    If you receive an error when you run this command, see the Troubleshooting section.

  2. Export the activation ID and code values.

    export ACTIVATION_ID=<activation-ID>
export ACTIVATION_CODE=<activation-code>

  3. Download the installation script to your VM.

    curl --proto "https" -o "ecs-anywhere-install.sh" \
> "https://amazon-ecs-agent.s3.amazonaws.com/ecs-anywhere-install-latest.sh"

  4. Run the installation script on your VM.

    sudo bash ecs-anywhere-install.sh \
--cluster EcsAnywhereCluster \
--activation-id $ACTIVATION_ID \
--activation-code $ACTIVATION_CODE \
--region <region-name>

This sets up your VM was an Amazon ECS Anywhere external instance and registers the instance in the Amazon ECS cluster. For more information, see Registering an external instance to a cluster in the Amazon ECS documentation. If you experience any issues, see the Troubleshooting section.

DevOps engineer

Verify the status of Amazon ECS Anywhere and the external VM.

To verify whether your VM is connected to the Amazon ECS control plane and running, use the following commands.

$aws ssm describe-instance-information
$aws ecs list-container-instances --cluster $CLUSTER_NAME
DevOps engineer
TaskDescriptionSkills required

Create a branch in the CodeCommit repo.

Create a branch named main in the CodeCommit repo by creating the first commit for the repository. You can follow AWS documentation to Create a commit in CodeCommit. The following command is an example.

aws codecommit put-file \
  --repository-name EcsAnywhereRepo \
  --branch-name main \
  --file-path README.md \
  --file-content "Test" \
  --name "Dev Ops" \
  --email "devops@example.com" \
  --commit-message "Adding README."
DevOps engineer

Set up repo mirroring.

You can mirror a GitLab repository to and from external sources. You can select which repository serves as the source. Branches, tags, and commits are synced automatically. Set up a push mirror between the GitLab repository that hosts your application and the CodeCommit repository. For instructions, see Set up a push mirror from GitLab to CodeCommit (GitLab documentation).

Note

By default, mirroring automatically syncs the repository. If you want to manually update the repositories, see Update a mirror (GitLab documentation).

DevOps engineer

Deploy the CI/CD pipeline stack.

Deploy the EcsAnywherePipelineStack stack by entering the following command.

$cdk  deploy EcsAnywherePipelineStack
DevOps engineer

Test the CI/CD pipeline.

  1. Make application code changes and push it to the source, on-premises GitLab repo. For more information, see Push Options (GitLab documentation). For example, edit the ../application/index.html file to update the application version value.

  2. When the code is replicated to the CodeCommit repo, this starts the CI/CD pipeline. Do one of the following:

    • If you are using automatic mirroring to synchronize the GitLab repo with the CodeCommit repo, continue to the next step.

    • If you are using manual mirroring, push the application code changes to the CodeCommit repo by following the instructions in Update a mirror (GitLab documentation).

  3. On your local machine, in a web browser, enter http://localhost:80. This opens the NGINX web page because port 80 is forwarded to localhost in Vagrantfile. Confirm that you can view the updated application version value. This validates the pipeline and image deployment.

  4. (Optional) If you want to verify the deployment in the AWS Management Console, do the following:

    1. Open the Amazon ECS console at https://console.aws.amazon.com/ecs/.

    2. From the navigation bar, select the Region to use.

    3. In the navigation pane, choose Clusters.

    4. On the Clusters page, select the EcsAnywhereCluster cluster.

    5. Choose Task Definitions.

    6. Confirm the container is running.

DevOps engineer
TaskDescriptionSkills required

Clean up and delete the resources.

After you walk through this pattern, you should remove the proof-of-concept resources you created. To clean up, enter the following commands.

$cdk destroy EcsAnywherePipelineStack
$cdk destroy EcsAnywhereInfraStack
DevOps engineer

Troubleshooting

IssueSolution

Errors about missing packages when installing package dependencies.

Enter one of the following commands to resolve missing packages.

$npm ci

or

$npm install -g @aws-cdk/<package_name>

When you run the aws ssm create-activation command on the VM, you receive the following error.

An error occurred (ValidationException) when calling the CreateActivation operation: Nonexistent role or missing ssm service principal in trust policy: arn:aws:iam::000000000000:role/EcsAnywhereInstanceRole

The EcsAnywhereInfraStack stack isn’t fully deployed, and the IAM role necessary to run this command hasn’t been created yet. Check the stack status in the CloudFormation console. Retry the command after the status changes to CREATE_COMPLETE.

An Amazon ECS health check returns UNHEALTHY, and you see the following error in the Services section of the cluster in the Amazon ECS console.

service EcsAnywhereService was unable to place a task because no container instance met all of its requirements. Reason: No Container Instances were found in your cluster.

Restart the Amazon ECS agent on your Vagrant VM by entering the following commands.

$vagrant ssh
$sudo systemctl restart ecs
$sudo systemctl status ecs

Related resources