Jenkins on AWS
Jenkins on AWS

Containerized Deployment

Containers are a form of operating system virtualization that allow you to run an application and its dependencies in resource-isolated processes.

Overview of Container Technology

Containers allow you to easily package an application's code, configurations, and dependencies into a template called an image, which is used to launch the container. Containers can help ensure that applications deploy quickly, reliably, and consistently regardless of the deployment environment. You also have more granular control over resources, which can improve the efficiency of your infrastructure.

Overview of Container-based Jenkins

Another approach to deploying and running Jenkins is to run the master and the worker nodes in Docker containers. A Docker container packages an application, its configurations, and all its dependencies into a single unit of deployment that contains everything the application needs to run.

Running Jenkins in Docker containers allows you to use servers running Jenkins worker nodes more efficiently. It also simplifies the configuration of the worker node servers. Using containers to manage builds allows the underlying servers to be pooled into a cluster. The Jenkins worker containers can then run and execute a build on any of the servers with resources available to support the build. This ability to pool multiple builds to run independently of each other on the server improves the utilization of the server.

Another benefit of containers is that they allow for tests to be run in “clean” environments every time, as opposed to environments that can potentially become “dirty” over time and corrupt the tests.

Configuration is also simplified when you use containers for running the workers. The same base configuration can be reused multiple times. By making the worker containers fungible you can easily run multiple compiler types and versions, libraries, and any other dependencies.

Amazon ECS

Amazon EC2 Container Service (Amazon ECS) is a container management service that supports creating a cluster of Amazon EC2 instances for running the Jenkins master nodes and worker nodes as Docker containers. This cluster of servers is typically referred to as a build farm.

Amazon ECS eliminates the need for you to install, operate and scale your own cluster management infrastructure. You can use API calls to launch and stop Docker-enabled applications, query the complete state of your cluster, and access many familiar features like security groups, load balancers, Amazon EBS volumes, and IAM roles.

The build farm based on Amazon ECS consists of the following components:

  • Cluster – A logical grouping of container instances into which you can place Amazon ECS tasks for the Jenkins master or worker node.

  • Amazon ECS Container Agent – Allows container instances to connect to the cluster and is included in the AMI that is optimized for Amazon ECS. The Amazon ECS container agent can also be installed on any EC2 instance that supports the Amazon ECS specification.

  • Container instance – The EC2 instance that is running the Amazon ECS container agent and has been registered into a cluster. The Jenkins master and worker nodes can run on any of the container instances in the cluster.

  • Task definition – An application description that contains one or more container definitions. The task definition captures how the task running the Jenkins master is configured, for example, how much memory the Jenkins master needs, what port it would be accessed on, etc.

  • Scheduler – The method used for placing tasks on container instances. The scheduler determines what container instance a Jenkins worker node or master runs on when it’s started.

  • Service – Allows you to run and maintain a specified number of instances of a task definition simultaneously. The Jenkins master is a long-running task and runs as a service.

  • Task – An instantiation of a task definition running on a container instance. The Jenkins worker nodes run as tasks.

  • Container – A Docker container created as part of a task.

            Amazon ECS components

Figure: Amazon ECS components


In this section we review the steps needed to implement containerized Jenkins.

Setting Up the Amazon ECS Cluster for Our Build Farm

The first step in setting up the build farm is to launch an Amazon ECS cluster in a VPC. You must specify the instance type to be used for the build servers and the number of Amazon EC2 instances needed in the build farm.

Creating the Jenkins Master Service

To create the Jenkins master, create a Dockerfile that uses the official Jenkins Docker image as the base. The Dockerfile should also contain the steps to install the Jenkins Amazon ECS plugin and any other plugins needed. Specifically, the Amazon ECS plugin will allow the Jenkins master to use Amazon ECS to orchestrate starting up and tearing down the Jenkins workers as needed.

# Dockerfile for Jenkins Master FROM Jenkins # Add the entry amazon-ecs to plugin.txt to preload the Amazon ECS plugin COPY plugins.txt /usr/share/jenkins/plugins.txt RUN /usr/local/bin/ /usr/share/jenkins/plugins.txt

Use the Dockerfile to create an image and push the image to the Amazon EC2 Container Registry (Amazon ECR). Amazon ECR is a fully-managed Docker container registry that makes it easy for developers to store, manage, and deploy Docker container images. The custom image can now be used to run one or more Jenkins masters in the Amazon ECS cluster. (You might do this if, for example, you have several teams that each need their own Jenkins environment.)

# Login using docker command returned by the command below aws ecr get-login --region region # Build your Docker image using the Dockerfile docker build -t jenkins-master . # Tag the jenkins_master image docker tag jenkins_master:latest AWS Account # Push the jenkins-master image to ECR docker push AWS Account

Jenkins uses the JENKINS_HOME directory to store information on installed plugins, logs, build jobs, and other configuration settings. By design, the data in the Jenkins master container is ephemeral and is lost when the container is stopped or restarted. If you don’t want to lose this data, Amazon ECS supports persisting the data by using data volume containers.

The Jenkins master container persists the data in JENKINS_HOME by mounting a volume from the data volume container.

Behind the scenes, the data volume in the data container is associated with a directory in a file system in an Amazon EBS volume attached to the container instance.

Note that Amazon ECS does not sync your data volume across container instances. The Jenkins master container has to be started on the same container instance that has the data container volume.

To create a data volume container for use by the Jenkins master container, create a Dockerfile that exports the Jenkin home directory /var/jenkins_home.

# Dockerfile for Jenkins Data Volume Container FROM Jenkins VOLUME ["/var/jenkins_home"] # Creating a Jenkins Data Volume docker build -t jenkins_dv . #Tag the jenkins_dv image docker tag jenkins_dv:latest AWS Account # Push the jenkins-dv image to ECR docker push AWS Account

You can run the Jenkins master as an Amazon ECS service using a Docker compose file, which starts the Jenkins master and the Jenkins data volume containers. When running as an Amazon ECS service the Jenkins master will restart if its container fails or stops.

Create a Docker compose file with the entries below:

jenkins_master: image: jenkins_master cpu_shares: 100 mem_limit: 2000M ports: ports: - "8080:8080", “50000:50000” volumes_from: jenkins_dv jenkins_dv: image: jenkins_dv cpu_shares: 100 mem_limit: 500M

Use the Amazon ECS command line to start the Jenkins master container as a service:

# Create an ECS service from Jenkins compose file ecs-cli compose --file compose file service up

The Jenkins master should also be configured with AWS credentials that give the Jenkins master the appropriate privileges to register task definitions, run tasks, and stop tasks in the Amazon ECS cluster.

Setting Up Jenkins Builders

To set up the Docker image used by the Jenkins master to start Amazon ECS Jenkins worker tasks, use the Jenkins continuous integration (CI) official image for the Docker-based JNLP worker as the base image in the Dockerfile. Then add any additional tools needed to support the build process into the Dockerfile. Store the custom Jenkins worker image created with the Dockerfile in Amazon ECR.

CloudBees also provides a Docker image for JNLP worker nodes that includes the tools to build Java applications that can be used as the image to run Amazon ECS workers.

Use the custom Jenkins worker image to configure the worker template in the Jenkins master. The worker template specifies the label used to refer to the worker, the Docker image used to create the worker, and the resources (such as CPU and memory) needed by the worker Amazon ECS task.

When creating a job in the Jenkins master you can specify which type of Amazon ECS worker you want for the job. Just specify the label used in the

Amazon ECS worker template configuration in the Restrict where this Project can run field.

Security Considerations

Security Groups

The security considerations when running the Jenkins master and worker nodes in Amazon ECS are similar to the traditional deployment of Jenkins. The security group associated with the container instances in the cluster should only allow TCP traffic to 8080 and only allow access to port 5000 on container instances from other container instances in the clusters.