Automatically build and deploy a Java application to Amazon EKS using a CI/CD pipeline - AWS Prescriptive Guidance
SummaryPrerequisites and limitationsArchitectureToolsBest practicesEpicsRelated resourcesAdditional information

Automatically build and deploy a Java application to Amazon EKS using a CI/CD pipeline

Created by MAHESH RAGHUNANDANAN (AWS), James Radtke (AWS), and Jomcy Pappachen (AWS)

Code repository: aws-cicd-java-eks

Environment: Production

Technologies: Containers & microservices; CloudNative; DevOps; Modernization

Workload: All other workloads

AWS services: AWS CloudFormation; AWS CodeCommit; AWS CodePipeline; Amazon EC2 Container Registry; Amazon EKS

Summary

This pattern describes how to create a continuous integration and continuous delivery (CI/CD) pipeline that automatically builds and deploys a Java application with recommended DevSecOps practices to an Amazon Elastic Kubernetes Service (Amazon EKS) cluster on the Amazon Web Services (AWS) Cloud. This pattern uses a greeting application developed with a Spring Boot Java framework and that uses Apache Maven.

You can use this pattern’s approach to build the code for a Java application, package the application artifacts as a Docker image, security scan the image, and upload the image as a workload container on Amazon EKS. This pattern's approach is useful if you want to migrate from a tightly coupled monolithic architecture to a microservices architecture. The approach also helps you to monitor and manage the entire lifecycle of a Java application, which ensures a higher level of automation and helps avoid errors or bugs.

Prerequisites and limitations

Prerequisites 

  • An active AWS account.

  • AWS Command Line Interface (AWS CLI) version 2, installed and configured. For more information about this, see Installing, updating, and uninstalling the AWS CLI version 2 in the AWS CLI documentation.

  • AWS CLI version 2 must be configured with the same IAM role that creates the Amazon EKS cluster because only that role is authorized to add other IAM roles to the aws-auth ConfigMap. For information and steps to configure AWS CLI, see Configuration basics in the AWS CLI documentation.

  • AWS Identity and Access Management (IAM) roles and permissions with full access to AWS CloudFormation. For more information about this, see Controlling access with IAM in the AWS CloudFormation documentation.

  • An existing Amazon EKS cluster, with details of the IAM role name and IAM role Amazon Resource Name (ARN) of worker nodes in the EKS cluster.

  • Kubernetes Cluster Autoscaler, installed and configured in your Amazon EKS cluster. For more information, see Cluster Autoscaler in the Amazon EKS documentation. 

  • Access to code in the GitHub repository.

Important note

AWS Security Hub is enabled as part of the AWS CloudFormation templates that are in the code. By default, after Security Hub is enabled, it comes with a 30–day free trial, after which there is a cost associated with this AWS service. For more information about pricing, see AWS Security Hub pricing.

Product versions

  • Helm version 3.4.2 or later

  • Apache Maven version 3.6.3 or later

  • BridgeCrew Checkov version 2.2 or later

  • Aqua Security Trivy version 0.37 or later

Architecture

Technology stack

  • AWS CodeBuild

  • AWS CodeCommit

  • Amazon CodeGuru

  • AWS CodePipeline

  • Amazon Elastic Container Registry

  • Amazon Elastic Kubernetes Service

  • Amazon EventBridge

  • AWS Security Hub

  • Amazon Simple Notification Service (Amazon SNS)

Target architecture

""

The diagram shows the following workflow:

  1. The developer updates the Java application code in the base branch of the CodeCommit repository, which creates a pull request (PR).

  2. As soon as the PR is submitted, Amazon CodeGuru Reviewer automatically reviews the code, analyzes it based on best practices for Java, and gives recommendations to the developer.

  3. After the PR is merged to the base branch, an Amazon EventBridge event is created.

  4. The EventBridge event initiates the CodePipeline pipeline, which starts .

  5. CodePipeline runs the CodeSecurity Scan stage (continuous security).

  6. CodeBuild starts the security scan process in which the Dockerfile and Kubernetes deployment Helm files are scanned using Checkov, and application source code is scanned based on incremental code changes. The application source code scan is performed by the CodeGuru Reviewer Command Line Interface (CLI) wrapper.

  7. If the security scan stage is successful, the Build stage (continuous integration) is initiated.

  8. In the Build stage, CodeBuild builds the artifact, packages the artifact to a Docker image, scans the image for security vulnerabilities by using Aqua Security Trivy, and stores the image in Amazon ECR.

  9. The vulnerabilities detected from step 8 are uploaded to Security Hub for further analysis by developers or engineers. Security Hub provides an overview and recommendations for remediating the vulnerabilities.

  10. Email notifications of various phases within the CodePipeline pipeline are sent through Amazon SNS.

  11. After the continuous integration phases are complete, CodePipeline enters the Deploy stage (continuous delivery).

  12. The Docker image is deployed to Amazon EKS as a container workload (pod) by using Helm charts.

  13. The application pod is configured with Amazon CodeGuru Profiler Agent which will send the profiling data of the application (CPU, heap usage, and latency) to Amazon CodeGuru Profiler, which helps developers to understand the behavior of the application.

Tools

AWS services

  • AWS CloudFormation helps you set up AWS resources, provision them quickly and consistently, and manage them throughout their lifecycle across AWS accounts and Regions.

  • AWS CodeBuild is a fully managed build service that helps you compile source code, run unit tests, and produce artifacts that are ready to deploy.

  • AWS CodeCommit is a version control service that helps you privately store and manage Git repositories, without needing to manage your own source control system.

  • Amazon CodeGuru Profiler collects runtime performance data from your live applications, and provides recommendations that can help you fine-tune your application performance.

  • Amazon CodeGuru Reviewer uses program analysis and machine learning to detect potential defects that are difficult for developers to find and offers suggestions for improving your Java and Python code.

  • AWS CodePipeline helps you quickly model and configure the different stages of a software release and automate the steps required to release software changes continuously.

  • Amazon Elastic Container Registry (Amazon ECR) is a managed container image registry service that’s secure, scalable, and reliable.

  • Amazon Elastic Kubernetes Service (Amazon EKS) helps you run Kubernetes on AWS without needing to install or maintain your own Kubernetes control plane or nodes.

  • Amazon EventBridge is a serverless event bus service that helps you connect your applications with real-time data from a variety of sources. For example, AWS Lambda functions, HTTP invocation endpoints using API destinations, or event buses in other AWS accounts.

  • AWS Identity and Access Management (IAM) helps you securely manage access to your AWS resources by controlling who is authenticated and authorized to use them.

  • AWS Security Hub provides a comprehensive view of your security state in AWS. It also helps you check your AWS environment against security industry standards and best practices.

  • Amazon Simple Notification Service (Amazon SNS) helps you coordinate and manage the exchange of messages between publishers and clients, including web servers and email addresses.

  • Amazon Simple Storage Service (Amazon S3) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data.

Other services

  • Helm is an open-source package manager for Kubernetes.

  • Apache Maven is a software project management and comprehension tool.

  • BridgeCrew Checkov is a static code analysis tool for scanning infrastructure as code (IaC) files for misconfigurations that might lead to security or compliance problems.

  • Aqua Security Trivy is a comprehensive scanner for vulnerabilities in container images, file systems, and Git repositories, in addition to configuration issues.

Code 

The code for this pattern is available in the GitHub aws-codepipeline-devsecops-amazoneks repository.

Best practices

  • The principle of least privilege has been followed for IAM entities across all the phases of this solution. If you want to extend the solution with additional AWS services or third-party tools, we recommend following the principle of least privilege.

  • If you have multiple Java applications, we recommend creating separate CI/CD pipelines for each application.

  • If you have a monolith application, we recommend breaking the application into microservices as much as possible. Microservices are more flexible, they make it easier to deploy applications as containers, and they provide better visibility into the overall build and deployment of the application.

Epics

TaskDescriptionSkills required

Clone the GitHub repository.

To clone the repository, run the following command.

git clone https://github.com/aws-samples/aws-codepipeline-devsecops-amazoneks
App developer, DevOps engineer

Create an S3 bucket and upload the code.

  1. Sign in to the AWS Management Console, open the Amazon S3 console, and then create an S3 bucket in the AWS Region where you plan to deploy this solution. For more information, see Creating a bucket in the Amazon S3 documentation.

  2. In the S3 bucket, create a folder named code.

  3. Navigate to where you cloned the repository. To a create compressed version of the entire code with the .zip extension (cicdstack.zip) and validate the .zip file, run the following commands in order.

    Note: If the python command fails and states that Python was not found, use python3 instead.

    cd aws-codepipeline-devsecops-amazoneks
python -m zipfile -c cicdstack.zip *
python -m zipfile -t cicdstack.zip

  4. Upload the cicdstack.zip file to the code folder that you previously created in the S3 bucket.

AWS DevOps, DevOps engineer, Cloud administrator, DevOps

Create an AWS CloudFormation stack.

  1. Open the AWS CloudFormation console and choose Create stack.

  2. In Specify template, choose Upload a template file, upload the cf_templates/codecommit_ecr.yaml file, and then choose Next.

  3. In Specify stack details, enter the stack name, and then provide the following input parameter values:

    • CodeCommitRepositoryBranchName: The branch-name where your code will be residing(the default is main)

    • CodeCommitRepositoryName: The name of the CodeCommit repo to be created.

    • CodeCommitRepositoryS3Bucket: The name of the S3 bucket where you created the code folder

    • CodeCommitRepositoryS3BucketObjKey: code/cicdstack.zip

    • ECRRepositoryName: The name of the Amazon ECR repo to be created

  4. Choose Next, use the default settings for the Configure stack options, and then choose Next.

  5. In the Review section, verify the template and stack details, and then choose Create stack. The stack is then created, including the CodeCommit and Amazon ECR repositories.

  6. Note the names of the CodeCommit and Amazon ECR repositories, which will be required for the Java CI/CD pipeline setup.

AWS DevOps, DevOps

Validate the CloudFormation stack deployment.

  1. Under Stacks on the CloudFormation console, verify the status of CloudFormation stack that you deployed. The status of the stack should be CREATE COMPLETE.

  2. Additionally, from the console, validate that CodeCommit and Amazon ECR have been provisioned and are ready.

DevOps engineer

Delete the S3 bucket.

Empty and delete the S3 bucket that you created earlier. For more information, see Deleting a bucket in the Amazon S3 documentation.

AWS DevOps, DevOps
TaskDescriptionSkills required

Configure the Helm charts of your Java application.

  1. In the location where you cloned the GitHub repository, navigate to the folder helm_charts/aws-proserve-java-greeting. In this folder, the values.dev.yaml file contains information about Kubernetes resources configuration that you can modify for your container deployments to Amazon EKS. Update the Docker repository parameter by providing your AWS account ID, AWS Region, and Amazon ECR repository name.

    image:
  repository: <account-id>.dkr.ecr.<region>.amazonaws.com/<app-ecr-repo-name>

  2. The Java pod's service type is set to LoadBalancer.

    service:
  type: LoadBalancer
  port: 80
  targetPort: 8080
  path: /hello
  initialDelaySeconds: 60
  periodSeconds: 30

    To use a different service (for example, NodePort), you can change the parameters. For more information, see the Kubernetes documentation.

  3. You can activate the Kubernetes Horizontal Pod Autoscaler by changing the autoscaling parameter to enabled: true.

    autoscaling:
  enabled: true
  minReplicas: 1
  maxReplicas: 100
  targetCPUUtilizationPercentage: 80
  # targetMemoryUtilizationPercentage: 80

You can enable different features for the Kubernetes workloads by changing the values in the values.<ENV>.yaml file, where <ENV> is your development, production, UAT, or QA environment.

DevOps

Validate Helm charts for syntax errors.

  1. From the terminal, verify that Helm v3 is installed in your local workstation by running the following command.

    helm --version

    If Helm v3 isn’t installed, install it.

  2. In the terminal, navigate to the Helm charts directory (helm_charts/aws-proserve-java-greeting), and run the following command.

    helm lint . -f values.dev.yaml

    This will check the Helm charts for any syntax errors.

DevOps engineer
TaskDescriptionSkills required

Create the CI/CD pipeline.

  1. Open the AWS CloudFormation console, and choose Create stack.

  2. In Specify template, choose Upload a template file, upload the cf_templates/build_deployment.yaml template, and then choose Next.

  3. In Specify stack details, specify the Stack name, and then provide the following values for the input parameters:

    • CodeBranchName: Branch name of CodeCommit repo, where your code resides

    • EKSClusterName: Name of your EKS cluster (not the EKSCluster ID)

    • EKSCodeBuildAppName: Name of the app Helm chart (aws-proserve-java-greeting)

    • EKSWorkerNodeRoleARN: ARN of the Amazon EKS worker nodes IAM role

    • EKSWorkerNodeRoleName: Name of the IAM role assigned to the Amazon EKS worker nodes

    • EcrDockerRepository: Name of Amazon ECR repo where the Docker images of your code will be stored

    • EmailRecipient: Email address where build notifications need to be sent

    • EnvType: Environment (for example, dev, test, or prod)

    • SourceRepoName: Name of the CodeCommit repo, where your code resides

  4. Choose Next. Use the default settings in Configure stack options, and then choose Next.

  5. In the Review section, verify the AWS CloudFormation template and stack details, and then choose Next.

  6. Choose Create stack. 

  7. During the CloudFormation stack deployment, the owner of the email address that you provided in the parameters will receive a message to subscribe to an SNS topic. To subscribe to Amazon SNS, the owner must choose the link in the message.

  8. After the stack is created, open the Outputs tab of the stack, and then record the ARN value for the EksCodeBuildkubeRoleARN output key. This IAM ARN value will be required later for providing the CodeBuild IAM role with permissions to deploy workloads in the Amazon EKS cluster.

AWS DevOps
TaskDescriptionSkills required

Turn on Aqua Security integration.

This step is required for uploading the Docker image vulnerability findings reported by Trivy to Security Hub. Because AWS CloudFormation doesn’t support Security Hub integrations, this process must be done manually.

  1. Open the AWS Security Hub console, and navigate to Integrations.

  2. Search for Aqua Security, and select Aqua Security: Aqua Security.

  3. Choose Accept findings.

AWS administrator, DevOps engineer
TaskDescriptionSkills required

Allow CodeBuild to run Helm or kubectl commands in the Amazon EKS cluster.

For CodeBuild to be authenticated to use Helm or kubectl commands with the EKS cluster, you must add the IAM roles to the aws-auth ConfigMap. In this case, add the ARN of IAM role EksCodeBuildkubeRoleARN, which is the IAM role created for the CodeBuild service to access the EKS cluster and deploy workloads on it. This is a one-time activity.

Important: The following procedure must be completed before the deployment approval stage in CodePipeline.

  1. Open the cf_templates/kube_aws_auth_configmap_patch.sh shell script in your Amazon Linux or macOS environment.

  2. Authenticate to the Amazon EKS cluster by running the following command.

    aws eks --region <aws-region> update-kubeconfig --name <eks-cluster-name>

  3. Run the shell script by using the following command, replacing <rolearn-eks-codebuild-kubectl> with the ARN value of EksCodeBuildkubeRoleARN that you recorded earlier.

    bash cf_templates/kube_aws_auth_configmap_patch.sh <rolearn-eks-codebuild-kubectl> 

The aws_auth ConfigMap is configured, and access is granted.

DevOps
TaskDescriptionSkills required

Verify that the CI/CD pipeline automatically initiates.

  1. The CodeSecurity Scan stage in the pipeline will usually fail if Checkov detects vulnerabilities in the Dockerfile or  Helm charts. However, the purpose of this example is to establish a process of identifying potential security vulnerabilities rather than fixing it through the CI/CD pipeline, typically a DevSecOps process. In the file buildspec/buildspec_secscan.yaml, the checkov command uses the --soft-fail flag to avoid pipeline failure.

          - echo -e "\n Running Dockerfile Scan"
      - checkov -f code/app/Dockerfile --framework dockerfile --soft-fail --summary-position bottom
      - echo -e "\n Running Scan of Helm Chart files"
      - cp -pv helm_charts/$EKS_CODEBUILD_APP_NAME/values.dev.yaml helm_charts/$EKS_CODEBUILD_APP_NAME/values.yaml
      - checkov -d helm_charts/$EKS_CODEBUILD_APP_NAME --framework helm --soft-fail --summary-position bottom
      - rm -rfv helm_charts/$EKS_CODEBUILD_APP_NAME/values.yaml

    For the pipeline to fail when vulnerabilities are reported for the Dockerfile and Helm charts, the --soft-fail option must be removed from the checkov command. Developers or engineers can then fix the vulnerabilities and commit the changes to the CodeCommit source code repository.

  2. Similar to CodeSecurity Scan, the Build stage uses Aqua Security Trivy for identifying HIGH and CRITICAL Docker image vulnerabilities before pushing the application. to Amazon ECR. In this example, we are not making pipeline fail for Docker image vulnerabilities. In the file buildspec/buildspec.yml, the trivy command includes the flag --exit-code with a value 0, which is why pipeline doesn’t fail when HIGH or CRITICAL Docker image vulnerabilities are reported.

          - AWS_REGION=$AWS_DEFAULT_REGION AWS_ACCOUNT_ID=$AWS_ACCOUNT_ID trivy -d image --no-progress --ignore-unfixed --exit-code 0 --severity HIGH,CRITICAL --format template --template "@securityhub/asff.tpl" -o securityhub/report.asff $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$CODEBUILD_RESOLVED_SOURCE_VERSION

    For the pipeline to fail when HIGH, CRTICAL vulnerabilities are reported, change the value of --exit-code to 1.

    Developers or engineers can then fix the vulnerabilities and commit the changes to the CodeCommit source code repository.

  3. Docker image vulnerabilities reported by Aqua Security Trivy are uploaded to Security Hub. On the AWS Security Hub console, navigate to Findings. Filter the findings with Record State = Active and Product = Aqua Security. This will list the Docker image vulnerabilities in Security Hub. It can take 15 minutes–1 hour for vulnerabilities to appear on Security Hub.

For more information about starting the pipeline by using CodePipeline, see Start a pipeline in CodePipeline, Start a pipeline manually, and Start a pipeline on a schedule in the AWS CodePipeline documentation.

DevOps

Approve the deployment.

  1. After the build phase is complete, there is a deployment approval gate. The reviewer or a release manager should inspect the build and, if all requirements are met, approve it. This is the recommended approach for teams that use continuous delivery for application deployment.

  2. After approval, the pipeline initiates the Deploy stage.

  3. After the Deploy stage is successful, the CodeBuild log for this stage provides the URL of the application. Use the URL to validate the readiness of the application.

DevOps

Validate application profiling.

After the deployment is complete and the application pod is deployed in Amazon EKS, the Amazon CodeGuru Profiler agent configured in the application will try to send profiling data of the application (CPU, heap summary, latency, and bottlenecks) to Amazon CodeGuru Profiler.

For the initial deployment of an application, Amazon CodeGuru Profiler takes about 15 minutes to visualize the profiling data.

AWS DevOps

Related resources

Additional information

CodeGuru Profiler should not be confused with the AWS X-Ray service in terms of functionality. CodeGuru Profiler is preferred for identifying the most expensive lines of codes, which might cause bottlenecks or security issues, and fix them before they become a potential risk. AWS X-Ray service is for application performance monitoring.

In this pattern, event rules are associated with the default event bus. If needed, you can extend the pattern to use a custom event bus.

This pattern uses CodeGuru Reviewer  as a static application security testing (SAST) tool for application code. You can also use this pipeline for other tools, such as SonarQube or Checkmarx. The corresponding scan setup instructions of any of these tools can be added in buildspec/buildspec_secscan.yaml, replacing the scan instructions of CodeGuru.