Activate mTLS in AWS App Mesh using AWS Private CA on Amazon EKS

Created by Omar Kahil (AWS), Emmanuel Saliu (AWS), Muhammad Shahzad (AWS), and Andy Wong (AWS)

Environment: PoC or pilot

Technologies: Containers & microservices

AWS services: AWS App Mesh; Amazon EKS; AWS Certificate Manager (ACM)

Summary

This pattern shows how to implement Mutual Transport Layer Security (mTLS) on Amazon Web Services (AWS) using certificates from AWS Private Certificate Authority (AWS Private CA) in AWS App Mesh. It uses the Envoy secret discovery service (SDS) API through the Secure Production Identity Framework for Everyone (SPIFFE). SPIFFE is a Cloud Native Computing Foundation (CNCF) open-source project with wide community support that provides fine-grained and dynamic workload identity management. To implement SPIFFE standards, use the SPIRE SPIFFE runtime environment.

Using mTLS in App Mesh offers two-way peer authentication, because it adds a layer of security over TLS and allows services in the mesh to verify the client that’s making the connection. The client in the client-server relationship also provides an X.509 certificate during the session negotiation process. The server uses this certificate to identify and authenticate the client. This helps to verify if the certificate is issued by a trusted certificate authority (CA) and if the certificate is a valid one.

Prerequisites and limitations

Prerequisites 

• An Amazon Elastic Kubernetes Service (Amazon EKS) cluster with self-managed or managed node groups

• App Mesh controller deployed on the cluster with SDS activated

• A private certificate from AWS Certificate Manager (ACM) that is issued by AWS Private CA

Limitations 

• SPIRE cannot be installed on AWS Fargate because the SPIRE Agent must be run as a Kubernetes DaemonSet.

Product versions

• AWS App Mesh Controller chart 1.3.0 or later

Architecture

The following diagram shows the EKS cluster with App Mesh in the VPC. The SPIRE server in one worker node communicates with the SPIRE Agents in other worker nodes, and with AWS Private CA. Envoy is used for mTLS communication between the SPIRE Agent worker nodes.

The diagram illustrates the following steps:

1. Certificate is issued.

2. Request cert signing and certificate.

Tools

AWS services

• AWS Private CA – AWS Private Certificate Authority (AWS Private CA) enables creation of private certificate authority (CA) hierarchies, including root and subordinate CAs, without the investment and maintenance costs of operating an on-premises CA.

• AWS App Mesh – AWS App Mesh is a service mesh that makes it easier to monitor and control services. App Mesh standardizes how your services communicate, giving you consistent visibility and network traffic controls for every service in an application.

• Amazon EKS – Amazon Elastic Kubernetes Service (Amazon EKS) is a managed service that you can use to run Kubernetes on AWS without needing to install, operate, and maintain your own Kubernetes control plane or nodes.

Other tools

• Helm – Helm is a package manager for Kubernetes that helps you install and manage applications on your Kubernetes cluster. This pattern uses Helm to deploy AWS App Mesh Controller.

• AWS App Mesh Controller chart – AWS App Mesh Controller chart is used by this pattern to enable AWS App Mesh on Amazon EKS.

Epics

Set up the environment

Task	Description	Skills required
Set up App Mesh with Amazon EKS.	Follow base deployment steps that are provided in the repository.	DevOps engineer
Install SPIRE.	Install SPIRE on the EKS cluster by using spire_setup.yaml.	DevOps engineer
Install the AWS Private CA certificate.	Create and install a certificate for your private root CA by following the instructions in the AWS documentation.	DevOps engineer
Grant permissions to the cluster node instance role.	To attach policies to the cluster node instance role, use the code that’s in the Additional information section.	DevOps engineer
Add the SPIRE plugin for AWS Private CA.	To add the plugin to the SPIRE server configuration, use the code that’s in the Additional information section. Replace the certificate_authority_arn Amazon Resource Name (ARN) to your private CA ARN. The signing algorithm used must be the same as the signing algorithm on the private CA. Replace your_region with your AWS Region. For more information about the plugin, see Server plugin: UpstreamAuthority "aws_pca".	DevOps engineer
Update bundle.cert.	After you create the SPIRE server, a spire-bundle.yaml file will be created. Change the bundle.crt value in the spire-bundle.yaml file from the private CA to the public certificate.	DevOps engineer

Deploy and register the workloads

Task	Description	Skills required
Register node and workload entries with SPIRE.	To register node and workload (services) with SPIRE Server, use the code in the repository.	DevOps engineer
Create a mesh in App Mesh with mTLS activated.	Create a new mesh in App Mesh with all the components for your microservices application (for example, virtual service, virtual router, and virtual nodes).	DevOps engineer
Inspect the registered entries.	You can inspect the registered entries for your nodes and workloads by running the following command. kubectl exec -n spire spire-server-0 -- /opt/spire/bin/spire-server entry show This will show the entries for the SPIRE Agents.	DevOps engineer

Verify mTLS traffic

Task	Description	Skills required
Verify mTLS traffic.	From the frontend service, send an HTTP header to the backend service, and verify a successful response with the services that are registered in SPIRE. For mutual TLS authentication, you can inspect the ssl.handshake statistic by running the following command. kubectl exec -it $POD -n $NAMESPACE -c envoy -- curl http://localhost:9901/stats | grep ssl.handshake After running the previous command, you should see the listener ssl.handshake count, which will look similar to the following example: listener.0.0.0.0_15000.ssl.handshake: 2	DevOps engineer
Verify that certificates are being issued from AWS Private CA.	You can check that the plugins have been configured correctly and certificates are being issued from your upstream private CA by viewing the logs in your SPIRE server. Run the following command. kubectl logs spire-server-0 -n spire Then view the logs that are produced. This code assumes that your server is named spire-server-0 and is hosted in your spire namespace. You should see successful loading of the plugins and a connection being made to your upstream private CA.	DevOps engineer

Related resources

• Using mTLS with SPIFFE/SPIRE in AWS App Mesh on Amazon EKS 

• Enabling mTLS in AWS App Mesh using SPIFFE/SPIRE in a multi-account Amazon EKS environment 

• Walkthrough used in this pattern 

• Server plugin: UpstreamAuthority "aws_pca" 

• Quickstart for Kubernetes

Additional information

Attach permissions to the cluster node instance role

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ACMPCASigning",
            "Effect": "Allow",
            "Action": [
                "acm-pca:DescribeCertificateAuthority",
                "acm-pca:IssueCertificate",
                "acm-pca:GetCertificate",
                "acm:ExportCertificate"
            ],
            "Resource": "*"
        }
    ]
}
AWS Managed Policy: "AWSAppMeshEnvoyAccess"

Add the SPIRE plugin for ACM

Add the SPIRE plugin for ACM
Change certificate_authority_arn to your PCA ARN. The signing algorithm used must be the same as the signing algorithm on the PCA. Change your_region to the appropriate AWS Region.
UpstreamAuthority "aws_pca" {
    plugin_data {
        region = "your_region"
        certificate_authority_arn = "arn:aws:acm-pca:...."
        signing_algorithm = "your_signing_algorithm"
    }
}