Private clusters

This topic describes how to deploy a private cluster without outbound internet access. If you're not familiar with Amazon EKS networking, see De-mystifying cluster networking for Amazon EKS worker nodes.

Requirements

The following requirements must be met to run Amazon EKS in a private cluster without outbound internet access.

• A container image must be in or copied to Amazon Elastic Container Registry (Amazon ECR) or to a registry inside the VPC to be pulled. For more information, see Creating local copies of container images.

• Endpoint private access is required for nodes to register with the cluster endpoint. Endpoint public access is optional. For more information, see Amazon EKS cluster endpoint access control.

• You may need to include the VPC endpoints found at VPC endpoints for private clusters.

• You must include the following text to the bootstrap arguments when launching self-managed nodes. This text bypasses the Amazon EKS introspection and does not require access to the Amazon EKS API from within the VPC. Replace cluster-endpoint and cluster-certificate-authority with the values from your Amazon EKS cluster.

  --apiserver-endpoint cluster-endpoint --b64-cluster-ca cluster-certificate-authority

• The aws-auth ConfigMap must be created from within the VPC. For more information about create the aws-auth ConfigMap, see Managing users or IAM roles for your cluster.

Considerations

Here are some things to consider when running Amazon EKS in a private cluster without outbound internet access.

• eksctl is not supported with private clusters.

• AWS X-Ray is not supported with private clusters.

• Amazon CloudWatch Logs is supported with private clusters, but you must use an Amazon CloudWatch Logs VPC endpoint. For more information, see VPC endpoints for private clusters.

• Self-managed and managed nodes are supported. The instances for nodes must have access to the VPC endpoints. If you create a managed node group, the VPC endpoint security group must allow the CIDR for the subnets, or you must add the created node security group to the VPC endpoint security group.

• IAM roles for service accounts is supported. You must include the STS VPC endpoint. For more information, see VPC endpoints for private clusters.

• The Amazon EBS CSI driver is supported. Before deploying, the kustomization.yaml file must be changed to set the container images to use the same Region as the Amazon EKS cluster.

• The Amazon EFS CSI driver is supported. Before deploying, the kustomization.yaml file must be changed to set the container images to use the same Region as the Amazon EKS cluster.

• The Amazon FSx for Lustre CSI driver is not supported.

• The ALB Ingress Controller on Amazon EKS does not work in private clusters.

• AWS Fargate is supported with private clusters. You must include the STS VPC endpoint. For more information, see VPC endpoints for private clusters. You must use a third-party ingress controller with AWS Fargate because the ALB Ingress Controller on Amazon EKS does not work in private clusters and because Classic Load Balancers and Network Load Balancers are not supported on pods running on Fargate.

• App Mesh is supported with private clusters when you use the App Mesh Envoy VPC endpoint. For more information, see VPC endpoints for private clusters.

  • The App Mesh sidecar injector for Kubernetes is supported. For more information, see App Mesh sidecar injector on GitHub.

  • The App Mesh controller for Kubernetes is not supported. For more information, see App Mesh controller on GitHub.

Creating local copies of container images

Because a private cluster has no outbound internet access, container images cannot be pulled from external sources such as Docker Hub. Instead, container images must be copied locally to Amazon ECR or to an alternative registry accessible in the VPC. A container image can be copied to Amazon ECR from outside the private VPC. The private cluster accesses the Amazon ECR repository using the Amazon ECR VPC endpoints. You must have Docker and the AWS CLI installed on the workstation that you use to create the local copy.

To create a local copy of a container image

1. Create an Amazon ECR repository. For more information, see Creating a repository.

2. Pull the container image from the external registry using docker pull.

3. Tag your image with the Amazon ECR registry, repository, and optional image tag name combination using docker tag.

4. Authenticate to the registry. For more information, see Registry authentication.

5. Push the image to Amazon ECR using docker push.

   Note

   Be sure to update your resource configuration to use the new image location.

   The following example pulls the amazon/aws-node-termination-handler image, using tag v1.3.1-linux-amd64, from Docker Hub and creates a local copy in Amazon ECR.

   aws ecr create-repository --repository-name amazon/aws-node-termination-handler
   docker pull amazon/aws-node-termination-handler:v1.3.1-linux-amd64
   docker tag amazon/aws-node-termination-handler 111122223333.dkr.ecr.region-code.amazonaws.com/amazon/aws-node-termination-handler:v1.3.1-linux-amd64
   aws ecr get-login-password --region region-code | docker login --username AWS --password-stdin 111122223333.dkr.ecr.region-code.amazonaws.com
   docker push 111122223333.dkr.ecr.region-code.amazonaws.com/amazon/aws-node-termination-handler:v1.3.1-linux-amd64

VPC endpoints for private clusters

The following VPC endpoints may be required.

• com.amazonaws.region.ec2

• com.amazonaws.region.ecr.api

• com.amazonaws.region.ecr.dkr

• com.amazonaws.region.s3 – For pulling container images

• com.amazonaws.region.logs – For CloudWatch Logs

• com.amazonaws.region.sts – If using AWS Fargate or IAM roles for service accounts

• com.amazonaws.region.elasticloadbalancing – If using Application Load Balancers

• com.amazonaws.region.autoscaling – If using Cluster Autoscaler

• com.amazonaws.region.appmesh-envoy-management – If using App Mesh