Private cluster requirements - Amazon EKS
RequirementsConsiderationsCreating local copies of container imagesAWS STS endpoints for IAM roles for service accounts

Private cluster requirements

This topic describes how to deploy an Amazon EKS private cluster without outbound internet access. If you're not familiar with Amazon EKS networking, see De-mystifying cluster networking for Amazon EKS worker nodes.

Requirements

The following requirements must be met to run Amazon EKS in a private cluster without outbound internet access.

  • A container image must be in or copied to Amazon Elastic Container Registry (Amazon ECR) or to a registry inside the VPC to be pulled. For more information, see Creating local copies of container images.

  • Endpoint private access is required for nodes to register with the cluster endpoint. Endpoint public access is optional. For more information, see Amazon EKS cluster endpoint access control.

  • For Linux and Windows nodes, you must include bootstrap arguments when launching self-managed nodes. This text bypasses the Amazon EKS introspection and doesn't require access to the Amazon EKS API from within the VPC. Replace api-server-endpoint and certificate-authority with the values from your Amazon EKS cluster.

    • For Linux nodes:

      --apiserver-endpoint api-server-endpoint --b64-cluster-ca certificate-authority

      For additional arguments, see the bootstrap script on GitHub.

    • For Windows nodes:

      -APIServerEndpoint api-server-endpoint -Base64ClusterCA certificate-authority

      For additional arguments, see Amazon EKS optimized Windows AMI.

  • The aws-auth ConfigMap must be created from within the VPC. For more information about create the aws-auth ConfigMap, see Enabling IAM user and role access to your cluster.

Considerations

Here are some things to consider when running Amazon EKS in a private cluster without outbound internet access.

  • Many AWS services support private clusters, but you must use a VPC endpoint. For more information, see VPC endpoints. Some commonly-used services and endpoints include:

    Service Endpoint
    Amazon Elastic Container Registry com.amazonaws.region-code.ecr.api and com.amazonaws.region-code.ecr.dkr and the Amazon S3 gateway endpoint
    Application Load Balancers and Network Load Balancers com.amazonaws.region-code.elasticloadbalancing
    AWS X-Ray com.amazonaws.region-code.xray
    Amazon CloudWatch Logs com.amazonaws.region-code.logs
    IAM roles for service accounts

    com.amazonaws.region-code.sts

    App Mesh

    • The App Mesh sidecar injector for Kubernetes is supported. For more information, see App Mesh sidecar injector on GitHub.

    • The App Mesh controller for Kubernetes isn't supported. For more information, see App Mesh controller on GitHub.

    • Cluster Autoscaler is supported. When deploying Cluster Autoscaler pods, make sure that the command line includes --aws-use-static-instance-list=true. For more information, see Use Static Instance List on GitHub. The worker node VPC must also include the STS VPC endpoint and autoscaling VPC endpoint.

    		 com.amazonaws.region-code.appmesh-envoy-management

  • Before deploying the Amazon EFS CSI driver , the kustomization.yaml file must be changed to set the container images to use the same AWS Region as the Amazon EKS cluster.

  • Self-managed and managed nodes are supported. The instances for nodes must have access to the VPC endpoints. If you create a managed node group, the VPC endpoint security group must allow the CIDR for the subnets, or you must add the created node security group to the VPC endpoint security group.

  • The Amazon FSx for Lustre CSI driver isn't supported.

  • AWS Fargate is supported with private clusters. You can use the AWS Load Balancer Controller to deploy AWS Application Load Balancers (ALBs) and Network Load Balancers with. The controller supports network load balancers with IP targets, which are required for use with Fargate. For more information, see Application load balancing on Amazon EKS and Create a network load balancer.

  • Installing the AWS Load Balancer Controller add-on is supported. However, while installing, you should use command line flags to set enable-shield, enable-waf, and enable-wafv2 to false. In addition, certificate discovery with hostnames from the Ingress objects isn't supported. This is because the controller needs to reach ACM, which doesn't have a VPC endpoint.

  • Some container software products use API calls that access the AWS Marketplace Metering service to monitor usage. Private clusters do not allow these calls, so these container types cannot be used for private clusters.

Creating local copies of container images

Because a private cluster has no outbound internet access, container images can't be pulled from external sources such as Docker Hub. Instead, container images must be copied locally to Amazon ECR or to an alternative registry accessible in the VPC. A container image can be copied to Amazon ECR from outside the private VPC. The private cluster accesses the Amazon ECR repository using the Amazon ECR VPC endpoints. You must have Docker and the AWS CLI installed on the workstation that you use to create the local copy.

To create a local copy of a container image

  1. Create an Amazon ECR repository. For more information, see Creating a repository.

  2. Pull the container image from the external registry using docker pull.

  3. Tag your image with the Amazon ECR registry, repository, and the optional image tag name combination using docker tag.

  4. Authenticate to the registry. For more information, see Registry authentication.

  5. Push the image to Amazon ECR using docker push.

    Note

    Make sure to update your resource configuration to use the new image location.

    The following example pulls the amazon/aws-node-termination-handler image, using tag v1.3.1-linux-amd64, from Docker Hub and creates a local copy in Amazon ECR.

    aws ecr create-repository --repository-name amazon/aws-node-termination-handler
docker pull amazon/aws-node-termination-handler:v1.3.1-linux-amd64
docker tag amazon/aws-node-termination-handler 111122223333.dkr.ecr.region-code.amazonaws.com/amazon/aws-node-termination-handler:v1.3.1-linux-amd64
aws ecr get-login-password --region region-code | docker login --username AWS --password-stdin 111122223333.dkr.ecr.region-code.amazonaws.com
docker push 111122223333.dkr.ecr.region-code.amazonaws.com/amazon/aws-node-termination-handler:v1.3.1-linux-amd64

AWS STS endpoints for IAM roles for service accounts

Pods configured with IAM roles for service accounts acquire credentials from an AWS Security Token Service (AWS STS) API call. If there is no outbound internet access, you must create and use an AWS STS VPC endpoint in your VPC. Most AWS v1 SDKs use the global AWS STS endpoint by default (sts.amazonaws.com), which doesn't use the AWS STS VPC endpoint. To use the AWS STS VPC endpoint, you may need to configure the SDK to use the regional AWS STS endpoint (sts.region-code.amazonaws.com). You can do this by setting the AWS_STS_REGIONAL_ENDPOINTS environment variable with a value of regional, along with the AWS Region.

For example, in a pod spec:

...
containers:
- env:
- name: AWS_REGION
value: region-code
    - name: AWS_STS_REGIONAL_ENDPOINTS
    value: regional
    ...
    ```

Replace region-code with the AWS Region that your cluster is in.