Configure common add-ons for hybrid nodes

This page describes considerations for running Amazon EKS add-ons from AWS on Amazon EKS Hybrid Nodes. To learn more about the Amazon EKS add-ons from AWS and the processes for creating, upgrading, and removing add-ons from your cluster, see Amazon EKS add-ons. The processes for creating, upgrading, and removing Amazon EKS add-ons is the same for Amazon EKS clusters with hybrid nodes as it is for Amazon EKS clusters with nodes running in AWS Cloud unless otherwise noted on this page.

The following Amazon EKS add-ons from AWS are compatible with Amazon EKS Hybrid Nodes.

EKS add-on	Compatible add-on versions
kube-proxy	v1.25.14-eksbuild.2 and above
CoreDNS	v1.9.3-eksbuild.7 and above
AWS Distro for OpenTelemetry (ADOT)	v0.102.1-eksbuild.2 and above
CloudWatch Observability Agent	v2.2.1-eksbuild.1 and above
EKS Pod Identity Agent	v1.3.3-eksbuild.1 and above
CSI snapshot controller	v8.1.0-eksbuild.1 and above

In addition to the Amazon EKS add-ons in the table above, the Amazon Managed Service for Prometheus Collector, and the AWS Load Balancer Controller for application ingress (HTTP) and load balancing (TCP/UDP) are compatible with hybrid nodes.

Amazon EKS add-ons from AWS that are not compatible with Amazon EKS Hybrid Nodes have been updated with an affinity rule for the default eks.amazonaws.com/compute-type: hybrid label applied to hybrid nodes. This prevents them from running on hybrid nodes when deployed in your clusters. If you have clusters with both hybrid nodes and nodes running in AWS Cloud, Amazon EKS add-ons that are not compatible with hybrid nodes can still be deployed in your cluster to nodes running in AWS Cloud. The Amazon VPC CNI is not compatible with hybrid nodes, and Cilium and Calico are supported as the Container Networking Interfaces (CNIs) for Amazon EKS Hybrid Nodes. See Configure a CNI for hybrid nodes for more information.

The rest of this page describes differences between running compatible Amazon EKS add-ons from AWS on hybrid nodes, compared to the other Amazon EKS compute types.

kube-proxy and CoreDNS

Kube-proxy and CoreDNS are installed as unmanaged add-ons by default when an EKS cluster is created. These add-ons can be managed as Amazon EKS add-ons after cluster creation. Reference the EKS documentation for details on Manage kube-proxy in Amazon EKS clusters and Manage CoreDNS for DNS in Amazon EKS clusters. If you are running a cluster with hybrid nodes and nodes in AWS Cloud, it is recommended to have at least one CoreDNS replica on hybrid nodes and at least one CoreDNS replica on your nodes in AWS Cloud.

CloudWatch Observability Agent add-on

Node-level metrics are not available for hybrid nodes because CloudWatch Container Insights depends on the availability of Instance Metadata Service (IMDS) for node-level metrics. Cluster, workload, pod, and container-level metrics are available for hybrid nodes.

After installing the add-on by following the steps described in Install the CloudWatch agent with the Amazon CloudWatch Observability, the add-on manifest must be updated before the agent can run successfully on hybrid nodes. Edit the amazoncloudwatchagents resource on the cluster to add the RUN_WITH_IRSA environment variable as shown below.

kubectl edit amazoncloudwatchagents -n amazon-cloudwatch cloudwatch-agent

apiVersion: v1
items:
- apiVersion: cloudwatch.aws.amazon.com/v1alpha1
  kind: AmazonCloudWatchAgent
  metadata:
    ...
    name: cloudwatch-agent
    namespace: amazon-cloudwatch
    ...
  spec:
    ...
    env:
    - name: RUN_WITH_IRSA # <-- Add this
      value: "True" # <-- Add this
    - name: K8S_NODE_NAME
      valueFrom:
        fieldRef:
          fieldPath: spec.nodeName
          ...

Amazon Managed Prometheus managed collector for hybrid nodes

An Amazon Managed Service for Prometheus (AMP) managed collector consists of a scraper that discovers and collects metrics from the resources in an Amazon EKS cluster. AMP manages the scraper for you, removing the need to manage any instances, agents, or scrapers yourself.

You can use AMP managed collectors without any additional configuration specific to hybrid nodes. However the metric endpoints for your applications on the hybrid nodes must be reachable from the VPC, including routes from the VPC to remote pod network CIDRs and the ports open in your on-premises firewall. Additionally, your cluster must have private cluster endpoint access.

Follow the steps in Using an AWS managed collector in the Amazon Managed Service for Prometheus User Guide.

AWS Distro for OpenTelemetry (ADOT) add-on

You can use the AWS Distro for OpenTelemetry (ADOT) Amazon EKS add-on to collect metrics, logs, and tracing data from your applications running on hybrid nodes. Note, ADOT uses admission webhooks to mutate and validate the Collector Custom Resource requests and you must configure your remote pod network when creating your Amazon EKS cluster.

Follow the steps in Getting Started with AWS Distro for OpenTelemetry using EKS Add-Ons in the AWS Distro for OpenTelemetry documentation.

AWS Load Balancer Controller

You can use the AWS Load Balancer Controller and Application Load Balancer (ALB) or Network Load Balancer (NLB) with the target type ip for workloads on hybrid nodes connected with AWS Direct Connect or AWS Site-to-Site VPN. As the AWS Load Balancer Controller uses webhooks, you must configure your remote pod network when creating your Amazon EKS cluster.

To install the AWS Load Balancer Controller, follow the steps at Install AWS Load Balancer Controller with Helm or Install AWS Load Balancer Controller with manifests.

For ingress with ALB, you must specify the annotations below. See Route application and HTTP traffic with Application Load Balancers for instructions.

alb.ingress.kubernetes.io/scheme: internal
alb.ingress.kubernetes.io/target-type: ip

For load balancing with NLB, you must specify the annotations below. See Route TCP and UDP traffic with Network Load Balancers for instructions.

service.beta.kubernetes.io/aws-load-balancer-type: "external"
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"

EKS Pod Identity Agent add-on

The original Amazon EKS Pod Identity Agent DaemonSet relies on the availability of EC2 IMDS on the node to obtain the required AWS credentials. As IMDS isn’t available on hybrid nodes, starting in add-on version 1.3.3-eksbuild.1, the Pod Identity Agent add-on optionally deploys a second DaemonSet that specifically targets hybrid nodes. This DaemonSet mounts the required credentials to the pods created by the Pod Identity Agent add-on.

1. To use the Pod Identity agent on hybrid nodes, set enableCredentialsFile: true in the hybrid section of nodeadm config as shown below:

   apiVersion: node.eks.aws/v1alpha1
   kind: NodeConfig
   spec:
       hybrid:
           enableCredentialsFile: true # <-- Add this

   This will configure nodeadm to create a credentials file to be configured on the node under /eks-hybrid/.aws/credentials, which will be used by eks-pod-identity-agent pods. This credentials file will contain temporary AWS credentials that will be refreshed periodically.

2. After you update the nodeadm config on each node, run the following nodeadm init command with your nodeConfig.yaml to join your hybrid nodes to your Amazon EKS cluster. If your nodes have joined the cluster previous, still run the init command again.

   nodeadm init -c file://nodeConfig.yaml

3. Install eks-pod-identity-agent with support for hybrid nodes enabled, by either using the AWS CLI or AWS Management Console.

   1. AWS CLI: From the machine that you’re using to administer the cluster, run the following command to install eks-pod-identity-agent with support for hybrid nodes enabled.

      aws eks create-addon \
          --cluster-name cluster-name \
          --addon-name eks-pod-identity-agent \
          --configuration-values '{"daemonsets":{"hybrid":{"create": true}}}'

   2. AWS Management Console: If you are installing the Pod Identity Agent add-on through the AWS console, add the following to the optional configuration to deploy the daemonset that targets hybrid nodes.

      {"daemonsets":{"hybrid":{"create": true}}}

CSI snapshot controller add-on

Starting with version v8.1.0-eksbuild.2, the CSI snapshot controller add-on applies a soft anti-affinity rule for hybrid nodes, preferring the controller deployment to run on EC2 in the same AWS Region as the Amazon EKS control plane. Co-locating the deployment in the same AWS Region as the Amazon EKS control plane improves latency.

📝 Edit this page on GitHub