Help improve this page
To contribute to this user guide, choose the Edit this page on GitHub link that is located in the right pane of every page.
This page describes considerations for running AWS add-ons and Community add-ons on Amazon EKS Hybrid Nodes. To learn more about Amazon EKS add-ons and the processes for creating, upgrading, and removing add-ons from your cluster, see Amazon EKS add-ons. Unless otherwise noted on this page, the processes for creating, upgrading, and removing Amazon EKS add-ons is the same for Amazon EKS clusters with hybrid nodes as it is for Amazon EKS clusters with nodes running in AWS Cloud. Only the add-ons included on this page have been validated for compatibility with Amazon EKS Hybrid Nodes.
The following AWS Add-ons are compatible with Amazon EKS Hybrid Nodes.
AWS add-on | Compatible add-on versions |
---|---|
kube-proxy |
v1.25.14-eksbuild.2 and above |
CoreDNS |
v1.9.3-eksbuild.7 and above |
AWS Distro for OpenTelemetry (ADOT) |
v0.102.1-eksbuild.2 and above |
CloudWatch Observability Agent |
v2.2.1-eksbuild.1 and above |
EKS Pod Identity Agent |
v1.3.3-eksbuild.1 and above |
CSI snapshot controller |
v8.1.0-eksbuild.1 and above |
The following Community add-ons are compatible with Amazon EKS Hybrid Nodes. To learn more about support for Community add-ons, see Amazon EKS add-ons.
Community add-on | Compatible add-on versions |
---|---|
Kubernetes Metrics Server |
v0.7.2-eksbuild.1 and above |
In addition to the Amazon EKS add-ons in the tables above, the Amazon Managed Service for Prometheus Collector, and the AWS Load Balancer Controller for application ingress (HTTP) and load balancing (TCP/UDP) are compatible with hybrid nodes.
There are AWS add-ons and Community add-ons that aren’t compatible with Amazon EKS Hybrid Nodes. The latest versions of these add-ons have an anti-affinity rule for the default eks.amazonaws.com/compute-type: hybrid
label applied to hybrid nodes. This prevents them from running on hybrid nodes when deployed in your clusters. If you have clusters with both hybrid nodes and nodes running in AWS Cloud, you can deploy these add-ons in your cluster to nodes running in AWS Cloud. The Amazon VPC CNI is not compatible with hybrid nodes, and Cilium and Calico are supported as the Container Networking Interfaces (CNIs) for Amazon EKS Hybrid Nodes. See Configure a CNI for hybrid nodes for more information.
The rest of this page describes differences between running compatible Amazon EKS add-ons on hybrid nodes, compared to the other Amazon EKS compute types.
AWS Add-ons
kube-proxy and CoreDNS
EKS installs Kube-proxy and CoreDNS as self-managed add-ons by default when you create an EKS cluster with the AWS API and AWS SDKs, including from the AWS CLI. You can overwrite these add-ons as Amazon EKS add-ons after cluster creation. Reference the EKS documentation for details on Manage kube-proxy in Amazon EKS clusters and Manage CoreDNS for DNS in Amazon EKS clusters. If you are running a cluster with hybrid nodes and nodes in AWS Cloud, we recommend that you have at least one CoreDNS replica on hybrid nodes and at least one CoreDNS replica on your nodes in AWS Cloud.
CloudWatch Observability Agent add-on
As the CloudWatch Observability Agent runs webhooks
Node-level metrics are not available for hybrid nodes because CloudWatch Container Insights depends on the availability of Instance Metadata Service (IMDS) for node-level metrics. Cluster, workload, pod, and container-level metrics are available for hybrid nodes.
After installing the add-on by following the steps described in Install the CloudWatch agent with the Amazon CloudWatch Observability, the add-on manifest must be updated before the agent can run successfully on hybrid nodes. Edit the amazoncloudwatchagents
resource on the cluster to add the RUN_WITH_IRSA
environment variable as shown below.
kubectl edit amazoncloudwatchagents -n amazon-cloudwatch cloudwatch-agent
apiVersion: v1
items:
- apiVersion: cloudwatch.aws.amazon.com/v1alpha1
kind: AmazonCloudWatchAgent
metadata:
...
name: cloudwatch-agent
namespace: amazon-cloudwatch
...
spec:
...
env:
- name: RUN_WITH_IRSA # <-- Add this
value: "True" # <-- Add this
- name: K8S_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
...
Amazon Managed Prometheus managed collector for hybrid nodes
An Amazon Managed Service for Prometheus (AMP) managed collector consists of a scraper that discovers and collects metrics from the resources in an Amazon EKS cluster. AMP manages the scraper for you, removing the need to manage any instances, agents, or scrapers yourself.
You can use AMP managed collectors without any additional configuration specific to hybrid nodes. However the metric endpoints for your applications on the hybrid nodes must be reachable from the VPC, including routes from the VPC to remote pod network CIDRs and the ports open in your on-premises firewall. Additionally, your cluster must have private cluster endpoint access.
Follow the steps in Using an AWS managed collector in the Amazon Managed Service for Prometheus User Guide.
AWS Distro for OpenTelemetry (ADOT) add-on
You can use the AWS Distro for OpenTelemetry (ADOT) Amazon EKS add-on to collect metrics, logs, and tracing data from your applications running on hybrid nodes. ADOT uses admission webhooks
Follow the steps in Getting Started with AWS Distro for OpenTelemetry using EKS Add-Ons
AWS Load Balancer Controller
You can use the AWS Load Balancer Controller and Application Load Balancer (ALB) or Network Load Balancer (NLB) with the target type ip for workloads on hybrid nodes connected with AWS Direct Connect or AWS Site-to-Site VPN. As the AWS Load Balancer Controller uses webhooks
To install the AWS Load Balancer Controller, follow the steps at Install AWS Load Balancer Controller with Helm or Install AWS Load Balancer Controller with manifests.
For ingress with ALB, you must specify the annotations below. See Route application and HTTP traffic with Application Load Balancers for instructions.
alb.ingress.kubernetes.io/target-type: ip
For load balancing with NLB, you must specify the annotations below. See Route TCP and UDP traffic with Network Load Balancers for instructions.
service.beta.kubernetes.io/aws-load-balancer-type: "external"
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"
EKS Pod Identity Agent add-on
The original Amazon EKS Pod Identity Agent DaemonSet relies on the availability of EC2 IMDS on the node to obtain the required AWS credentials. As IMDS isn’t available on hybrid nodes, starting in add-on version 1.3.3-eksbuild.1
, the Pod Identity Agent add-on optionally deploys a second DaemonSet that specifically targets hybrid nodes. This DaemonSet mounts the required credentials to the pods created by the Pod Identity Agent add-on.
-
To use the Pod Identity agent on hybrid nodes, set
enableCredentialsFile: true
in the hybrid section ofnodeadm
config as shown below:apiVersion: node.eks.aws/v1alpha1 kind: NodeConfig spec: hybrid: enableCredentialsFile: true # <-- Add this
This will configure
nodeadm
to create a credentials file to be configured on the node under/eks-hybrid/.aws/credentials
, which will be used byeks-pod-identity-agent
pods. This credentials file will contain temporary AWS credentials that will be refreshed periodically. -
After you update the
nodeadm
config on each node, run the followingnodeadm init
command with yournodeConfig.yaml
to join your hybrid nodes to your Amazon EKS cluster. If your nodes have joined the cluster previous, still run theinit
command again.nodeadm init -c file://nodeConfig.yaml
-
Install
eks-pod-identity-agent
with support for hybrid nodes enabled, by either using the AWS CLI or AWS Management Console.-
AWS CLI: From the machine that you’re using to administer the cluster, run the following command to install
eks-pod-identity-agent
with support for hybrid nodes enabled. Replacemy-cluster
with the name of your cluster.aws eks create-addon \ --cluster-name my-cluster \ --addon-name eks-pod-identity-agent \ --configuration-values '{"daemonsets":{"hybrid":{"create": true}}}'
-
AWS Management Console: If you are installing the Pod Identity Agent add-on through the AWS console, add the following to the optional configuration to deploy the daemonset that targets hybrid nodes.
{"daemonsets":{"hybrid":{"create": true}}}
-
CSI snapshot controller add-on
Starting with version v8.1.0-eksbuild.2
, the CSI snapshot controller add-on applies a soft anti-affinity rule for hybrid nodes, preferring the controller deployment
to run on EC2 in the same AWS Region as the Amazon EKS control plane. Co-locating the deployment
in the same AWS Region as the Amazon EKS control plane improves latency.
Community add-ons
Kubernetes Metrics Server add-on
The control plane needs to reach Metrics Server’s pod IP (or node IP if hostNetwork is enabled). Therefore, unless you run Metrics Server in hostNetwork mode, you must configure a remote pod network when creating your Amazon EKS cluster, and you must make your pod IP addresses routable. Implementing Border Gateway Protocol (BGP) with the CNI is one common way to make your pod IP addresses routable.