CNI metrics helper - Amazon EKS

CNI metrics helper

The CNI metrics helper is a tool that you can use to scrape network interface and IP address information, aggregate metrics at the cluster level, and publish the metrics to Amazon CloudWatch.

When managing an Amazon EKS cluster, you may want to know how many IP addresses have been assigned and how many are available. The CNI metrics helper helps you to:

  • Track these metrics over time

  • Troubleshoot and diagnose issues related to IP assignment and reclamation

  • Provide insights for capacity planning

When a node is provisioned, the CNI plugin automatically allocates a pool of secondary IP addresses from the node’s subnet to the primary network interface (eth0). This pool of IP addresses is known as the warm pool, and its size is determined by the node’s instance type. For example, a c4.large instance can support three network interfaces and nine IP addresses per interface. The number of IP addresses available for a given pod is one less than the maximum (of ten) because one of the IP addresses is reserved for the elastic network interface itself. For more information, see IP Addresses Per Network Interface Per Instance Type in the Amazon EC2 User Guide for Linux Instances.

As the pool of IP addresses is depleted, the plugin automatically attaches another elastic network interface to the instance and allocates another set of secondary IP addresses to that interface. This process continues until the node can no longer support additional elastic network interfaces.

The following metrics are collected for your cluster and exported to CloudWatch:

  • The maximum number of network interfaces that the cluster can support

  • The number of network interfaces have been allocated to pods

  • The number of IP addresses currently assigned to pods

  • The total and maximum numbers of IP addresses available

  • The number of ipamD errors

Deploying the CNI metrics helper

The CNI metrics helper requires cloudwatch:PutMetricData permissions to send metric data to CloudWatch. It also uses the Amazon EC2 instance tag data to populate the CloudWatch metric namespace. This section helps you to create an IAM policy with those permissions, apply it to your node instance role, and then deploy the CNI metrics helper.

To create an IAM policy for the CNI metrics helper

  1. Create a file called allow_put_metrics_data.json and populate it with the following policy document.

    { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "cloudwatch:PutMetricData", "ec2:DescribeTags" ], "Resource": "*" } ] }
  2. Create an IAM policy called CNIMetricsHelperPolicy for your node instance profile that allows the CNI metrics helper to make calls to AWS APIs on your behalf. Use the following AWS CLI command to create the IAM policy in your AWS account.

    aws iam create-policy --policy-name CNIMetricsHelperPolicy \ --description "Grants permission to write metrics to CloudWatch" \ --policy-document file://allow_put_metrics_data.json

    Take note of the policy ARN that is returned.

  3. Get the IAM role name for your nodes. Use the following command to print the aws-auth configmap.

    kubectl -n kube-system describe configmap aws-auth


    Name: aws-auth Namespace: kube-system Labels: <none> Annotations: <none> Data ==== mapRoles: ---- - groups: - system:bootstrappers - system:nodes rolearn: arn:aws:iam::<111122223333>:role/<eksctl-prod-nodegroup-standard-wo-NodeInstanceRole-GKNS581EASPU> username: system:node:{{EC2PrivateDNSName}} Events: <none>

    Record the role name for any rolearn values that have the system:nodes group assigned to them. In the above example output, the role name is <eksctl-prod-nodegroup-standard-wo-NodeInstanceRole-GKNS581EASPU>. You should have one value for each node group in your cluster.

  4. Attach the new CNIMetricsHelperPolicy IAM policy to each of the node IAM roles you identified earlier with the following command, substituting the red text with your own AWS account number and node IAM role name.

    aws iam attach-role-policy \ --policy-arn arn:aws:iam::<111122223333>:policy/CNIMetricsHelperPolicy \ --role-name <eksctl-prod-nodegroup-standard-wo-NodeInstanceRole-GKNS581EASPU>

To deploy the CNI metrics helper

  • Apply the CNI metrics helper manifest.

    kubectl apply -f

Creating a metrics dashboard

After you have deployed the CNI metrics helper, you can view the CNI metrics in the CloudWatch console. This topic helps you to create a dashboard for viewing your cluster's CNI metrics.

To create a CNI metrics dashboard

  1. Open the CloudWatch console at

  2. In the left navigation, choose Metrics.

  3. Under Custom Namespaces, choose Kubernetes.

  4. Choose CLUSTER_ID.

  5. On the All metrics tab, select the metrics you want to add to the dashboard.

  6. Choose Actions, and then Add to dashboard.

  7. In the Select a dashboard section, choose Create new and enter a name for your dashboard, such as "EKS-CNI-metrics".

  8. In the Select a widget type section, choose Number.

  9. In the Customize the widget title section, enter a logical name for your dashboard title, such as "EKS CNI metrics".

  10. Choose Add to dashboard to finish. Now your CNI metrics are added to a dashboard that you can monitor, as shown below.

                        EKS CNI metrics