EFA DRA driver vs. EFA device plugin EFA and EFA-only interfaces Choosing an EFA-enabled instance type Creating EKS nodes with EFA interfaces EKS Auto Mode Karpenter EKS managed node groups and self-managed nodes Using EKS-optimized AMIs with EFA Conserving IP address allocation Install the EFA DRA driver (DRANET)Create the EFA DeviceClass Topology-aware EFA and GPU/Neuron device allocation EFA device sharing between Pods is not supported Install the EFA Kubernetes device plugin Test EFA performance with NCCL tests

Manage EFA devices on Amazon EKS

Elastic Fabric Adapter (EFA) is a network device for Amazon EC2 instances that enables high-performance inter-node communication and RDMA (Remote Direct Memory Access) for artificial intelligence, machine learning, and High Performance Computing (HPC) workloads. Amazon EKS supports two mechanisms for managing EFA devices in EKS clusters: the EFA Dynamic Resource Allocation (DRA) driver (DRANET) and the EFA device plugin.

EFA DRA driver vs. EFA device plugin

Feature	EFA DRA driver	EFA device plugin
Minimum Kubernetes version	1.34	All EKS-supported Kubernetes versions
EKS Compute	EKS Auto Mode (Kubernetes 1.36 and later), Karpenter, managed node groups, self-managed nodes	EKS Auto Mode, Karpenter, managed node groups, self-managed nodes
EKS-optimized AMIs	AL2023, Bottlerocket	AL2023, Bottlerocket
Device advertisement	Rich attributes via `ResourceSlice` objects including device type, topology, and PCIe locality	Integer count of `vpc.amazonaws.com/efa` extended resources
GPU-EFA affinity	DRA-native topology-awareness	Automatic topology-awareness (EKS-optimized AL2023 AMIs only)
Neuron-EFA affinity	DRA-native topology-awareness	Automatic topology-awareness (EKS-optimized AL2023 AMIs only)
Device sharing	Not supported. Each EFA device is exclusively allocated to one Pod. For more information, see EFA device sharing between Pods is not supported.	Not supported. Each EFA device is exclusively allocated to one Pod.

We recommend using DRA drivers for new deployments with Kubernetes versions 1.34 and later. With the EFA DRA driver, you can configure topology-aware allocation that pairs EFA interfaces with their topologically-local GPUs or Neuron devices.

EKS Auto Mode includes the EFA DRA driver in the node runtime on Kubernetes version 1.36 and later, so you do not install or manage it. For more information, see Using DRA with EKS Auto Mode. With Karpenter, EKS managed node groups, or self-managed nodes, you install the EFA DRA driver yourself. Karpenter supports DRA with both dynamic and static capacity provisioning as of Karpenter version 1.15. Dynamic provisioning requires enabling Karpenter feature gates. For more information, see Using DRA with Karpenter.

With EFA, High Performance Computing (HPC) applications using the Message Passing Interface (MPI) and machine learning applications using the NVIDIA Collective Communications Library (NCCL) can scale to thousands of CPUs or GPUs, which gives you the application performance of an on-premises HPC cluster with the elasticity of the AWS Cloud. Using EFA with Pods running on Amazon EKS can reduce the time to complete large-scale distributed training workloads without adding instances to your cluster. For more information about EFA, see Elastic Fabric Adapter.

EFA and EFA-only interfaces

An Elastic Fabric Adapter (EFA) is a network interface that combines the capabilities of an Elastic Network Adapter (ENA) and an OS-bypass interface, powered by the AWS Scalable Reliable Datagram (SRD) protocol. The OS-bypass interface lets applications communicate directly with the hardware for low-latency transport, while the ENA capabilities carry normal IP traffic.

Amazon EC2 offers two interface types for EFA:

efa : Provides both the ENA and OS-bypass capabilities on one interface. The interface receives an IP address from your subnet.
efa-only : Provides only the OS-bypass capability. The interface does not receive an IP address and communication is limited to interfaces within the same Availability Zone.

We recommend efa-only interfaces for all interfaces except the primary network interface. Distributed training and HPC traffic uses the OS-bypass path, not the IP path, so the IP addresses that efa interfaces consume go unused. On an instance such as p5.48xlarge or p6-b200.48xlarge with dozens of network interfaces, those unused addresses can exhaust a subnet’s available IP space. For more information, see Conserving IP address allocation.

Note the following requirements for efa-only interfaces:

You cannot set network card 0 to efa-only, because it is the primary network card and network interface of the instance. Configure network card 0, device index 0 as interface and use efa-only for the remaining interfaces.
Amazon VPC CNI version 1.18.5 or later is required. For more information about updating the VPC CNI, see Assign IPs to Pods with the Amazon VPC CNI.
You cannot use eksctl’s `efaEnabled setting to create nodes with efa-only interfaces, because efaEnabled configures every interface as type efa. Use a NodeClass, an EC2NodeClass, or a launch template instead, as shown in the examples that follow.

Choosing an EFA-enabled instance type

EFA is supported on a specific set of Amazon EC2 instance types. For the full list, see Supported instance types in the Amazon EC2 User Guide. To run machine learning workloads, we recommend an instance type that also has hardware acceleration chips such as NVIDIA GPUs or AWS Trainium chips. For the list of instance types that have both accelerators and EFA, see Accelerated computing in the Amazon EC2 User Guide.

As you compare instance types, consider the number of EFA network cards available alongside the number of accelerator cards, vCPUs, and memory. You can assign up to one EFA per network card, and an EFA counts as a network interface. To see how many EFAs each instance type supports, see the Network cards list in the Amazon EC2 User Guide. For the recommended interface layout to reach maximum network bandwidth on each instance type, see Maximize network bandwidth for EFA-enabled instance types in the Amazon EC2 User Guide.

To find which EFA-enabled instance types are available in an AWS Region, and which Availability Zones offer them, run the following commands. Replace region-code with your AWS Region.


aws ec2 describe-instance-types --region [.replaceable]`region-code` \
    --filters Name=network-info.efa-supported,Values=true \
    --query "InstanceTypes[*].[InstanceType]" --output text


aws ec2 describe-instance-type-offerings --region [.replaceable]`region-code` \
    --location-type availability-zone \
    --filters Name=instance-type,Values=[.replaceable]`p5.48xlarge` \
    --query 'InstanceTypeOfferings[*].Location' --output text

An example output is as follows.


us-west-2a    us-west-2c    us-west-2b

Your VPC must have subnets with available IP addresses in one of the Availability Zones returned. Because efa-only interfaces communicate only within an Availability Zone, place the nodes of a single distributed job in one Availability Zone.

EFA requires hugepages

EFA requires hugepages. The EFA installer, which is included in the EKS-optimized AL2023 accelerated AMIs and the Bottlerocket AMIs, reserves 2 MiB hugepages when a node boots. You request them as hugepages-2Mi resources in your workload specifications, and the examples in this topic request hugepages alongside EFA devices.

Note the following about this reservation:

It requires an EFA device, not just the EFA software. The reservation runs only on instances that have an EFA device attached. An accelerated node that has the EFA installer but no EFA interface reserves no hugepages, and a Pod that requests hugepages-2Mi on such a node stays in the Pending state.
The amount scales with the size of the instance. The reservation covers the packet pools that the libfabric EFA provider creates for each endpoint, so it grows with the number of vCPUs. A c5n.18xlarge reserves roughly 15 GiB and a p5.48xlarge roughly 41 GiB. Do not assume a fixed amount.
It only ever increases the reservation. If a node already has more 2 MiB hugepages reserved, for example from node user data, the EFA installer leaves that reservation in place.

To see what a node actually reserved, check its capacity:


kubectl get nodes -o custom-columns='NAME:.metadata.name,2Mi:.status.capacity.hugepages-2Mi'

Size your Pod requests against what your nodes report, rather than against a fixed figure. For more information about configuring hugepages on EKS nodes, see Configure hugepages in Amazon EKS clusters, and for the Kubernetes resource model, see Manage HugePages in the Kubernetes documentation.

Creating EKS nodes with EFA interfaces

When you create EKS nodes with EFA interfaces, the EFA interfaces are attached to the instance during instance provisioning. You can customize the per-device EFA configuration and use placement groups with EKS Auto Mode, Karpenter, EKS managed node groups, or EKS self-managed node groups. With EKS Auto Mode, you pass configuration for each network interface through the NodeClass under advancedNetworking.networkInterfaces. With Karpenter, you pass configuration for each network interface via the EC2NodeClass. With EKS managed node groups or self-managed nodes, you pass configuration for each network interface with launch templates.

When using eksctl for provisioning EKS nodes with the efaEnabled setting, all interfaces are configured with interface type EFA, an EFA-specific security group is created, and the EFA device plugin is installed on the cluster. If you need to customize the per-device EFA configuration when using eksctl, it is recommended to use `eksctl’s support for launch templates.

The following examples show how to configure NodeClass and launch templates with EFA interfaces. This is useful to customize the interfaces used for EFA vs standard IP-based traffic. For information on the number of EFA interfaces supported by each instance type and how to configure them for maximum network bandwidth, see Maximize network bandwidth for EFA-enabled instance types in the Amazon EC2 User Guide.

Dynamic capacity provisioning assumes all EFA interfaces are attached

When EKS Auto Mode or Karpenter uses dynamic capacity provisioning to launch a node for an EFA DRA claim, it chooses the instance type before the node exists, by evaluating the claim against device metadata for each candidate instance type. That metadata assumes the node is configured with an EFA interface on every EFA-capable network card the instance type supports, and it is what PCIe alignment constraints are evaluated against.

If a node is launched with fewer EFA interfaces than its instance type supports, EKS Auto Mode and Karpenter overestimate the EFA devices the node will offer, and can launch a node that cannot satisfy the claim. For EFA DRA workloads that use dynamic capacity provisioning, leave the network interface configuration unset and let EKS Auto Mode or Karpenter attach an EFA interface on every EFA-capable network card for you.

This assumption does not apply to static capacity provisioning, EKS managed node groups, or self-managed nodes. On those, the nodes exist before any claim is evaluated and the EFA DRA driver publishes ResourceSlice objects for the interfaces that are actually attached.

EKS Auto Mode

In EKS Auto Mode, you configure EFA network interfaces using the advancedNetworking.networkInterfaces field in the NodeClass (eks.amazonaws.com/v1). Each entry specifies a networkCardIndex, deviceIndex, and interfaceType. The interfaceType can be interface for standard network interfaces or efa-only for EFA interfaces dedicated to RDMA traffic without IP addresses assigned.

When networkInterfaces is configured, instances launched by the NodePool referencing the NodeClass use this configuration regardless of whether Pods request vpc.amazonaws.com/efa resources. EKS Auto Mode does not attach additional IPs, prefixes, or ENIs after instance launch for nodes with static network interface configuration — only the interfaces and IPs configured at launch are available to Pods.

This feature can be used with static capacity node pools to maintain pre-warmed, EFA-ready nodes for distributed training and inference workloads.

EFA nodes need a subnet with a route to the internet

Run EFA workloads on private subnets with a NAT gateway, or on subnets with the AWS PrivateLink endpoints your workloads need. Do not rely on a public subnet that assigns public IP addresses.

Amazon EC2 does not assign a public IP address to an instance whose launch specifies more than one network interface. An EFA-enabled node is launched with several interfaces, whether EKS Auto Mode attaches them for you or you configure more than one entry in advancedNetworking.networkInterfaces. Such a node receives no public IP address, even in a subnet that has MapPublicIpOnLaunch enabled. The node still joins the cluster over the Amazon EKS private endpoint, so it reports Ready, but it has no route to the internet. Image pulls then fail with a timeout and your Pods stay in the ImagePullBackOff or ErrImagePull state.

This applies to EFA workloads that use DRA and to EFA workloads that request the vpc.amazonaws.com/efa extended resource, because it depends only on how many network interfaces the node is launched with. It does not affect nodes that are launched with a single network interface, which do receive a public IP address.

Combining a custom interface layout with topology-aligned allocation

Don’t use dynamic capacity provisioning, static network interface configuration, and topology-aligned GPU and EFA allocation together. The behavior of this combination is undefined.

EKS Auto Mode still launches a node, but the Pod can then fail to schedule onto it and remain in the Pending state. You don’t get an error when the node is provisioned. This happens because the device metadata that EKS Auto Mode evaluates the claim against assumes an EFA interface on every EFA-capable network card. An interface list that configures fewer than that makes EKS Auto Mode overestimate the topologically-aligned devices the node will offer.

Static network interface configuration is intended for static capacity provisioning. If you need a custom interface layout together with topology-aligned allocation, use static capacity node pools. For more information, see Using DRA with EKS Auto Mode.

When you do not configure networkInterfaces and a Pod claims EFA devices through DRA, EKS Auto Mode attaches the EFA interfaces for you. EKS Auto Mode includes the EFA DRA driver in the node runtime on Kubernetes version 1.36 and later, so you do not install the driver described in Install the EFA DRA driver (DRANET). You do need to create the EFA DeviceClass yourself, because EKS Auto Mode runs the driver from the node runtime rather than from the Helm chart that would otherwise create it. For the definition to apply, see DeviceClass objects on EKS Auto Mode.

The interfaces that EKS Auto Mode attaches for you are efa interfaces, so each one receives a private IP address from your subnet. Size your subnets for one private IP address per EFA-capable network card on every node you expect to launch this way.

To use the efa-only interfaces recommended in Conserving IP address allocation instead, configure advancedNetworking.networkInterfaces on the NodeClass and choose the interfaces attached at launch yourself, as described earlier in this section. The combination to avoid is dynamic capacity provisioning together with a custom interface list and topology-aligned GPU and EFA allocation.

You don’t select DRA rather than the EFA device plugin for a NodePool. With dynamic capacity provisioning, EKS Auto Mode infers which of the two a node uses from the workloads it launches the node for. To pin every node a NodePool launches to one of them, which static capacity provisioning requires, set the eks.amazonaws.com/device-manager-efa label to dra or device-plugin on the NodePool template. For more information, see Using DRA with EKS Auto Mode.


apiVersion: eks.amazonaws.com/v1
kind: NodeClass
metadata:
  name: efa-node-class
spec:
  role: MyNodeRole
  subnetSelectorTerms:
    - tags:
        Name: "private-subnet"
  securityGroupSelectorTerms:
    - tags:
        Name: "efa-security-group"
  placementGroupSelector:
    name: "ml-training-pg"
  advancedNetworking:
    networkInterfaces:
    - deviceIndex: 0
      interfaceType: interface
      networkCardIndex: 0
      secondaryIPv4PrefixCount: 1
    - networkCardIndex: 0
      deviceIndex: 1
      interfaceType: efa-only
    - networkCardIndex: 1
      deviceIndex: 0
      interfaceType: efa-only
    - networkCardIndex: 2
      deviceIndex: 0
      interfaceType: efa-only
    - networkCardIndex: 3
      deviceIndex: 0
      interfaceType: efa-only

For the full list of constraints on static network interface configuration, see Static Network Interface Configuration in the NodeClass documentation.

Karpenter

Each entry in networkInterfaces specifies a networkCardIndex, deviceIndex, and interfaceType. The interfaceType can be interface for standard network interfaces or efa-only for EFA interfaces that are dedicated to RDMA traffic and do not have IP addresses assigned. When networkInterfaces is configured, instances launched by the NodePool referencing the NodeClass use this configuration regardless of whether Pods request vpc.amazonaws.com/efa resources.

When using Karpenter without specifying networkInterfaces in your NodeClass, instances created for Pods requesting vpc.amazonaws.com/efa have all interfaces configured with interface type EFA.

The networkInterfaces configuration for EC2NodeClass was added in Karpenter v1.11. The following example shows an EC2NodeClass configured for a P6-B200 instance with 1 ENA interface and 8 EFA-only interfaces.

Combining a custom interface layout with topology-aligned allocation

Don’t use Karpenter dynamic capacity provisioning, static network interface configuration, and topology-aligned GPU and EFA allocation together. The behavior of this combination is undefined.

Karpenter still launches a node, but the Pod can then fail to schedule onto it and remain in the Pending state. You don’t get an error when the node is provisioned. This happens because the device metadata that Karpenter evaluates the claim against assumes an EFA interface on every EFA-capable network card. A spec.networkInterfaces list that configures fewer than that makes Karpenter overestimate the topologically-aligned devices the node will offer.

Static network interface configuration is intended for static capacity provisioning. If you use the EFA DRA driver and need a custom interface layout such as the efa-only configuration below, use Karpenter static capacity provisioning, EKS managed node groups, or self-managed nodes. For more information, see Using DRA with Karpenter.


apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: efa-node-class
spec:
  networkInterfaces:
  - networkCardIndex: 0
    deviceIndex: 0
    interfaceType: interface
  - networkCardIndex: 0
    deviceIndex: 1
    interfaceType: efa-only
  - networkCardIndex: 1
    deviceIndex: 0
    interfaceType: efa-only
  - networkCardIndex: 2
    deviceIndex: 0
    interfaceType: efa-only
  - networkCardIndex: 3
    deviceIndex: 0
    interfaceType: efa-only
  - networkCardIndex: 4
    deviceIndex: 0
    interfaceType: efa-only
  - networkCardIndex: 5
    deviceIndex: 0
    interfaceType: efa-only
  - networkCardIndex: 6
    deviceIndex: 0
    interfaceType: efa-only
  - networkCardIndex: 7
    deviceIndex: 0
    interfaceType: efa-only

EKS managed node groups and self-managed nodes

With EKS managed node groups or self-managed nodes, you pass configuration for each network interface with launch templates.

The following example shows a launch template configured for a P6-B200 instance with 1 ENA interface and 8 EFA-only interfaces. The primary network interface (network card 0, device index 0) uses a standard interface type for IP traffic, while additional interfaces use efa-only for dedicated RDMA traffic. Adjust the number of efa-only interfaces based on your instance type. For the number of EFA interfaces supported by each instance type, see Maximize network bandwidth for EFA-enabled instance types in the Amazon EC2 User Guide.

Replace security-group-id with your values. The security group must allow all inbound and outbound traffic to and from itself to enable EFA OS-bypass functionality. For more information, see Step 1: Prepare an EFA-enabled security group in the Amazon EC2 User Guide.

Important

Do not specify SubnetId in the launch template when using EKS managed node groups. EKS requires that all subnets are specified through the CreateNodegroup API and rejects launch templates that include subnet configuration.


{
  "LaunchTemplateName": "efa-launch-template",
  "LaunchTemplateData": {
    "InstanceType": "p6-b200.48xlarge",
    "NetworkInterfaces": [
      {
        "NetworkCardIndex": 0,
        "DeviceIndex": 0,
        "InterfaceType": "interface",
        "Groups": ["security-group-id"]
      },
      {
        "NetworkCardIndex": 0,
        "DeviceIndex": 1,
        "InterfaceType": "efa-only",
        "Groups": ["security-group-id"]
      },
      {
        "NetworkCardIndex": 1,
        "DeviceIndex": 0,
        "InterfaceType": "efa-only",
        "Groups": ["security-group-id"]
      },
      {
        "NetworkCardIndex": 2,
        "DeviceIndex": 0,
        "InterfaceType": "efa-only",
        "Groups": ["security-group-id"]
      },
      {
        "NetworkCardIndex": 3,
        "DeviceIndex": 0,
        "InterfaceType": "efa-only",
        "Groups": ["security-group-id"]
      },
      {
        "NetworkCardIndex": 4,
        "DeviceIndex": 0,
        "InterfaceType": "efa-only",
        "Groups": ["security-group-id"]
      },
      {
        "NetworkCardIndex": 5,
        "DeviceIndex": 0,
        "InterfaceType": "efa-only",
        "Groups": ["security-group-id"]
      },
      {
        "NetworkCardIndex": 6,
        "DeviceIndex": 0,
        "InterfaceType": "efa-only",
        "Groups": ["security-group-id"]
      },
      {
        "NetworkCardIndex": 7,
        "DeviceIndex": 0,
        "InterfaceType": "efa-only",
        "Groups": ["security-group-id"]
      }
    ]
  }
}

Using EKS-optimized AMIs with EFA

The EKS-optimized AL2023 AMIs and all Bottlerocket AMIs include the host-level components required to use EFA, specifically the components installed by the aws-efa-installer. The EKS AL2023 and Bottlerocket AMIs do not include the EFA DRA driver or EFA device plugin, and these must be installed separately on your cluster before deploying workloads.

Conserving IP address allocation

EFA-enabled instances such as p5.48xlarge and p6-b200.48xlarge support many network interfaces. By default, the Amazon VPC CNI allocates IP addresses across all IP-enabled attached ENIs, which can consume a large number of IP addresses from your subnet even when those addresses are not actively used by Pods. On instances with dozens of network interfaces, this can quickly exhaust your subnet’s available IP space.

To reduce IP address consumption on EFA-enabled nodes, configure your network interfaces to use efa-only for all interfaces except the primary. EFA-only interfaces are dedicated to RDMA traffic and do not have IP addresses assigned, so they do not consume addresses from your subnet. For example configurations, see Karpenter and EKS managed node groups and self-managed nodes. For the recommended interface layout for each instance type, see Maximize network bandwidth for EFA-enabled instance types in the Amazon EC2 User Guide.

In addition to using efa-only interfaces, you can configure the Amazon VPC CNI to limit the number of warm (pre-allocated) IP addresses and ENIs. By default, the VPC CNI pre-allocates a warm pool of ENIs and IP addresses for faster Pod startup, but on large instances this can reserve hundreds of unused IP addresses. Set the WARM_IP_TARGET and WARM_ENI_TARGET environment variables on the aws-node DaemonSet to control how many spare IP addresses and ENIs the CNI maintains. For more information on these settings, see Amazon VPC CNI best practices.

Note

The WARM_ENI_TARGET and WARM_IP_TARGET settings are cluster-wide and apply to all nodes managed by the VPC CNI. There is currently no way to set different values for each node group or instance type. If you need more granular control of these settings, provide feedback on containers-roadmap issue #1834 on GitHub.

Install the EFA DRA driver (DRANET)

The EFA DRA driver is built in the upstream DRANET project on GitHub, which provides cloud-aware network device management for Kubernetes DRA. EFA DRA driver and DRANET are used interchangeably throughout this documentation and refer to the same tool.

The EFA DRA driver advertises EFA devices as ResourceSlice objects with the driver name dra.net and the DeviceClass name efa.networking.k8s.aws. The EFA DRA driver runs as a DaemonSet on each node and automatically discovers EFA devices.

EKS Auto Mode includes this driver

Skip this section if you use EKS Auto Mode. EKS Auto Mode includes the EFA DRA driver in the node runtime on Kubernetes version 1.36 and later. You do need to create the DeviceClass object, as described in Create the EFA DeviceClass. For more information, see Using DRA with EKS Auto Mode.

Prerequisites

An Amazon EKS cluster running Kubernetes version 1.34 or later with static capacity provisioned by Karpenter, EKS managed node groups, or self-managed node groups.
Nodes with EFA-enabled Amazon EC2 instance types. For a list of supported instance types, see Supported instance types in the Amazon EC2 User Guide.
Nodes with host-level components installed for EFA, see Install the EFA software for more information. The EKS-optimized AL2023 NVIDIA and Neuron AMIs, and the Bottlerocket AMIs include the EFA host-level components.
Helm installed in your command-line environment, see the Setup Helm instructions for more information.
kubectl configured to communicate with your cluster, see Install or update kubectl for more information.

Procedure

Important

Do not install the EFA DRA driver on nodes where the EFA device plugin is running. The two mechanisms cannot coexist on the same node. Doing so can cause silent oversubscription of the underlying devices to multiple pods on the same node.

Add the EKS Helm chart repository.


helm repo add eks https://aws.github.io/eks-charts

Update your local Helm repository.
```
helm repo update
```
Install the EFA DRA driver on your cluster using Helm. The EFA DRA driver automatically detects that it is running on EC2 instances via the Instance Metadata Service (IMDS) and enables EFA device discovery. The EFA DRA driver is deployed as a DaemonSet in the kube-system namespace by default. See the Helm values.yaml in the EKS Helm chart repository on GitHub for the configurable parameters.
```
helm install aws-dranet eks/aws-dranet --namespace kube-system
```

Verify that the DRANET DaemonSet is running.


kubectl get daemonset -n kube-system aws-dranet


NAME          DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
aws-dranet    2         2         2       2            2           <none>          60s

Verify that the DeviceClass was created.


kubectl get deviceclass


NAME                    AGE
efa.networking.k8s.aws  60s

Verify that ResourceSlice objects are advertised for your nodes.
```
kubectl get resourceslices --field-selector spec.driver=dra.net
```
If you experience errors with the steps above, you can check the logs for DRANET with the following command.
```
kubectl logs -n kube-system -l app=aws-dranet
```

To request EFA devices using the DRA driver, create a ResourceClaim or ResourceClaimTemplate that references the EFA DeviceClass and reference it in your Pod specification. The following example requests a single EFA device.


apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: single-efa-claim
spec:
  spec:
    devices:
      requests:
      - name: efa
        exactly:
          deviceClassName: efa.networking.k8s.aws
          count: 1
---
apiVersion: v1
kind: Pod
metadata:
  name: efa-workload
spec:
  containers:
  - name: app
    ...
    resources:
      claims:
      - name: efa-device
  resourceClaims:
  - name: efa-device
    resourceClaimTemplateName: single-efa-claim

Create the EFA DeviceClass

A DeviceClass names a category of devices and defines the expression that selects them. Every ResourceClaim and ResourceClaimTemplate references one by name, so the EFA DeviceClass must exist before an EFA claim can be allocated. A claim that references a DeviceClass that does not exist is never allocated, and the Pod stays in the Pending state.

When you install the EFA DRA driver with Helm, as described in Install the EFA DRA driver (DRANET), the chart creates this object for you and there is nothing more to do. EKS Auto Mode runs the driver from the node runtime rather than from a Helm chart, so on EKS Auto Mode you create it yourself.


cat <<EOF | kubectl apply -f -
apiVersion: resource.k8s.io/v1
kind: DeviceClass
metadata:
  name: efa.networking.k8s.aws
spec:
  selectors:
  - cel:
      expression: |
        device.driver == "dra.net" &&
        device.attributes["dra.net"].pciDevice == 'Elastic Fabric Adapter (EFA)'
EOF

The EFA DRA driver advertises every network interface on the node, not only EFA devices, so the selector filters to EFA devices. Without that filter, a claim could be allocated a network interface that is not an EFA device.

Confirm that it exists.


kubectl get deviceclass

An example output is as follows.


NAME                     AGE
efa.networking.k8s.aws   5s

This definition matches the one the driver’s Helm chart installs, so the same claims work on EKS Auto Mode and on clusters where you manage the driver yourself. For more information about DRA on EKS Auto Mode, see Using DRA with EKS Auto Mode.

Topology-aware EFA and GPU/Neuron device allocation

The EFA DRA driver supports topology-aware allocation that pairs EFA interfaces with GPUs or Neuron devices on the same PCIe root. Use the matchAttribute constraint to align EFA and GPU or Neuron device allocations. To use this capability, you must also use the NVIDIA or Neuron DRA drivers. For more information, see Manage NVIDIA GPUs on Amazon EKS and Manage Neuron devices on Amazon EKS.

The following example requests 1 EFA interface aligned with 1 NVIDIA GPU:


apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: aligned-efa-nvidia
spec:
  spec:
    devices:
      requests:
      - name: 1-efa
        exactly:
          deviceClassName: efa.networking.k8s.aws
          count: 1
      - name: 1-gpu
        exactly:
          deviceClassName: gpu.nvidia.com
          count: 1
      constraints:
      - requests: ["1-gpu", "1-efa"]
        matchAttribute: "resource.kubernetes.io/pcieRoot"

The following example requests 4 EFA interfaces aligned with 4 Neuron devices:


apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: aligned-efa-neuron
spec:
  spec:
    devices:
      requests:
      - name: 4-neurons
        exactly:
          deviceClassName: neuron.aws.com
          count: 4
      - name: 4-efas
        exactly:
          deviceClassName: efa.networking.k8s.aws
          count: 4
      constraints:
      - requests: ["4-neurons", "4-efas"]
        matchAttribute: "resource.aws.com/devicegroup4_id"

The number in the devicegroup attribute name corresponds to the number of Neuron devices in the connected topology group. For example, resource.aws.com/devicegroup1_id identifies a single Neuron device, resource.aws.com/devicegroup4_id identifies a group of 4 connected devices, and resource.aws.com/devicegroup8_id and resource.aws.com/devicegroup16_id identify groups of 8 and 16 connected devices respectively. Choose the matchAttribute that matches the device count in your request so that the allocated Neuron devices and EFA interfaces belong to the same connected topology group. For more information on these attributes, see the Neuron DRA driver documentation.

You can use allocationMode to simplify how EFA devices are allocated to aligned GPU or Neuron accelerators. The allocationMode field supports two values: ExactCount (the default) requests a specific number of devices specified by count, and All requests all matching devices in a pool. For example, on p5.48xlarge instances there are four EFA devices that share the same PCIe root with one GPU. To allocate these groups of EFA devices with aligned GPUs, even if you do not know the exact EFA-GPU device mapping and count of aligned EFA devices, you can configure your ResourceClaimTemplate with allocationMode: All for the EFA devices.


apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: aligned-all-efa-one-nvidia
spec:
  spec:
    devices:
      requests:
      - name: all-efas
        exactly:
          deviceClassName: efa.networking.k8s.aws
          allocationMode: All
      - name: one-gpu
        exactly:
          deviceClassName: gpu.nvidia.com
          allocationMode: ExactCount
          count: 1
      constraints:
      - requests: ["all-efas", "one-gpu"]
        matchAttribute: "resource.kubernetes.io/pcieRoot"

Request an aligned claim from a Pod

A ResourceClaimTemplate only describes the devices to allocate. To use it, reference it from your Pod specification. List the template under resourceClaims with a name that is local to the Pod, then reference that local name from each container’s resources.claims so the container receives the allocated devices. Kubernetes generates a separate ResourceClaim for each Pod, so each Pod gets its own aligned set of EFA interfaces and GPUs.

The following example references the aligned-efa-nvidia template from the first example in this section, which allocates one EFA interface aligned with one NVIDIA GPU on the same PCIe root.


apiVersion: v1
kind: Pod
metadata:
  name: training-worker
spec:
  containers:
  - name: worker
    image: my-training-image
    resources:
      claims:
      - name: aligned-devices
      limits:
        hugepages-2Mi: 5120Mi
      requests:
        hugepages-2Mi: 5120Mi
  resourceClaims:
  - name: aligned-devices
    resourceClaimTemplateName: aligned-efa-nvidia
  tolerations:
  - key: nvidia.com/gpu
    operator: Exists
    effect: NoSchedule

Note the following about this specification:

resourceClaims names the template to use. The name field (aligned-devices) is an identifier local to this Pod, not the name of a cluster object. The resourceClaimTemplateName field is the name of the ResourceClaimTemplate object.
resources.claims gives the container access to the devices from that claim. A container that omits this does not receive the devices, even though the Pod holds the claim. If a Pod has several containers, list the claim in each container that needs the devices.
hugepages-2Mi is required because EFA transfers use hugepages. Adjust the amount to match your workload and the hugepages your nodes reserve, which depends on instance size. For more information, see Choosing an EFA-enabled instance type and Configure hugepages in Amazon EKS clusters.
The toleration is needed only if your GPU nodes carry a taint such as nvidia.com/gpu. Adjust or remove it to match your NodePool.
No extended resources are requested. Do not add vpc.amazonaws.com/efa or nvidia.com/gpu to resources.limits. Those are device plugin resources, and requesting them makes the Pod a device plugin Pod rather than a DRA Pod. For more information, see Manage hardware devices on Amazon EKS.

To use one of the other templates in this section, change resourceClaimTemplateName to that template’s name. For a Neuron-aligned claim, replace the GPU toleration with the taint your Neuron nodes use.

Give each Pod its own EFA devices. EFA devices cannot be shared between Pods. For more information, see EFA device sharing between Pods is not supported.

An EFA device is allocated to one Pod at a time. Unlike NVIDIA GPUs, which several Pods can share through a ResourceClaim that each of them references, EFA devices do not support this pattern. Use a ResourceClaimTemplate, which generates a separate claim and a separate set of devices for each Pod, as shown in Topology-aware EFA and GPU/Neuron device allocation.

This is a property of the EFA DRA driver rather than a limitation of DRA. The driver prepares a claim for one Pod only, and rejects a claim that is reserved for more than one Pod before it attaches any device.

If you create a ResourceClaim and reference it from several Pods with resourceClaimName, Kubernetes allows the reference and schedules every Pod onto the node where the devices were allocated. The driver then fails to prepare the claim and the kubelet reports a FailedPrepareDynamicResources event with the message driver only supports one pod per claim. Every Pod that references the claim stays in the ContainerCreating state, and none of them receive the devices. If you then reduce the Pods that reference the claim to one, the remaining Pod starts on the next retry.

A ResourceClaim referenced by a single Pod is allocated and prepared normally. Use a ResourceClaimTemplate instead when each Pod needs its own devices, so that a second Pod cannot reference a claim that is already in use.

Processes that need to share one EFA device must run in the same Pod. The driver prepares the claim once for the Pod, so list the claim in each container’s resources.claims and every container receives the same devices.

Sharing an RDMA device such as an EFA device between Pods is tracked upstream in dranet issue 63.

Install the EFA Kubernetes device plugin

The EFA Kubernetes device plugin advertises EFA devices as vpc.amazonaws.com/efa extended resources. You request EFA devices in container resource requests and limits.

Important

Topology-aligned allocation of NVIDIA GPUs or Neuron devices with EFA interfaces happens automatically when using the EKS-optimized AL2023 accelerated AMIs. This automatic alignment does not occur when using Bottlerocket EKS-optimized AMIs or custom AMIs. If you need topology-aligned accelerator and EFA device allocation with Bottlerocket or custom AMIs, use the EFA DRA driver and the corresponding Neuron DRA driver. To use the NVIDIA DRA driver on Bottlerocket, you must first disable the NVIDIA device plugin that is bundled with the Bottlerocket NVIDIA variants, which requires Bottlerocket version 1.63.0 or later. For more information, see Topology-aware EFA and GPU/Neuron device allocation and Install the NVIDIA DRA driver.

Important

Starting with NVIDIA k8s-device-plugin v0.19.0, the --mofed-enabled flag defaults to true, which causes the NVIDIA device plugin to mount all /dev/infiniband/uverbs* devices into containers requesting GPUs. This conflicts with the EFA device plugin, which should be the component managing EFA device allocation at /dev/infiniband. If you are using EKS managed node groups or self-managed nodes with the NVIDIA device plugin, you must explicitly disable MOFED. For instructions, see Install the NVIDIA Kubernetes device plugin.

EKS Auto Mode does not enable MOFED by default and is not affected by this issue.

Prerequisites

An Amazon EKS cluster.
Nodes with EFA-enabled Amazon EC2 instance types. For a list of supported instance types, see Supported instance types in the Amazon EC2 User Guide.
Nodes with host-level components installed for EFA, see Install the EFA software for more information. The EKS-optimized AL2023 AMIs and the Bottlerocket AMIs include the EFA host-level components.
Helm installed in your command-line environment, see the Setup Helm instructions for more information.
kubectl configured to communicate with your cluster, see Install or update kubectl for more information.

Procedure

Add the EKS Helm chart repository.


helm repo add eks https://aws.github.io/eks-charts

Update your local Helm repository.
```
helm repo update
```

Install the EFA device plugin.


helm install efa eks/aws-efa-k8s-device-plugin -n kube-system

Verify the EFA device plugin DaemonSet is running.


kubectl get daemonset -n kube-system efa-aws-efa-k8s-device-plugin


NAME                                  DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
efa-aws-efa-k8s-device-plugin         2         2         2       2            2           <none>          60s

Verify that your nodes have allocatable EFA resources.


kubectl get nodes "-o=custom-columns=NAME:.metadata.name,EFA:.status.allocatable.vpc\.amazonaws\.com/efa"


NAME                                           EFA
ip-192-168-11-225.us-west-2.compute.internal   4
ip-192-168-24-96.us-west-2.compute.internal    4

To request EFA devices using the device plugin, specify the vpc.amazonaws.com/efa resource in your container resource requests and limits.


apiVersion: v1
kind: Pod
metadata:
  name: efa-workload
spec:
  containers:
  - name: app
    ...
    resources:
      limits:
        vpc.amazonaws.com/efa: 4
        hugepages-2Mi: ...
      requests:
        vpc.amazonaws.com/efa: 4
        hugepages-2Mi: ...

Test EFA performance with NCCL tests

After you install either the EFA DRA driver or the EFA device plugin, we recommend validating the setup before you run production workloads. The NCCL tests measure collective-communication performance across nodes and confirm that GPUDirect RDMA over EFA is working. You can also find packaged versions of these tests in the awsome-distributed-training repository on GitHub.

The following procedure runs a two-node all_reduce_perf benchmark on p5.48xlarge instances. Each worker requests all eight GPUs, all EFA devices, and hugepages-2Mi on its node, so each worker consumes an entire instance. Adjust the instance type, GPU count, EFA count, and hugepages request to match your nodes.

Prerequisites

Two or more nodes with an EFA-enabled, GPU-accelerated instance type, in the same Availability Zone. For more information, see Choosing an EFA-enabled instance type.
Either the EFA DRA driver or the EFA device plugin installed. For more information, see Install the EFA DRA driver (DRANET) or Install the EFA Kubernetes device plugin.
NVIDIA GPUs exposed to Pods, through either the NVIDIA DRA driver or the NVIDIA device plugin. For more information, see Manage NVIDIA GPUs on Amazon EKS. Use the same mechanism for GPUs that you use for EFA: DRA claims for both, or device plugin resources for both.

Step 1: Deploy the Kubeflow MPI Operator

The NCCL tests use MPI to launch ranks across nodes. The Kubeflow MPI Operator runs Allreduce-style distributed jobs on Kubernetes through the MPIJob custom resource. For installation instructions, see MPI Operator on GitHub.

Step 2: Create the MPIJob manifest

Use the manifest that matches how you expose EFA and GPU devices. Copy it to a file named nccl-tests.yaml.

Using the EFA device plugin

With the device plugin, the worker requests vpc.amazonaws.com/efa and nvidia.com/gpu as extended resources.


apiVersion: kubeflow.org/v2beta1
kind: MPIJob
metadata:
  name: nccl-tests
spec:
  runPolicy:
    cleanPodPolicy: Running
    backoffLimit: 20
  slotsPerWorker: 8
  mpiReplicaSpecs:
    Launcher:
      replicas: 1
      template:
         spec:
          restartPolicy: OnFailure
          containers:
          - image: public.ecr.aws/hpc-cloud/nccl-tests:latest
            imagePullPolicy: IfNotPresent
            name: test-nccl-launcher
            env:
             - name: PATH
               value: $PATH:/opt/amazon/efa/bin:/usr/bin
            command:
            - /opt/amazon/openmpi/bin/mpirun
            - --allow-run-as-root
            - --tag-output
            - -np
            - "16"
            - -N
            - "8"
            - --bind-to
            - none
            - -x
            - PATH
            - -x
            - LD_LIBRARY_PATH
            - -x
            - NCCL_DEBUG=INFO
            - -x
            - NCCL_BUFFSIZE=8388608
            - -x
            - NCCL_P2P_NET_CHUNKSIZE=524288
            - -x
            - NCCL_TUNER_PLUGIN=/opt/amazon/ofi-nccl/lib/x86_64-linux-gnu/libnccl-ofi-tuner.so
            - --mca
            - pml
            - ^cm,ucx
            - --mca
            - btl
            - tcp,self
            - --mca
            - btl_tcp_if_exclude
            - lo,docker0,veth_def_agent
            - /opt/nccl-tests/build/all_reduce_perf
            - -b
            - "8"
            - -e
            - "16G"
            - -f
            - "2"
            - -g
            - "1"
            - -c
            - "1"
            - -n
            - "100"
    Worker:
      replicas: 2
      template:
        spec:
          nodeSelector:
            node.kubernetes.io/instance-type: "p5.48xlarge"
          containers:
          - image: public.ecr.aws/hpc-cloud/nccl-tests:latest
            imagePullPolicy: IfNotPresent
            name: nccl-tests-worker
            volumeMounts:
            - name: shmem
              mountPath: /dev/shm
            resources:
              limits:
                nvidia.com/gpu: 8
                hugepages-2Mi: 5120Mi
                vpc.amazonaws.com/efa: 32
                memory: 32000Mi
              requests:
                nvidia.com/gpu: 8
                hugepages-2Mi: 5120Mi
                vpc.amazonaws.com/efa: 32
                memory: 32000Mi
          volumes:
          - name: shmem
            hostPath:
              path: /dev/shm

The vpc.amazonaws.com/efa: 32 request claims all EFA devices on a p5.48xlarge. On the EKS-optimized AL2023 accelerated AMIs, the device plugin aligns the allocated EFA devices with their topologically-local GPUs automatically.

Using the EFA DRA driver

With DRA, the worker references a ResourceClaimTemplate instead of requesting extended resources. Because each worker in this test consumes a whole node, the template requests every EFA device and every GPU with allocationMode: All and omits a topology constraint — there is nothing to align when a single Pod receives all devices on the node. For a job whose Pods take only part of a node, add a matchAttribute constraint so that each Pod’s EFA devices and GPUs share a PCIe root. For more information, see Topology-aware EFA and GPU/Neuron device allocation.


apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: nccl-efa-gpus
spec:
  spec:
    devices:
      requests:
      - name: all-efas
        exactly:
          deviceClassName: efa.networking.k8s.aws
          allocationMode: All
      - name: all-gpus
        exactly:
          deviceClassName: gpu.nvidia.com
          allocationMode: All
---
apiVersion: kubeflow.org/v2beta1
kind: MPIJob
metadata:
  name: nccl-tests
spec:
  runPolicy:
    cleanPodPolicy: Running
    backoffLimit: 20
  slotsPerWorker: 8
  mpiReplicaSpecs:
    Launcher:
      replicas: 1
      template:
         spec:
          restartPolicy: OnFailure
          containers:
          - image: public.ecr.aws/hpc-cloud/nccl-tests:latest
            imagePullPolicy: IfNotPresent
            name: test-nccl-launcher
            env:
             - name: PATH
               value: $PATH:/opt/amazon/efa/bin:/usr/bin
            command:
            - /opt/amazon/openmpi/bin/mpirun
            - --allow-run-as-root
            - --tag-output
            - -np
            - "16"
            - -N
            - "8"
            - --bind-to
            - none
            - -x
            - PATH
            - -x
            - LD_LIBRARY_PATH
            - -x
            - NCCL_DEBUG=INFO
            - -x
            - NCCL_BUFFSIZE=8388608
            - -x
            - NCCL_P2P_NET_CHUNKSIZE=524288
            - -x
            - NCCL_TUNER_PLUGIN=/opt/amazon/ofi-nccl/lib/x86_64-linux-gnu/libnccl-ofi-tuner.so
            - --mca
            - pml
            - ^cm,ucx
            - --mca
            - btl
            - tcp,self
            - --mca
            - btl_tcp_if_exclude
            - lo,docker0,veth_def_agent
            - /opt/nccl-tests/build/all_reduce_perf
            - -b
            - "8"
            - -e
            - "16G"
            - -f
            - "2"
            - -g
            - "1"
            - -c
            - "1"
            - -n
            - "100"
    Worker:
      replicas: 2
      template:
        spec:
          nodeSelector:
            node.kubernetes.io/instance-type: "p5.48xlarge"
          containers:
          - image: public.ecr.aws/hpc-cloud/nccl-tests:latest
            imagePullPolicy: IfNotPresent
            name: nccl-tests-worker
            volumeMounts:
            - name: shmem
              mountPath: /dev/shm
            resources:
              claims:
              - name: efa-gpus
              limits:
                hugepages-2Mi: 5120Mi
                memory: 32000Mi
              requests:
                hugepages-2Mi: 5120Mi
                memory: 32000Mi
          resourceClaims:
          - name: efa-gpus
            resourceClaimTemplateName: nccl-efa-gpus
          volumes:
          - name: shmem
            hostPath:
              path: /dev/shm

Hugepages remain an extended resource under DRA, so the worker still requests hugepages-2Mi directly. Only the EFA and GPU devices move to the ResourceClaimTemplate.

Step 3: Run the test and read the results

Apply the manifest. This schedules the job onto two EFA-enabled nodes.
```
kubectl apply -f nccl-tests.yaml
```
An example output is as follows.
```
mpijob.kubeflow.org/nccl-tests created
```

Verify that the job started its Pods.


kubectl get pods

An example output is as follows.


NAME                          READY   STATUS     RESTARTS   AGE
nccl-tests-launcher-nbql9     0/1     Init:0/1   0          2m49s
nccl-tests-worker-0           1/1     Running    0          2m49s
nccl-tests-worker-1           1/1     Running    0          2m49s

The MPI Operator creates one launcher Pod and one worker Pod per node.

Follow the launcher log to watch the benchmark. Replace nbql9 with the suffix from your output.
```
kubectl logs -f nccl-tests-launcher-[.replaceable]`nbql9`
```
The all_reduce_perf output reports bus bandwidth for each message size. Confirm that bandwidth scales as the message size grows and that the NCCL_DEBUG=INFO output shows the ofi (libfabric) transport rather than a TCP fallback, which indicates NCCL is using EFA. If the test completes successfully, your cluster is ready for distributed training workloads that use NCCL.

Warning Javascript is disabled or is unavailable in your browser.

To use the Amazon Web Services Documentation, Javascript must be enabled. Please refer to your browser's Help pages for instructions.

Document Conventions

Device management

Neuron accelerators