Maximize network bandwidth on accelerated computing instances with EFA
To maximize bandwidth on the following accelerated instance types, you can use multiple Elastic Fabric Adapter (EFA) interfaces.
-
P5 and P5e instances support up to 32 network cards, and can deliver up to 3200 Gbps of networking bandwidth.
-
G6e instances support up to four network cards, and can deliver up to 400 Gbps of networking bandwidth
For more information about getting started with GPU accelerated instances, see Performance acceleration with GPU instances.
We recommend that you define a single EFA network interface per network card. To configure these interfaces at launch, we recommend the following settings:
-
For network interface
0
, specify device index0
-
For network interfaces
1
through31
, specify device index1
If you are using the Amazon EC2 console, in the Launch Instance Wizard, choose Edit
in the Network settings section. Expand Advanced network
configuration and choose Add network interface to add the required
number of network interfaces. For each network interface, for EFA, select
Enable. For all network interfaces, except the primary network interface, for
Device index, specify 1
. Configure the remaining settings as needed.
If you are using the AWS CLI, use the
run-instances command, for --network-interfaces
, specify the required number of
network interfaces. For each network interface, for InterfaceType
, specify
efa
. For the primary network interface, for NetworkCardIndex
and
DeviceIndex
specify 0
. For the remaining network interfaces, for
NetworkCardIndex
specify a unique value from 1
to 31
,
and for DeviceIndex
specify 1
.
The following example command snippet shows a request with 32 EFA network interfaces.
$
aws --region $REGION ec2 run-instances \ --instance-type p5.48xlarge \ --count 1 \ --key-namekey_pair_name
\ --image-idami_id
\ --network-interfaces "NetworkCardIndex=0,DeviceIndex=0,Groups=security_group_id
,SubnetId=subnet_id
,InterfaceType=efa" \ "NetworkCardIndex=1,DeviceIndex=1,Groups=security_group_id
,SubnetId=subnet_id
,InterfaceType=efa" \ "NetworkCardIndex=2,DeviceIndex=1,Groups=security_group_id
,SubnetId=subnet_id
,InterfaceType=efa" \ "NetworkCardIndex=3,DeviceIndex=1,Groups=security_group_id
,SubnetId=subnet_id
,InterfaceType=efa" \ "NetworkCardIndex=4,DeviceIndex=1,Groups=security_group_id
,SubnetId=subnet_id
,InterfaceType=efa" \ "NetworkCardIndex=5,DeviceIndex=1,Groups=security_group_id
,SubnetId=subnet_id
,InterfaceType=efa" \ "NetworkCardIndex=6,DeviceIndex=1,Groups=security_group_id
,SubnetId=subnet_id
,InterfaceType=efa" \ "NetworkCardIndex=7,DeviceIndex=1,Groups=security_group_id
,SubnetId=subnet_id
,InterfaceType=efa" \ "NetworkCardIndex=8,DeviceIndex=1,Groups=security_group_id
,SubnetId=subnet_id
,InterfaceType=efa" \ "NetworkCardIndex=9,DeviceIndex=1,Groups=security_group_id
,SubnetId=subnet_id
,InterfaceType=efa" \ "NetworkCardIndex=10,DeviceIndex=1,Groups=security_group_id
,SubnetId=subnet_id
,InterfaceType=efa" \ "NetworkCardIndex=11,DeviceIndex=1,Groups=security_group_id
,SubnetId=subnet_id
,InterfaceType=efa" \ "NetworkCardIndex=12,DeviceIndex=1,Groups=security_group_id
,SubnetId=subnet_id
,InterfaceType=efa" \ "NetworkCardIndex=13,DeviceIndex=1,Groups=security_group_id
,SubnetId=subnet_id
,InterfaceType=efa" \ "NetworkCardIndex=14,DeviceIndex=1,Groups=security_group_id
,SubnetId=subnet_id
,InterfaceType=efa" \ "NetworkCardIndex=15,DeviceIndex=1,Groups=security_group_id
,SubnetId=subnet_id
,InterfaceType=efa" \ "NetworkCardIndex=16,DeviceIndex=1,Groups=security_group_id
,SubnetId=subnet_id
,InterfaceType=efa" \ "NetworkCardIndex=17,DeviceIndex=1,Groups=security_group_id
,SubnetId=subnet_id
,InterfaceType=efa" \ "NetworkCardIndex=18,DeviceIndex=1,Groups=security_group_id
,SubnetId=subnet_id
,InterfaceType=efa" \ "NetworkCardIndex=19,DeviceIndex=1,Groups=security_group_id
,SubnetId=subnet_id
,InterfaceType=efa" \ "NetworkCardIndex=20,DeviceIndex=1,Groups=security_group_id
,SubnetId=subnet_id
,InterfaceType=efa" \ "NetworkCardIndex=21,DeviceIndex=1,Groups=security_group_id
,SubnetId=subnet_id
,InterfaceType=efa" \ "NetworkCardIndex=22,DeviceIndex=1,Groups=security_group_id
,SubnetId=subnet_id
,InterfaceType=efa" \ "NetworkCardIndex=23,DeviceIndex=1,Groups=security_group_id
,SubnetId=subnet_id
,InterfaceType=efa" \ "NetworkCardIndex=24,DeviceIndex=1,Groups=security_group_id
,SubnetId=subnet_id
,InterfaceType=efa" \ "NetworkCardIndex=25,DeviceIndex=1,Groups=security_group_id
,SubnetId=subnet_id
,InterfaceType=efa" \ "NetworkCardIndex=26,DeviceIndex=1,Groups=security_group_id
,SubnetId=subnet_id
,InterfaceType=efa" \ "NetworkCardIndex=27,DeviceIndex=1,Groups=security_group_id
,SubnetId=subnet_id
,InterfaceType=efa" \ "NetworkCardIndex=28,DeviceIndex=1,Groups=security_group_id
,SubnetId=subnet_id
,InterfaceType=efa" \ "NetworkCardIndex=29,DeviceIndex=1,Groups=security_group_id
,SubnetId=subnet_id
,InterfaceType=efa" \ "NetworkCardIndex=30,DeviceIndex=1,Groups=security_group_id
,SubnetId=subnet_id
,InterfaceType=efa" \ "NetworkCardIndex=31,DeviceIndex=1,Groups=security_group_id
,SubnetId=subnet_id
,InterfaceType=efa" ...
If you are using a launch template, specify the required number of network interfaces in the launch
template. For each network interface, for InterfaceType
, specify efa
. For
the primary network interface, for NetworkCardIndex
and DeviceIndex
specify
0
. For the remaining network interfaces, for NetworkCardIndex
specify a
unique value from 1
to 31
, and for DeviceIndex
specify
1
. The following snippet shows an example with 3 network interfaces out of the possible
32 network interfaces.
"NetworkInterfaces":[ { "NetworkCardIndex":0, "DeviceIndex":0, "InterfaceType": "efa", "AssociatePublicIpAddress":false, "Groups":[ "
security_group_id
" ], "DeleteOnTermination":true }, { "NetworkCardIndex": 1, "DeviceIndex": 1, "InterfaceType": "efa", "AssociatePublicIpAddress":false, "Groups":[ "security_group_id
" ], "DeleteOnTermination":true }, { "NetworkCardIndex": 2, "DeviceIndex": 1, "InterfaceType": "efa", "AssociatePublicIpAddress":false, "Groups":[ "security_group_id
" ], "DeleteOnTermination":true } ...
When launching a P5 or P5e instance with more than one network interface, you cannot auto-assign public IP addresses. However, you can attach an Elastic IP address to the primary network interface (NetworkCardIndex=0, DeviceIndex=0) after launch for internet connectivity. Both Ubuntu 20.04 and later and Amazon Linux 2 and later are configured to use the primary network interface for internet traffic when the instance is launched as recommended on this page.
To achieve the highest network performance on G6e instances, you can use IMDS to map out attached network interfaces and optimize by using networking instances on disjoint NetworkCardIndexes.
The following example script pulls together details from the attached NetworkCardIndexes.
$ TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X- aws-ec2-metadata-token-ttl-seconds: 21600") count=$(ls -l /sys/class/net/eth* | wc -l) for ((i = 0 ; i < ${count} ; i++)); do mac=$(cat /sys/class/net/eth$i/address) network_card=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" "http://169.254.169.254/latest/meta-data/network/interfaces/macs/$mac/ network-card") device_number=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" "http://169.254.169.254/latest/meta-data/network/interfaces/macs/$mac/ device-number") device_bdf=$(realpath /sys/class/net/eth${i}/device) echo "eth${i} ${network_card} ${device_number} ${device_bdf}" done