Amazon EFS performance - Amazon Elastic File System

Amazon EFS performance

Amazon EFS provides a serverless, set-and-forget elastic file system that you can access from any compute service within AWS and on-premises, including:

  • Amazon Elastic Compute Cloud (Amazon EC2)

  • Amazon Elastic Container Service (Amazon ECS)

  • Amazon Elastic Kubernetes Service (Amazon EKS)

  • AWS Fargate

  • AWS Lambda

Amazon EFS delivers more than 10 gibibytes per second (GiBps) of throughput over 500,000 IOPS, and sub-millisecond or low single digit millisecond latencies.

The following sections provide an overview of Amazon EFS performance, and describe how your file system configuration impacts key performance dimensions. We also provide some important tips and recommendations for optimizing the performance of your file system.

Performance summary

File system performance is typically measured using the dimensions of latency, throughput, and I/O operations per second (IOPS). Amazon EFS performance across these dimensions depends on your file system's configuration. The following configurations impact the performance of an Amazon EFS file system:

  • Storage class – EFS One Zone or EFS Standard

  • Performance mode – General Purpose or Max I/O

  • Throughput mode – Bursting or Provisioned

The following table illustrates Amazon EFS file system performance for the available combinations of storage class and performance mode settings.

File system performance for storage class and performance mode combinations
Latency1 Maximum IOPS Maximum throughput

File system configuration – Storage class and performance mode

Read operations

Write operations

Read operations

Write operations

Per-file-system read2

Per-file-system write2

Per-client read/write

One Zone storage and General Purpose

As low as 600 microseconds (µs)

Low single-digit milliseconds

35,000 7,000

3 – 5 GiBps

1 – 3 GiBps

500 MiBps

Standard storage and General Purpose

As low as 600 µs

Low single-digit milliseconds

35,000 7,000

3 – 5 GiBps

1 – 3 GiBps

500 MiBps

Standard storage and Max I/O

Single-digit milliseconds

Single-digit to double-digit milliseconds

>500,000 >100,000

3 – 5 GiBps

1 – 3 GiBps

500 MiBps
Note

Footnotes:

  1. Latencies for file data reads and writes to the cost-optimized storage classes (Standard-IA and One Zone-IA) are double-digit milliseconds.

  2. Maximum read and write throughput depend on the AWS Region and on the file system's throughput mode (Bursting or Provisioned). For more information, see the table of default throughput quotas for Bursting and Provisioned Throughput modes.

    Throughput in excess of an AWS Region's maximum throughput requires a throughput quota increase. Any request for additional throughput is considered on a case-by-case basis by the Amazon EFS service team. Approval might depend on your type of workload. To learn more about requesting quota increases, see Amazon EFS quotas and limits.

Storage classes and performance

Amazon EFS uses the following storage classes:

  • EFS One Zone storage classes – EFS One Zone and EFS One Zone-Infrequent Access (EFS One Zone-IA). The EFS One Zone storage classes replicate data within a single Availability Zone.

  • EFS Standard storage classes – EFS Standard and EFS Standard-IA. The EFS Standard storage classes replicate data across multiple Availability Zones (Multi-AZ).

First-byte latency when reading from or writing to either of the IA storage classes is higher than that for the EFS Standard or EFS One Zone storage classes.

For more information about EFS storage classes, see EFS storage classes.

Performance modes

Amazon EFS offers two performance modes, General Purpose and Max I/O:

  • General Purpose mode supports up to 35,000 IOPS and has the lowest per-operation latency. File systems with EFS One Zone storage classes always use General Purpose performance mode. For file systems with EFS Standard storage classes, you can use either the default General Purpose performance mode or the Max I/O performance mode.

  • Max I/O mode supports 500,000+ IOPS and has higher per-operation latencies when compared to General Purpose mode.

You set the performance mode when you create a file system, and you can't change it after it is created.

We recommend using General Purpose performance mode for the vast majority of applications. If you are not sure which performance mode to use, choose the General Purpose performance mode. To help ensure that your workload stays within the IOPS limit available to file systems using General Purpose mode, you can monitor the PercentIOLimit CloudWatch metric. For more information, see Amazon CloudWatch metrics for Amazon EFS.

Applications can scale their IOPS elastically up to the limit associated with the performance mode. You are not billed separately for IOPS; they are included in a file system's throughput accounting. Every Network File System (NFS) request is accounted for as 4 KB of throughput, or its actual request and response size, whichever is larger. For example, a file system that can drive 100 MBps of throughput can drive up to 25,600 4-KB writes per second (100 MBps divided by 4 KB per request = 25,600 requests per second).

Throughput modes

A file system's throughput mode determines the throughput available to your file system. Amazon EFS offers two throughput modes, Bursting Throughput and Provisioned Throughput. Read throughput is discounted to allow you to drive higher read throughput than write throughput. Depending on the AWS Region, the discount for reads is between 1.66 and 3x. For more information, see the table of default throughput quotas for Bursting and Provisioned Throughput modes. Discounting reduces throughput metered for reads, and does not impact writes or burst credit accruals. Furthermore, read discounting never reduces the throughput metered for a single NFS request below a minimum request size of 4 KB.

Understanding metered throughput

All Amazon EFS file systems have an associated metered throughput. For file systems using Provisioned Throughput mode, the metered throughput is determined by the amount of provisioned throughput. For file systems using Bursting Throughput mode, the metered throughput is determined by the amount of data stored in the EFS Standard or EFS One Zone storage class.

Read requests and write requests are metered at different rates. Amazon EFS meters read requests at one-third the rate of other requests.

Example of EFS metered throughput

For example, if you are driving 30 mebibytes per second (MiBps) of both read throughput and write throughput, the read portion counts as 10 MiBps of metered throughput, the write portion counts as 30 MiBps, and the combined metered throughput is 40 MiBps. This combined throughput adjusted for metering rates is reflected in the MeteredIOBytes Amazon CloudWatch metric. For more information, see Amazon CloudWatch metrics for Amazon EFS.

Bursting Throughput mode

Bursting Throughput mode is the default Amazon EFS throughput mode. It is a good fit for traditional applications that have a bursty throughput pattern. When throughput is low, Bursting Throughput mode uses burst buckets to save burst credits. When throughput is higher, it uses burst credits.

In Bursting Throughput mode, file system throughput is proportional to the file system size in the Standard storage class, up to a maximum that depends on the Amazon EFS Region. For more information about per-Region limits, see the table of default throughput quotas for Bursting and Provisioned Throughput modes.

When burst credits are available, a file system can drive up to 100 MBps per terabyte (TB) of storage, with a minimum of 100 MBps. If no burst credits are available, a file system can drive up to 50 MBps per TB of storage with a minimum of 1 MBps.

Read and write throughput are metered, and burst credits are deducted from the burst credit balance for metered throughput. Burst credits accrue in proportion to the file system's size at the file system's base rate. 50 MBps of burst credits are accrued for every TB of storage. Whenever a file system consumes less than its base rate, it accrues burst credits. Whenever a file system consumes more than its base rate, it consumes burst credits. The metered size of ValueInStandard is used to determine your I/O throughput baseline and burst rates. The metered size of the file system for Bursting Throughput performance excludes the amount of data stored in the Infrequent Access storage class.

Amazon EFS burst credits

Amazon EFS uses a credit system to determine when file systems can burst. Each file system earns credits over time at a baseline rate that is determined by the size of the file system that is stored in the EFS Standard or One Zone Standard storage class. A file system uses credits whenever it reads or writes data. The baseline rate is 50 MiBps per TiB of storage (equivalent to 50 KiB/s per GiB of storage). Because Amazon EFS meters read operations at a one-third the rate of other operations toward the baseline rate, an EFS file system can drive up to 150 KiB/s per GiB of read throughput, or 50 KiB/s per GiB of write throughput at this baseline rate.

A file system can drive throughput at its baseline metered rate continuously. A file system accumulates burst credits whenever it is inactive or driving throughput below its baseline metered rate. Accumulated burst credits give the file system the ability to drive throughput above its baseline rate.

For example, a file system with 100 GiB of metered data in Standard storage can burst (at 100 MiBps) for 5 percent of the time if it's inactive for the remaining 95 percent. Over a 24-hour period, the file system earns 432,000 MiB worth of credit, which can be used to burst at 100 MiBps for 72 minutes.

File systems larger than 1 TiB can always burst for up to 50 percent of the time if they are inactive for the remaining 50 percent.

The following table provides examples of bursting behavior.

File system size Burst throughput Baseline throughput
A file system with 100 GiB of metered data in Standard storage can...
  • Burst to 300 mebibytes per second (MiBps) read-only for up to 72 minutes per day

  • Burst to 100 MiBps write-only for up to 72 minutes per day, or

  • Drive up to 15 MiBps read-only continuously

  • Drive up to 5 MiBps write-only continuously

A file system with 1-TiB of metered data in Standard storage can...
  • Burst to 300 MiBps read-only for 12 hour per day

  • Burst to 100 MiBps write-only for 12 hours per day, or

  • Drive 150 MiBps read-only continuously

  • Drive 50 MiBps write-only continuously

A file system with 10-TiB of metered data in Standard storage can...
  • Burst to 3 GiBps read-only for 12 hours per day

  • Burst to 1 GiBps write-only for 12 hours per day, or

  • Drive 1.5 GiBps read-only continuously

  • Drive 500 MiBps write-only continuously

Generally, a larger file system can...
  • Burst to 300 MiBps read-only per TiB of storage for 12 hours per day

  • Burst to 100 MiBps write-only per TiB of storage for 12 hours per day, or

  • Drive 150 MiBps read-only per TiB of storage continuously

  • Drive 50 MiBps write-only per TiB of storage continuously

Note

Amazon EFS gives a metered throughput of 1 MiBps to all file systems, even if the baseline rate is lower.

The file system size used to determine the baseline and burst rates is the ValueInStandard metered size available through the DescribeFileSystems API operation

File systems can earn credits up to a maximum credit balance of 2.1 TiB for file systems smaller than 1 TiB, or 2.1 TiB per TiB stored for file systems larger than 1 TiB. This approach implies that file systems can accumulate enough credits to burst for up to 12 hours continuously.

Provisioned Throughput mode

For applications that have a relatively constant throughput, we recommend Provisioned Throughput mode. In Provisioned Throughput mode, you specify a level of throughput that the file system can drive independent of the file system's size or burst credit balance. You are charged for the amount of throughput you provision that is in excess of the file system's base throughput rate based on the amount of storage if your file system were using Bursting Throughput mode. If the metered size of the data stored in Standard storage of your file system provides higher baseline throughput than the amount throughput you have provisioned, then your file system will automatically use the Bursting Throughput mode.

Switching throughput modes

You can switch an existing file system's throughput mode, with the restriction that you can make only one restricted change in a 24-hour period. The following are considered restricted changes:

  • Changing a file system's throughput mode (from Provisioned Throughput to Bursting Throughput, or from Bursting Throughput to Provisioned Throughput).

  • Reducing the amount of provisioned throughput in Provisioned Throughput mode.

Amazon EFS performance tips

When using Amazon EFS, keep the following performance tips in mind.

Average I/O size

The distributed nature of Amazon EFS enables high levels of availability, durability, and scalability. This distributed architecture results in a small latency overhead for each file operation. Because of this per-operation latency, overall throughput generally increases as the average I/O size increases, because the overhead is amortized over a larger amount of data.

Request model

If you enable asynchronous writes to your file system, pending write operations are buffered on the Amazon EC2 instance before they're written to Amazon EFS asynchronously. Asynchronous writes typically have lower latencies. When performing asynchronous writes, the kernel uses additional memory for caching.

A file system that has enabled synchronous writes, or one that opens files using an option that bypasses the cache (for example, O_DIRECT), issues synchronous requests to Amazon EFS. Every operation goes through a round trip between the client and Amazon EFS.

Note

Your chosen request model has tradeoffs in consistency (if you're using multiple Amazon EC2 instances) and speed. Using synchronous writes provides increased data consistency by completing each write request transaction before processing the next request. Using asynchronous writes provides increased throughput by buffering pending write operations.

NFS client mount settings

Verify that you're using the recommended mount options as outlined in Mounting EFS file systems and in Additional mounting considerations.

When mounting your file systems on Amazon EC2 instances, Amazon EFS supports the Network File System version 4.0 and 4.1 (NFSv4) protocols. NFSv4.1 provides better performance for parallel small-file read operations (greater than 10,000 files per second) compared to NFSv4.0 (less than 1,000 files per second). For Amazon EC2 macOS instances running macOS Big Sur, only NFSv4.0 is supported.

Don't use the following mount options:

  • noac, actimeo=0, acregmax=0, acdirmax=0 – These options disable the attribute cache, which has a very large performance impact.

  • lookupcache=pos, lookupcache=none – These options disable the file name lookup cache, which has a very large impact on performance.

  • fsc – This option enables local file caching, but does not change NFS cache coherency, and does not reduce latencies.

Note

When you mount your file system, consider increasing the size of the read and write buffers for your NFS client to 1 MB.

Optimizing small-file performance

You can improve small-file performance by minimizing file reopens, increasing parallelism, and bundling reference files where possible.

  • Minimize the number of round trips to the server.

    Don't unnecessarily close files if you will need them later in a workflow. Keeping file descriptors open enables direct access to the local copy in the cache. File open, close, and metadata operations generally cannot be made asynchronously or through a pipeline.

    When reading or writing small files, the two additional round trips are significant.

    Each round trip (file open, file close) can take as much time as reading or writing megabytes of bulk data. It's more efficient to open an input or output file once, at the beginning of your compute job, and hold it open for the entire length of the job.

  • Use parallelism to reduce the impact of round-trip times.

  • Bundle reference files in a .zip file. Some applications use a large set of small, mostly read-only reference files. Bundling these in a .zip file allows you to read many files with one open-close round trip.

    The .zip format allows for random access to individual files.

Optimizing directory performance

When performing a listing (ls) on very large directories (over 100k files) that are being modified concurrently, Linux NFS clients can hang, not returning a response. This issue is fixed in kernel 5.11, which has been ported to Amazon Linux 2 kernels 4.14, 5.4, and 5.10.

We recommend keeping the number of directories on your file system to less than 10,000, if possible. Use nested subdirectories as much as possible.

When listing a directory, avoid getting file attributes if they are not required, because they are not stored in the directory itself.

Optimizing the NFS read_ahead_kb size

The NFS read_ahead_kb attribute defines the number of kilobytes for the Linux kernel to read ahead or prefetch during a sequential read operation.

For Linux kernel versions prior to 5.4.*, the read_ahead_kb value is set by multiplying NFS_MAX_READAHEAD by the value for rsize (the client configured read buffer size set in the mount options). When using the recommended mount options, this formula sets read_ahead_kb to 15 MB.

Note

Starting with Linux kernel versions 5.4.*, the Linux NFS client uses a default read_ahead_kb value of 128 KB. We recommend increasing this value to 15 MB.

The Amazon EFS mount helper that is available in amazon-efs-utils version 1.33.2 and later automatically modifies the read_ahead_kb value to equal 15 * rsize, or 15 MB, after mounting the file system.

For Linux kernels 5.4 or later, if you do not use the mount helper to mount your file systems, consider manually setting read_ahead_kb to 15 MB for improved performance. After mounting the file system, you can reset the read_ahead_kb value by using the following command. Before using this command, replace the following values:

  • Replace read-ahead-value-kb with the desired size in kilobytes.

  • Replace efs-mount-point with the file system's mount point.

device_number=$(stat -c '%d' efs-mount-point) ((major = ($device_number & 0xFFF00) >> 8)) ((minor = ($device_number & 0xFF) | (($device_number >> 12) & 0xFFF00))) sudo bash -c "echo read-ahead-value-kb > /sys/class/bdi/$major:$minor/read_ahead_kb"

The following example sets the read_ahead_kb size to 15 MB.

device_number=$(stat -c '%d' efs) ((major = ($device_number & 0xFFF00) >> 8)) ((minor = ($device_number & 0xFF) | (($device_number >> 12) & 0xFFF00))) sudo bash -c "echo 15000 > /sys/class/bdi/$major:$minor/read_ahead_kb"