Performance optimization

This topic explains the causes for slow upload speeds, and provides some changes that you can make to increase speed.

Network bandwidth

File Transfer increases network use and saturation. It can’t deliver faster than the bandwidth that it’s allocated to use. If your machine has been allocated a network bandwidth of 500 Mbps, the fastest File Transfer can try to deliver at is 500 Mbps. If you want faster transfers, allocate additional bandwidth to the host system.

Disk throughput

Disk throughput must scale accordingly with increasing Network bandwidth. You need enough I/O throughput to support a high max active transfer or thread count configuration. Your transfers will slow down if the storage that is attached to the host machine (such as NAS, SAN, local SSD, and external HDD) doesn’t have enough I/O throughput. To avoid this, upgrade your infrastructure by upgrading your hardware, CPU, and internet.

Latency

We recommend that you deploy File Transfer infrastructure in the AWS Region that is geographically closest to the download and upload facility. The latency between the transfer profile's internet service provider to the destination will vary, unless the transfer profile is using AWS Direct Connect. For more information about AWS Direct Connect, see the AWS Direct Connect User Guide.

Throttling

File Transfer can’t throttle its bandwidth use. To work around this issue, use Quality of Service (QoS) to limit the firewall layer, or traffic shape at the virtual local area network (VLAN) layer.

Maximum limit of open files

Some host machines (mostly Linux and macOS) come with preconfigured soft and hard limits for the maximum number of open files. At minimum, File Transfer creates file descriptors to access disk and network resources. We recommend that your host machine has a 20,000 limit for maximum open files.

Bucket visibility

File Transfer uses the default Amazon S3 endpoints. You can choose to use the default Amazon S3 accelerated endpoints. For more information about accelerated endpoints, see Amazon Simple Storage Service (Amazon S3).

You can use the AWS Command Line Interface (AWS CLI) to list content in your bucket. Do this by using your File Transfer access and secret keys, or through the Amazon S3 console.

Optimize uploads (when not autotuning)

For all-around best performance, keep the chunk size between 25–100MB. Threads and max active transfers will vary depending on the characteristics of your upload package. Single session transfer speeds are limited by the protocols in use (TCP/HTTP). The optimal configuration includes lower chunk sizes, and it includes higher thread and max active transfers settings. It is a best practice to set the chunk size to be slightly bigger than the median file size. However, the best practice is to not exceed 50 MB on most hardware.

Configuration and database file location

The configuration file and the database file are located in any directory, as defined by the FILETRANSFER_CONFIG_DIR environment variable. If the variable isn’t set, these files are located in ~/.filetransfer by default. The configuration file is named configuration.yaml and the database file is named checksum-cache.db.

Turning off the API server

By default, File Transfer listens on port 50005 for incoming connections from the File Transfer graphical user interface (GUI) application. To turn this off, define api_server.enabled in the configuration file and set it to false.