File Transfer best practices
To maximize benefits from Nimble Studio File Transfer, we recommend that you perform the best practices on this page.
Contents
Amazon Simple Storage Service (Amazon S3)
-
Follow the Amazon Simple Storage Service (Amazon S3) bucket naming practices that are described in the Creating object key names tutorial.
-
To optimize transfer speeds from across the world into Amazon S3 buckets, follow the instructions in the Configuring fast, secure file transfers using Amazon S3 Transfer Acceleration tutorial.
-
To minimize storage costs, configure a lifecycle rule by following the instructions in the Configuring a bucket lifecycle configuration to abort incomplete multipart uploads tutorial. For more information about decreasing costs, see the blog post, Discovering and Deleting Incomplete Multipart Uploads to Lower Amazon S3 Costs
.
AWS Key Management Service (AWS KMS)
-
When you create an S3 bucket in Create an S3 bucket, we recommend that you choose an AWS Key Management Service key (SSE-KMS). For more information about KMS keys, see Customer keys and AWS keys.
Hardware
We recommend that your computer meets the following requirements for you to use File Transfer.
-
8 logical CPU cores
-
8 GB RAM
File Transfer can run on a machine with fewer specifications than these, but that can decrease performance.
Configuration
You have the option to use autotuning or manually tune parameters to best meet your file transfer use cases. If you choose to manually tune your settings, poor performance may be encountered if done improperly. We recommend most users keep autotuning enabled.
Autotuning configuration
We recommended that most users leave the Transfer autotuning setting enabled. If the Transfer autotuning setting is enabled, File Transfer automatically sets values for Number of threads and Chunk size. File Transfer determines the most effective settings based on the type of data that you transfer.
The Transfer autotuning setting adjusts your performance settings on a per-file basis. If you manually set the values for Number of threads and Chunk size, those settings are applied to the entire batch of transfers. This is why Transfer autotuning usually has increased performance when you transfer different sized files. The performance of the Transfer autotuning setting is comparable to manual settings adjustments when you transfer files of similar sizes. If you have advanced knowledge of your hardware and transfer data, your manual settings can outperform the values that Transfer autotuning chooses.
Threads
With File Transfer, each transfer is split into multiple, individual threads that are used to transfer each file. Threads are most effective when you upload large files (> 1 GB). Threads can also help with small files, but the differences in transfer speeds won't be noticeable.
By default, the number of threads is 10. We recommend that you raise this value by increments of 5 until you fully use your bandwidth resources. You can monitor the bandwidth resources from the graphical user interface (GUI) by looking at the download and upload speeds for transfers.
Chunk size
The chunk size is the size (in MB) that is delivered by each thread. Chunk size is helpful if a file size is repeated within the set of files being uploaded.
We recommend that you set the chunk size to 5 to 10 times greater than the average file size that you are transferring. For example, if the average file size in a dataset is 50 MB, set the chunk size between 55 and 60. If the file size is larger than 1 GB, this won’t show as much benefit.
Max active transfers
Max active transfers determine how many individual files that File Transfer processes at the same time. Adjusting max active transfers is most effective when you're transferring multiple small files that are less than 1 GB. We recommend that you increase the value of max active transfers as the file size decreases.
The following table shows the recommended starting points and increments for max active transfers. Start at the value in the max active transfers column and raise it by the increment amount until you reach the desired performance.
| File size | Max active transfers | Increments |
|---|---|---|
|
< 1 MB |
100 |
|
|
> 1 MB–< 100 MB |
50 |
|
|
> 100 MB–< 1 GB |
25 |
|
|
> 1 GB |
10 |
|
Checksums
File Transfer checksums are the number of individual checksums that File Transfer processes at a time. The checksum algorithm is the algorithm that File Transfer uses for file integrity when transferring files.
You can choose between four checksum algorithms: MD5, XXHash, XXHash64, and XXH3. This is a preference based on what level of security and speed that you want. The earliest and most standard checksum method that is secure is MD5.
The recommended max active checksums value is the total number of CPU cores minus 1.
