Using AWS DataSync to Transfer Files - AWS Snowcone User Guide

Using AWS DataSync to Transfer Files

AWS DataSync is an online data transfer service that simplifies, automates, and accelerates copying large amounts of data to and from AWS storage services over the internet or AWS Direct Connect. DataSync agent comes pre-installed on your Snowcone device. It can transfer data between the device and Amazon S3 buckets, Amazon EFS, and Amazon FSx for Windows File Server. AWS DataSync automatically handles moving files and objects, scheduling data transfers, monitoring the progress of transfers, encryption, verification of data transfers, and notifying customers of any issues.

Before starting the DataSync agent, enable Network File System (NFS) on your Snowcone device. See Configuring the NFS interface automatically with AWS OpsHub and Starting NFS and Restricting Access.

The DataSync agent is pre-installed on your Snowcone device as an AMI during the Snowcone job preparation. To transfer data online to AWS, connect the Snowcone device to the external network and use AWS OpsHub or the CLI to launch the DataSync agent AMI. Activate the DataSync agent the AWS Management Console or use the CLI, and set up your online data transfer task between the Snowcone NFS store, and Amazon S3, Amazon EFS, or Amazon FSx for Windows File Server.

You can use AWS DataSync running on Snowcone for the following:

  • Edge computing applications, to collect data, process the data to gain immediate insight, and then transfer the data online to AWS.

  • Transfer data that is continuously generated by sensors or machines online to AWS in a factory or at other edge locations.

  • Distribute media, scientific, or other content online from AWS storage services to your partners and customers.

  • Aggregate content by transferring media, scientific or other content online from your edge locations to AWS.

  • Ensure you use a static IP address for the VNI for the DataSync agent. Using DHCP may cause issues because any device reboots can mean that you’re using a different IP address, forcing a the need to reconfigure both the DataSync agent and DataSync activation inAWS cloud.

  • To use Datasync agent on Snowcone, you need to ensure that the “AllowedHosts” entries includes the DataSync Agent IP address in your NFS configuration on the Snowcone. This may require you to stop and restart the NFS service on Snowcone to enable the DataSync agent.

  • Whenever you stop the NFS service, you should also stop the DataSync agent too. Note: If you stop the NFS service which has allow the listed DataSync agent, it will stop Datasync agent from working.

  • Check the AWS DataSync task quotas for the maximum number of files per task for a Snowcone device. If you exceed the 200,000 files limit, the DataSync task will report a memory allocation error and abort execution.

For one-time edge compute or data transfer workflows or for Snowcone workflows in edge locations without a wide area network (WAN) link or inadequate WAN bandwidth, you should ship the Snowcone device back to AWS to complete the data transfer.