Migrate small sets of data from on premises to Amazon S3 using AWS SFTP
Created by Charles Gibson (AWS) and Sergiy Shevchenko (AWS)
R Type: Rehost | Source: Storage | Target: Amazon S3 |
Created by: AWS | Environment: Production | Technologies: Storage & backup; Migration |
AWS services: Amazon S3 |
Summary
This pattern describes how to migrate small sets of data (5 TB or less) from on-premises data centers to Amazon Simple Storage Service (Amazon S3) by using AWS Transfer for SFTP (AWS SFTP). The data can be either database dumps or flat files.
Prerequisites and limitations
Prerequisites
An active AWS account
An AWS Direct Connect link established between your data center and AWS
Limitations
The data files must be less than 5 TB. For files over 5 TB, you can perform a multipart upload to Amazon S3 or choose another data transfer method.
Architecture
Source technology stack
On-premises flat files or database dumps
Target technology stack
Amazon S3
Source and target architecture
Tools
AWS SFTP – Enables the transfer of files directly into and out of Amazon S3 using Secure File Transfer Protocol (SFTP).
AWS Direct Connect – Establishes a dedicated network connection from your on-premises data centers to AWS.
VPC endpoints – Enable you to privately connect a VPC to supported AWS services and VPC endpoint services powered by AWS PrivateLink without an internet gateway, network address translation (NAT) device, VPN connection, or AWS Direct Connect connection. Instances in a VPC don't require public IP addresses to communicate with resources in the service.
Epics
| Task | Description | Skills required |
|---|---|---|
| Document the current SFTP requirements. | Application owner, SA | |
| Identify the authentication requirements. | Requirements may include key-based authentication, user name or password, or identity provider (IdP). | Application owner, SA |
| Identify the application integration requirements. | Application owner | |
| Identify the users who require the service. | Application owner | |
| Determine the DNS name for the SFTP server endpoint. | Networking | |
| Determine the backup strategy. | SA, DBA (if data is transferred) | |
| Identify the application migration or cutover strategy. | Application owner, SA, DBA |
| Task | Description | Skills required |
|---|---|---|
| Create one or more virtual private clouds (VPCs) and subnets in your AWS account. | Application owner, AMS | |
| Create the security groups and network access control list (ACL). | Security, Networking, AMS | |
| Create the S3 bucket. | Application owner, AMS | |
| Create the identity and access management (IAM) role. | Create an IAM policy that includes the permissions to enable AWS SFTP to access your S3 bucket. This IAM policy determines what level of access you provide SFTP users. Create another IAM policy to establish a trust relationship with AWS SFTP. | Security, AMS |
| Associate a registered domain (optional). | If you have your own registered domain, you can associate it with the SFTP server. You can route SFTP traffic to your SFTP server endpoint from a domain or from a subdomain. | Networking, AMS |
| Create an SFTP server. | Specify the identity provider type used by the service to authenticate your users. | Application owner, AMS |
| Open an SFTP client. | Open an SFTP client and configure the connection to use the SFTP endpoint host. AWS SFTP supports any standard SFTP client. Commonly used SFTP clients include OpenSSH, WinSCP, Cyberduck, and FileZilla. You can get the SFTP server host name from the AWS SFTP console. | Application owner, AMS |
| Task | Description | Skills required |
|---|---|---|
| Plan the application migration. | Plan for any application configuration changes required, set the migration date, and determine the test schedule. | Application owner, AMS |
| Test the infrastructure. | Test in a non-production environment. | Application owner, AMS |
Related resources
References
Tutorials and videos
