Move mainframe files directly to Amazon S3 using Transfer Family - AWS Prescriptive Guidance
SummaryPrerequisites and limitationsArchitectureToolsEpicsRelated resources

Move mainframe files directly to Amazon S3 using Transfer Family

Created by Luis Gustavo Dantas (AWS)

Environment: Production

Source: Mainframe

Target: Amazon S3

R Type: N/A

Workload: IBM

Technologies: Mainframe; Storage & backup; Modernization

AWS services: AWS Transfer Family; Amazon S3

Summary

As part of the modernization journey, you can face the challenge of transferring files between your on-premises servers and the Amazon Web Services (AWS) Cloud. Transferring data from mainframes can be a significant challenge because mainframes typically can’t access modern data stores like Amazon Simple Storage Service (Amazon S3), Amazon Elastic Block Store (Amazon EBS), or Amazon Elastic File System (Amazon EFS).

Many customers use intermediate staging resources, such as on-premises Linux, Unix, or Windows servers, to transfer files to the AWS Cloud. You can avoid this indirect method by using AWS Transfer Family with the Secure Shell (SSH) File Transfer Protocol (SFTP) to upload mainframe files directly to Amazon S3.

Prerequisites and limitations

Prerequisites

  • An active AWS account

  • A virtual private cloud (VPC) with a subnet that’s reachable by your legacy platform

  • A Transfer Family endpoint for your VPC

  • Mainframe Virtual Storage Access Method (VSAM) files converted to sequential, fixed-length files (IBM documentation)

Limitations

  • SFTP transfers files in binary mode by default, which means that files are uploaded to Amazon S3 with EBCDIC encoding preserved. If your file doesn't contain binary or packed data, then you can use the sftp ascii subcommand (IBM documentation) to convert your files to text during the transfer.

  • You must unpack mainframe files (AWS Prescriptive Guidance) that contain packed and binary content to use these files in your target environment.

  • Amazon S3 objects can range in size from a minimum of 0 bytes to a maximum of 5 TB. For more information about Amazon S3 capabilities, see Amazon S3 FAQs.

Architecture

Source technology stack

  • Job control language (JCL)

  • z/OS Unix shell and ISPF

  • SFTP

  • VSAM and flat files

Target technology stack

  • Transfer Family

  • Amazon S3

  • Amazon Virtual Private Cloud (Amazon VPC)

Target architecture

The following diagram shows a reference architecture for using Transfer Family with SFTP to upload mainframe files directly to an S3 bucket.

Using Transfer Family with SFTP to upload mainframe files directly to an S3 bucket

The diagram shows the following workflow:

  1. You use a JCL job to transfer your mainframe files from the legacy mainframe to the AWS Cloud through Direct Connect.

  2. Direct Connect enables your network traffic to remain on the AWS global network and bypass the public internet. Direct Connect also enhances the network speed, starting at 50 Mbps and scaling up to 100 Gbps.

  3. The VPC endpoint enables connections between your VPC resources and the supported services without using the public internet. Access to Transfer Family and Amazon S3 achieves high availability by taking place through the elastic network interfaces located in two private subnets and Availability Zones.

  4. Transfer Family authenticates users and uses SFTP to receive your files from the legacy environment and move them to an S3 bucket.

Automation and scale

After the Transfer Family service is in place, you can transfer an unlimited number of files from the mainframe to Amazon S3 by using a JCL job as the SFTP client. You can also automate the file transfer by using a mainframe batch job scheduler to run the SFTP jobs when you’re ready to transfer the mainframe files.

Tools

  • Amazon Simple Storage Service (Amazon S3) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data.

  • Amazon Virtual Private Cloud (Amazon VPC) helps you launch AWS resources into a virtual network that you’ve defined. This virtual network resembles a traditional network that you’d operate in your own data center, with the benefits of using the scalable infrastructure of AWS.

  • AWS Transfer Family enables you to securely scale your recurring business-to-business file transfers to Amazon S3 and Amazon EFS by using SFTP, FTPS, and FTP protocols.

Epics

TaskDescriptionSkills required

Create the S3 bucket.

Create an S3 bucket to host the files that you transfer from your legacy environment.

General AWS

Create the IAM role and policy.

Transfer Family uses your AWS Identity and Access Management (IAM) role to grant access to the S3 bucket that you created earlier.

Create an IAM role that includes the following IAM policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "UserFolderListing",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::<your-bucket-name>"
            ]
        },
        {
            "Sid": "HomeDirObjectAccess",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObjectAcl",
                "s3:GetObject",
                "s3:DeleteObjectVersion",
                "s3:DeleteObject",
                "s3:PutObjectAcl",
                "s3:GetObjectVersion"
            ],
            "Resource": "arn:aws:s3:::<your-bucket-name>/*"
        }
    ]
}

Note: You must choose the Transfer use case when you create the IAM role.

General AWS
TaskDescriptionSkills required

Create the SFTP server.

  1. Sign in to the AWS Management console, open the Transfer Family console, and then choose Create server.

  2. Choose only SFTP (SSH File Transfer Protocol) - file transfer over Secure Shell protocol and then choose Next.

  3. For Identity provider, choose Service managed and then choose Next.

  4. For Endpoint type, choose VPC hosted.

  5. For Access, choose Internal.

  6. For VPC, choose your VPC.

  7. In the Availability Zones section, choose your Availability Zones and subnets.

  8. In the Security Groups section, choose your security group, and then choose Next.

  9. For Domain, choose Amazon S3 and then choose Next.

  10. Leave the default options on the Configure additional details page and then choose Next.

  11. Choose Create server.

Note: For more information about how to set up an SFTP server, see Create an SFTP-enabled server (AWS Transfer Family User Guide).

General AWS

Get the server address.

  1. Open the Transfer Family console and choose your server ID in the Server ID column.

  2. In the Endpoint details section, for Endpoint type, choose the endpoint ID. This takes you to the Amazon VPC console.

  3. On the Details tab of the Amazon VPC console, find the DNS names next to DNS names.

General AWS

Create the SFTP client key pair.

Create an SSH key pair for either Microsoft Windows or macOS/Linux/UNIX.

General AWS, SSH

Create the SFTP user.

  1. Open the Transfer Family console, choose Servers from the navigation pane, and then select your server.

  2. In the Server ID column, choose the server ID for your server and then choose Add user.

  3. For Username, enter a user name that matches your SSH key pair user name.

  4. For Role, choose the IAM role that you created earlier.

  5. For Home directory, choose the S3 bucket that you created earlier.

  6. For SSH public keys, enter the key pair that you created earlier.

  7. Choose Add.

General AWS
TaskDescriptionSkills required

Send the SSH private key to the mainframe.

Use SFTP or SCP to send the SSH private key to the legacy environment.

SFTP example:

sftp [USERNAME@mainframeIP]
[password]
cd [/u/USERNAME]
put [your-key-pair-file]

SCP example:

scp [your-key-pair-file] [USERNAME@MainframeIP]:/[u/USERNAME]

Next, store the SSH key in the z/OS Unix file system under the user name that will later run the file transfer batch job (for example, /u/CONTROLM). 

Note: For more information about z/OS Unix shell, see An introduction to the z/OS shells (IBM documentation).

Mainframe, z/OS Unix shell, FTP, SCP

Create the JCL SFTP client.

Because mainframes don't have a native SFTP client, you must use the BPXBATCH utility to run the SFTP client from the z/OS Unix shell.

In the ISPF editor, create the JCL SFTP client. For example:

//JOBNAM JOB ...
//**********************************************************************
//SFTP EXEC PGM=BPXBATCH,REGION=0M 
//STDPARM DD * 
SH cp "//'MAINFRAME.FILE.NAME'" filename.txt; 
echo 'put filename.txt' > uplcmd; 
sftp -b uplcmd -i ssh_private_key_file ssh_username@<transfer service ip or DNS>; 
//SYSPRINT DD SYSOUT=* 
//STDOUT DD SYSOUT=* 
//STDENV DD * 
//STDERR DD SYSOUT=*

Note: For more information about how to run a command in the z/OS Unix shell, see The BPXBATCH utility (IBM documentation). For more information about how to create or edit JCL jobs in z/OS, see What is ISPF? and The ISPF editor (IBM documentation).

JCL, Mainframe, z/OS Unix shell

Run the JCL SFTP client.

  1. In the ISPF editor, enter SUB, and then press the ENTER key after the JCL job is created.

  2. Monitor the mainframe's file transfer batch job activity in SDSF.

Note: For more information about how to check the activity of batch jobs, see z/OS SDSF User's Guide (IBM documentation).

Mainframe, JCL, ISPF

Validate the file transfer.

  1. Sign in to the AWS Management console, open the Amazon S3 console, and then choose Buckets from the navigation pane.

  2. Choose the bucket that’s associated with your Transfer Family.

  3. In the Objects section of the Objects tab, find the file that you transferred from the mainframe.

General AWS

Automate the JCL SFTP client.

Use job scheduler to automatically trigger the JCL SFTP client.

Note: You can use mainframe job schedulers, such as BMC Control-M or CA Workload Automation, to automate batch jobs for file transfers based on time and other batch job dependencies.

Job scheduler

Related resources