Using logical directories to simplify your Transfer Family directory structures

To simplify your AWS Transfer Family server directory structure, you can use logical directories. With logical directories, you can construct a virtual directory structure that uses user-friendly names that your users navigate when they connect to your Amazon S3 bucket or Amazon EFS file system. When you use logical directories, you can avoid disclosing absolute directory paths, Amazon S3 bucket names, and EFS file system names to your end users.

Note

You should use session policies so that your end users can only perform operations that you allow them to perform.

You should use logical directories to create a user-friendly, virtual directory for your end users and abstract away bucket names. Logical directory mappings only allow users to access their designated logical paths and subdirectories, and forbid relative paths that traverse the logical roots.

Transfer Family validates every path that might include relative elements and actively blocks these paths from resolving before we pass these paths to Amazon S3; this prevents your users from moving beyond their logical mappings.

Even though Transfer Family prevents your end users from accessing directories outside of their logical directory, we recommend you also use unique roles or session policies to enforce least privilege at the storage level.

You can use logical directories to set the user’s root directory to a desired location within your storage hierarchy, by performing what is known as a chroot operation. In this mode, users are not able to navigate to a directory outside of the home or root directory that you've configured for them.

For example, although an Amazon S3 user has been scoped down to access only /amzn-s3-demo-bucket/home/${transfer:UserName}, some clients allow users to traverse up a folder to /amzn-s3-demo-bucket/home. In this situation, the user lands back on their intended home directory only after logging out of and back in to the Transfer Family server again. Performing a chroot operation can prevent this situation from occurring.

You can create your own directory structure across buckets and prefixes. This feature is useful if you have a workflow that is expecting a specific directory structure that you can't replicate through bucket prefixes. You can also link to multiple non-contiguous locations within Amazon S3, similar to creating a symbolic link in a Linux file system where your directory path references a different location in the file system.

Logical directory FILE mappings

The HomeDirectoryMapEntry data type now includes a Type parameter. Before this parameter existed, you could have created a logical directory mapping where the target was a file. If you have previously created any of these kinds of logical directory mappings, you must explicitly set the Type to FILE, or these mappings won't work correctly going forward.

One way to do this is to call the UpdateUser API, and set the Type to FILE for the existing mapping.

Rules for using logical directories

Before you build your logical directory mappings, you should understand the following rules:

• When Entry is "/", you can have only one mapping because overlapping paths are not allowed.

• Logical directories support mappings of up to 2.1 MB (for service-managed users, this limit is 2,000 entries). That is, the data structure that contains the mappings has a maximum size of 2.1 MB. If you have a lot of mappings, you can calculate the size of your mappings as follows:

  1. Write out a typical mapping in the format {"Entry":"/entry-path","Target":"/target-path"}, where entry-path and target-path are the actual values that you will use.

  2. Count the characters in that string, then add one (1).

  3. Multiply that number by the approximate number of mappings that you have for your server.

  If the number that you estimated in step 3 is less than 2.1 MB, then your mappings are within the acceptable limit.

• Targets can use the ${transfer:UserName} variable if the bucket or file system path has been parameterized based on the username.

• Targets can be paths in different buckets or file systems, but you must make sure that the mapped AWS Identity and Access Management (IAM) role (the Role parameter in the response) provides access to those buckets or file systems.

• Don't specify the HomeDirectory parameter, because this value is implied by the Entry Target pairs when you're using the LOGICAL value for the HomeDirectoryType parameter.

• Targets must begin with a forward slash (/) character, but don't use trailing forward slashes (/) when you specify the Target. For example, /amzn-s3-demo-bucket/images is acceptable, but amzn-s3-demo-bucket/images and /amzn-s3-demo-bucket/images/ are not.

• Amazon S3 is an object store, which means that folders are a virtual concept, and there is no actual directory hierarchy. If your application issues a stat operation from a client, everything is classified as a file when you're using Amazon S3 for storage. This behavior is described in Organizing objects in the Amazon S3 console using folders in the Amazon Simple Storage Service User Guide. If your application requires that stat accurately show whether something is a file or folder, you can use Amazon Elastic File System (Amazon EFS) as the storage option for your Transfer Family servers.

• If you're specifying logical directory values for your user, the parameter that you use depends on the type of user:

  • For service-managed users, provide logical directory values in HomeDirectoryMappings.

  • For custom identity provider users, provide logical directory values in HomeDirectoryDetails.

Important

Unless you choose to optimize performance for your Amazon S3 directories (when you create or update a server), the root directory must exist on startup. For Amazon S3, this means that you must have already created a zero-byte object ending with a forward slash (/) to create the root folder. Avoiding this issue is a reason to consider optimizing Amazon S3 performance.

Implementing logical directories and chroot

To use logical directories and chroot features, you must do the following:

Turn on logical directories for each user. Do this by setting the HomeDirectoryType parameter to LOGICAL when you create or update your user.

"HomeDirectoryType": "LOGICAL"

chroot

For chroot, create a directory structure that consists of a single Entry and Target pairing for each user. The root folder is the Entry point, and the Target is the location in your bucket or file system to map to.

Example for Amazon S3

[{"Entry": "/", "Target": "/amzn-s3-demo-bucket/jane"}]

Example for Amazon EFS

[{"Entry": "/", "Target": "/fs-faa1a123/jane"}]

You can use an absolute path as in the previous example, or you can use a dynamic substitution for the username with ${transfer:UserName}, as in the following example.

[{"Entry": "/", "Target":
"/amzn-s3-demo-bucket/${transfer:UserName}"}]

In the preceding example, the user is locked to their root directory and cannot traverse up higher in the hierarchy.

Virtual directory structure

For a virtual directory structure, you can create multiple Entry Target pairings, with targets anywhere in your S3 buckets or EFS file systems, including across multiple buckets or file systems, as long as the user’s IAM role mapping has permissions to access them.

In the following virtual structure example, when the user logs into AWS SFTP, they are in the root directory with sub-directories of /pics, /doc, /reporting, and /anotherpath/subpath/financials.

Note

Unless you choose to optimize performance for your Amazon S3 directories (when you create or update a server), either the user or an administrator needs to create the directories if they don't already exist. Avoiding this issue is a reason to consider optimizing Amazon S3 performance.

For Amazon EFS, you still need the administrator to create the logical mappings or the / directory.

[
{"Entry": "/pics", "Target": "/amzn-s3-demo-bucket1/pics"}, 
{"Entry": "/doc", "Target": "/amzn-s3-demo-bucket1/anotherpath/docs"},
{"Entry": "/reporting", "Target": "/amzn-s3-demo-bucket2/Q1"},
{"Entry": "/anotherpath/subpath/financials", "Target": "/amzn-s3-demo-bucket2/financials"}]

Note

You can only upload files to the specific folders that you map. This means that in the previous example, you cannot upload to /anotherpath or anotherpath/subpath directories; only anotherpath/subpath/financials. You also cannot map to those paths directly, as overlapping paths are not allowed.

For example, assume that you create the following mappings:

{
   "Entry": "/pics", 
   "Target": "/amzn-s3-demo-bucket/pics"
}, 
{
   "Entry": "/doc", 
   "Target": "/amzn-s3-demo-bucket/mydocs"
}, 
{
   "Entry": "/temp", 
   "Target": "/amzn-s3-demo-bucket"
}

You can only upload files to those buckets. When you first connect through sftp, you are dropped into the root directory, /. If you attempt to upload a file to that directory, the upload fails. The following commands show an example sequence:

sftp> pwd
Remote working directory: /
sftp> put file
Uploading file to /file
remote open("/file"): No such file or directory

To upload to any directory/sub-directory, you must explicitly map the path to the sub-directory.

For more information about configuring logical directories and chroot for your users, including an AWS CloudFormation template that you can download and use, see Simplify your AWS SFTP Structure with chroot and logical directories in the AWS Storage Blog.

Configure logical directories example

In this example, we create a user and assign two logical directories. The following command creates a new user (for an existing Transfer Family server) with logical directories pics and doc.

aws transfer create-user --user-name marymajor-logical --server-id s-11112222333344445 --role arn:aws:iam::1234abcd5678:role/marymajor-role --home-directory-type LOGICAL \
 --home-directory-mappings "[{\"Entry\":\"/pics\", \"Target\":\"/amzn-s3-demo-bucket1/pics\"}, {\"Entry\":\"/doc\", \"Target\":\"/amzn-s3-demo-bucket2/test/mydocs\"}]" \
 --ssh-public-key-body file://~/.ssh/id_rsa.pub

If marymajor is an existing user and her home directory type is PATH, you can change it to LOGICAL with a similar command as the previous one.

aws transfer update-user --user-name marymajor-logical \
 --server-id s-11112222333344445 --role arn:aws:iam::1234abcd5678:role/marymajor-role \
 --home-directory-type LOGICAL --home-directory-mappings "[{\"Entry\":\"/pics\", \"Target\":\"/amzn-s3-demo-bucket1/pics\"}, \
 {\"Entry\":\"/doc\", \"Target\":\"/amzn-s3-demo-bucket2/test/mydocs\"}]"

Note the following:

• If the directories /amzn-s3-demo-bucket1/pics and /amzn-s3-demo-bucket2/test/mydocs don't already exist, the user (or an administrator) needs to create them.

• When marymajor connects to the server, and runs the ls -l command, Mary sees the following:

  drwxr--r--   1        -        -        0 Mar 17 15:42 doc
  drwxr--r--   1        -        -        0 Mar 17 16:04 pics

• marymajor cannot create any files or directories at this level. However, within pics and doc, she can add sub-directories.

• Files that Mary adds to pics and doc are added to Amazon S3 paths /amzn-s3-demo-bucket1/pics and /amzn-s3-demo-bucket2/test/mydocs respectively.

• In this example, we specify two different buckets to illustrate that possibility. However, you can use the same bucket for several or all of the logical directories that you specify for the user.

Configure logical directories for Amazon EFS

If your Transfer Family server uses Amazon EFS, the home directory for the user must be created with read and write access before the user can work in their logical home directory. The user cannot create this directory themselves, as they would lack permissions for mkdir on their logical home directory.

If the user's home directory does not exist, and they run an ls command, the system responds as follows:

sftp> ls
remote readdir ("/"): No such file or directory

A user with administrative access to the parent directory needs to create the user's logical home directory.

Custom AWS Lambda response

You can use logical directories with a Lambda function that connects to your custom identity provider. To do so, in your Lambda function, you specify the HomeDirectoryType as LOGICAL, and add Entry and Target values for the HomeDirectoryDetails parameter. For example:

HomeDirectoryType: "LOGICAL"
HomeDirectoryDetails: "[{\"Entry\": \"/\", \"Target\": \"/amzn-s3-demo-bucket/theRealFolder"}]"

The following code is an example of a successful response from a custom Lambda authentication call.

aws transfer test-identity-provider --server-id s-1234567890abcdef0 --user-name myuser
{
    "Url": "https://a1b2c3d4e5.execute-api.us-east-2.amazonaws.com/prod/servers/s-1234567890abcdef0/users/myuser/config", 
    "Message": "", 
    "Response": "{\"Role\": \"arn:aws:iam::123456789012:role/bob-usa-role\",\"HomeDirectoryType\": \"LOGICAL\",\"HomeDirectoryDetails\": \"[{\\\"Entry\\\":\\\"/myhome\\\",\\\"Target\\\":\\\"/amzn-s3-demo-bucket/theRealFolder\\\"}]\",\"PublicKeys\": \"[ssh-rsa myrsapubkey]\"}", 
    "StatusCode": 200
}

Note

The "Url": line is returned only if you are using an API Gateway method as your custom identity provider.