AWS Import/Export
Developer Guide (API Version 2010-06-03)
Did this page help you?  Yes | No |  Tell us about it...
« PreviousNext »
View the PDF for this guide.Go to the AWS Discussion Forum for this product.Go to the Kindle Store to download this guide in Kindle format.

Manipulating Amazon S3 Object Key Names

The file path on your storage device determines the corresponding object key when AWS loads your data to an Amazon S3 bucket. For example, if your storage device has an images top-level folder with a file sample.jpg, the uploaded file/object key will be images/sample.jpg.

You can use the substitutions option to change object key names when importing to Amazon S3. For example, you can add prefixes or suffixes to file names. For more information, see Working with Amazon S3 Substitutions.

Specifying Key Prefix

The AWS Import/Export prefix mechanism allows you to create a logical grouping of the objects in a bucket. The prefix value is similar to a directory name that enables you to store similar data under the same directory in a bucket. For example, if your Amazon S3 bucket name is my-bucket, and you set prefix to my-prefix/, and the file on your storage device is /jpgs/sample.jpg, then sample.jpg would be loaded to http://s3.amazonaws.com/my-bucket/my-prefix/jpgs/sample.jpg. If the prefix is not specified, sample.jpg would be loaded to http://s3.amazonaws.com/my-bucket/jpgs/sample.jpg. You can specify a prefix by adding the prefix option in the manifest.

Important

We do not include a forward slash (/) automatically. If you don't include the slash at the end of the value for prefix, the value is concatenated to the file name. For example, if your prefix is images and you import the file sample.jpg, your key would become imagessample.jpg instead of images/sample.jpg.

prefix: my-prefix/

Working with Amazon S3 Substitutions

The substitutions option is helpful for changing file names, appending prefixes or suffixes to file names, or other name changes during an Amazon S3 import or export job. For import jobs, use the substitutions option to change object key names. For export jobs, use the substitutions option to change names to write to your file system. For example, use the following entry to replace all the uppercase "A", "B" and "C" letters in your file names and directories with lowercase letters on the object key names before the data is uploaded to your Amazon S3 bucket.

The substitutions option can only be used when manifestVersion is set to 2.0, and is not available for Amazon EBS or Amazon Glacier import jobs.

substitutions: 
    "A" : "a"
    "B" : "b"
    "C" : "c"

Important

Avoid attempting to replace an entire value with an empty string. For import jobs, an empty string causes the import to fail and report a "400 InvalidSubstitution" error in the log. Also, avoid substitutions that result in multiple files mapping to the same file name. For import jobs, when multiple files map to the same file name, the behavior is undefined. For export jobs, when multiple files map to the same file name, the object falls into the recovery process. For more information about the recovery process, see Collecting Files That Have Naming Errors.

The substitutions option is applied after any prefix options are applied. First, the prefix option determines what Amazon S3 data to copy, then, the substitutions option determines the value to write. For export jobs, if a targetDirectory option is specified, the substitutions option is applied to the targetDirectory value while it is writing to your device.

Example: Using Both Prefix and Substitutions Options

prefix: myprefix$COLON$
substitutions:
    "$COLON$" : ":"
    "$QUOTE$" : "\""

When you use the above options, you can enter a fully qualified file name on your device, such as:

/backup/$COLON$/$QUOTE$computer1$QUOTE$/c$COLON$/image.dat

AWS Import/Export performs the substitution and the file name becomes the following object name on Amazon S3.

myprefix:backup/:/"computer1"/c:/image.dat

Example: Using Substitutions, Prefix, and targetDirectory Options

substitutions:
    ":" : "$COLON$"
    "," : "$COMMA$"   
operations:
  - exportBucket: mybucket
    prefix: myprefix,
    targetDirectory: data,

When you use the above options, you can enter an Amazon S3 object name, such as:

mybucket/myprefix,/backup/:/computer1/c:/image.dat

AWS Import/Export performs the substitution, and the object name becomes the following fully qualified file name on your device.

/data$COMMA$/myprefix$COMMA$/backup/$COLON$/computer1/c$COLON$/image.dat

Example: Changing Name Space Encoding

In most cases the file system name space requirements are more restrictive than the Amazon S3 name space requirements, and you might need to map between valid file system names and the Amazon S3 key name space. For an import job, you can take advantage of the more expressive name space support in Amazon S3 by encoding the file name with information that is decoded before upload. In the following example, character encoding for a colon is converted to a colon in the Amazon S3 name space:

substitutions:
    $COLON$ : ":"

For an export job, you can handle characters that don't map to the file system.

substitutions:
    ":" : "$COLON$"

Mapping Uppercase Characters to Lowercase Characters

You can define a rule to substitute all uppercase characters for file names with the equivalent lowercase characters for object names in your import job. For example, use the following entry to replace all the uppercase characters in your file names with lowercase letters for an entire alphabet. List all the letters in the alphabet (you need to specify each one) with the uppercase letters on the left side of the option parameter and lowercase letters on the right side of the option parameter (where "..." represents all the characters between C and Y):

substitutions:
    "A" : "a"
    "B" : "b"
    "C" : "c"
       ...
    "Y" : "y"
    "Z" : "z"

For more information, see the substitutions option in Common Manifest File Options.

Mapping File Directories to the Amazon S3 Root

Amazon S3 performs well even when there are millions of files in the same bucket. To import your data efficiently into Amazon S3 using AWS Import/Export, you might decide to eliminate your subdirectories. If you name your directories carefully, such that none of the names of the directories are substrings of your file names, you can use the substitutions manifest option to remove the directory from the key name. The following example assumes you have a directory structure that divides your data across the three subdirectories, ZZ1, ZZ2, ZZ3 in your file system.

ZZ1/
ZZ2/
ZZ3/

To remove the directory name from the Amazon S3 key names, define the following substitutions option in your manifest file:

substitutions:
    "ZZ1/" : ""
    "ZZ2/" : ""
    "ZZ3/" : ""

All of the files will be stored in the Amazon S3 bucket root.

Important

None of the files within the subdirectories should contain the substitutions strings in their file names (such as "ZZ1/", "ZZ2/", or "ZZ3/").

If two files have the same name, both files are uploaded to Amazon S3, but you will only retain the bytes of the last file transferred.

Use the forward slash (/) as the file separator character. Don't use the back-slash (\) or double back-slash (\\).