Manipulating Amazon S3 Object Key Names for Disk
The file path on your Disk storage device determines the corresponding object key
when AWS imports your data to an Amazon S3 bucket. For example, if your storage device has an
images top-level folder with a file
sample.jpg, the uploaded file/object key will be
You can use the substitutions option to change object key names when importing to Amazon S3. For example, you can add prefixes or suffixes to file names. For more information, see Working with Amazon S3 Substitutions.
Specifying Key Prefix
The AWS Import/Export prefix mechanism allows you to create a logical grouping of
the objects in a bucket. The
prefix value is similar to a
directory name that enables you to store similar data under the same directory in a
bucket. For example, if your Amazon S3 bucket name is
my-bucket, and you set
my-prefix/, and the file on your storage
would be loaded to
the prefix is not specified,
sample.jpg would be loaded to
http://s3.amazonaws.com/my-bucket/jpgs/sample.jpg. You can
specify a prefix by adding the
prefix option in the manifest.
We do not include a forward slash (/) automatically. If you don't include the
slash at the end of the value for
prefix, the value is
concatenated to the file name. For example, if your prefix is
images and you import the file
your key would become
imagessample.jpg instead of
Working with Amazon S3 Substitutions
substitutions option is helpful for changing file names,
appending prefixes or suffixes to file names, or other name changes during an Amazon S3
import or export job. For import jobs, use the
option to change object key names. For export jobs, use the
substitutions option to change names to write to your file
system. For example, use the following entry to replace all the uppercase "A", "B" and
"C" letters in your file names and directories with lowercase letters on the object key
names before the data is uploaded to your Amazon S3 bucket.
substitutions option can only be used when
manifestVersion is set to 2.0, and is not
available for Amazon EBS or Amazon Glacier import jobs.
substitutions: "A" : "a" "B" : "b" "C" : "c"
Avoid attempting to replace an entire value with an empty string. For import jobs, an empty string causes the import to fail and report a "400 InvalidSubstitution" error in the log. Also, avoid substitutions that result in multiple files mapping to the same file name. For import jobs, when multiple files map to the same file name, the behavior is undefined. For export jobs, when multiple files map to the same file name, the object falls into the recovery process. For more information about the recovery process, see Collecting Files That Have Naming Errors.
substitutions option is applied after any
prefix options are applied. First, the
prefix option determines what Amazon S3 data to copy, then, the
substitutions option determines the value to write. For export
jobs, if a
targetDirectory option is specified, the substitutions
option is applied to the
targetDirectory value while it is
writing to your device.
Example: Using Both Prefix and Substitutions Options
prefix: myprefix$COLON$ substitutions: "$COLON$" : ":" "$QUOTE$" : "\""
When you use the preceding options, you can enter a fully qualified file name on your device, such as:
AWS Import/Export performs the substitution and the file name becomes the following object name on Amazon S3.
Example: Using Substitutions, Prefix, and targetDirectory Options
substitutions: ":" : "$COLON$" "," : "$COMMA$" operations: - exportBucket: mybucket prefix: myprefix, targetDirectory: data,
When you use the preceding options, you can enter an Amazon S3 object name, such as:
AWS Import/Export performs the substitution, and the object name becomes the following fully qualified file name on your device.
Example: Changing Name Space Encoding
In most cases the file system name space requirements are more restrictive than the Amazon S3 name space requirements, and you might need to map between valid file system names and the Amazon S3 key name space. For an import job, you can take advantage of the more expressive name space support in Amazon S3 by encoding the file name with information that is decoded before upload. In the following example, character encoding for a colon is converted to a colon in the Amazon S3 name space:
substitutions: $COLON$ : ":"
For an export job, you can handle characters that don't map to the file system.
substitutions: ":" : "$COLON$"
Mapping Uppercase Characters to Lowercase Characters
You can define a rule to substitute all uppercase characters for file names with the equivalent lowercase characters for object names in your import job. For example, use the following entry to replace all the uppercase characters in your file names with lowercase letters for an entire alphabet. List all the letters in the alphabet (you need to specify each one) with the uppercase letters on the left side of the option parameter and lowercase letters on the right side of the option parameter (where "..." represents all the characters between C and Y):
substitutions: "A" : "a" "B" : "b" "C" : "c" ... "Y" : "y" "Z" : "z"
For more information, see the
substitutions option in Common Manifest File Options.
Mapping File Directories to the Amazon S3 Root
Amazon S3 performs well even when there are millions of files in the same bucket. To
import your data efficiently into Amazon S3 using AWS Import/Export, you might decide to
eliminate your subdirectories. If you name your directories carefully, such that none
of the names of the directories are substrings of your file names, you can use the
substitutions manifest option to remove the directory from
the key name. The following example assumes you have a directory structure that
divides your data across the three subdirectories,
ZZ3 in your file system.
ZZ1/ ZZ2/ ZZ3/
To remove the directory name from the Amazon S3 key names, define the following
substitutions option in your manifest file:
substitutions: "ZZ1/" : "" "ZZ2/" : "" "ZZ3/" : ""
All of the files will be stored in the Amazon S3 bucket root.
None of the files within the subdirectories should contain the
substitutions strings in their file names (such as
If two files have the same name, both files are uploaded to Amazon S3, but you will only retain the bytes of the last file transferred.
Use the forward slash (
/) as the file separator
character. Don't use the back-slash (
\) or double back-slash