| « PreviousNext » | |
   | Did this page help you? Yes | No | Tell us about it... |
Manipulating Amazon S3 Object Key Names
The file path on your storage device determines the corresponding object key
when AWS loads your data to an Amazon S3 bucket. For example, if your storage
device has an images top-level folder with a file sample.jpg, the uploaded
file/object key will be images/sample.jpg.
You can use the substitutions option to change object key names when importing to Amazon S3. For example, you can add prefixes or suffixes to file names. For more information, see Working with Amazon S3 Substitutions.
Specifying Key Prefix
The AWS Import/Export prefix mechanism allows you to create a logical
grouping of the objects in a bucket. The prefix value
is similar to a directory name that enables you to store similar data under
the same directory in a bucket. For example, if your Amazon S3 bucket name
is my-bucket, and you set prefix to
my-prefix/, and the file on your storage device
is /jpgs/sample.jpg, then
sample.jpg would be loaded to
http://s3.amazonaws.com/my-bucket/my-prefix/jpgs/sample.jpg.
If the prefix is not specified, sample.jpg would be
loaded to
http://s3.amazonaws.com/my-bucket/jpgs/sample.jpg.
You can specify a prefix by adding the prefix option in
the manifest.
Important
We do not include a forward slash (/) automatically. If you don't
include the slash at the end of the value for
prefix, the value is concatenated to the file
name. For example, if your prefix is images and you
import the file sample.jpg, your key would become
imagessample.jpg instead of
images/sample.jpg.
prefix: my-prefix/
Working with Amazon S3 Substitutions
The substitutions option is helpful for changing
file names, appending prefixes or suffixes to file names, or other name
changes during an Amazon S3 import or export job. For import jobs, use the
substitutions option to change object key names.
For export jobs, use the substitutions option to
change names to write to your file system. For example, use the following
entry to replace all the uppercase "A", "B" and "C" letters in your file
names and directories with lowercase letters on the object key names before
the data is uploaded to your Amazon S3 bucket.
The substitutions option can only be used when
manifestVersion is set to 2.0, and
is not available for Amazon EBS or Amazon Glacier import jobs.
substitutions:
"A" : "a"
"B" : "b"
"C" : "c"Important
Avoid attempting to replace an entire value with an empty string. For import jobs, an empty string causes the import to fail and report a "400 InvalidSubstitution" error in the log. Also, avoid substitutions that result in multiple files mapping to the same file name. For import jobs, when multiple files map to the same file name, the behavior is undefined. For export jobs, when multiple files map to the same file name, the object falls into the recovery process. For more information about the recovery process, see Collecting Files That Have Naming Errors.
The substitutions option is applied after any
prefix options are applied. First, the
prefix option determines what Amazon S3 data to
copy, then, the substitutions option determines the
value to write. For export jobs, if a targetDirectory
option is specified, the substitutions option is applied to the
targetDirectory value while it is writing to your
device.
Example: Using Both Prefix and Substitutions Options
prefix: myprefix$COLON$
substitutions:
"$COLON$" : ":"
"$QUOTE$" : "\""When you use the above options, you can enter a fully qualified file name on your device, such as:
/backup/$COLON$/$QUOTE$computer1$QUOTE$/c$COLON$/image.dat
AWS Import/Export performs the substitution and the file name becomes the following object name on Amazon S3.
myprefix:backup/:/"computer1"/c:/image.dat
Example: Using Substitutions, Prefix, and targetDirectory Options
substitutions:
":" : "$COLON$"
"," : "$COMMA$"
operations:
- exportBucket: mybucket
prefix: myprefix,
targetDirectory: data,When you use the above options, you can enter an Amazon S3 object name, such as:
mybucket/myprefix,/backup/:/computer1/c:/image.dat
AWS Import/Export performs the substitution, and the object name becomes the following fully qualified file name on your device.
/data$COMMA$/myprefix$COMMA$/backup/$COLON$/computer1/c$COLON$/image.dat
Example: Changing Name Space Encoding
In most cases the file system name space requirements are more restrictive than the Amazon S3 name space requirements, and you might need to map between valid file system names and the Amazon S3 key name space. For an import job, you can take advantage of the more expressive name space support in Amazon S3 by encoding the file name with information that is decoded before upload. In the following example, character encoding for a colon is converted to a colon in the Amazon S3 name space:
substitutions:
$COLON$ : ":"For an export job, you can handle characters that don't map to the file system.
substitutions:
":" : "$COLON$"Mapping Uppercase Characters to Lowercase Characters
You can define a rule to substitute all uppercase characters for file names with the equivalent lowercase characters for object names in your import job. For example, use the following entry to replace all the uppercase characters in your file names with lowercase letters for an entire alphabet. List all the letters in the alphabet (you need to specify each one) with the uppercase letters on the left side of the option parameter and lowercase letters on the right side of the option parameter (where "..." represents all the characters between C and Y):
substitutions:
"A" : "a"
"B" : "b"
"C" : "c"
...
"Y" : "y"
"Z" : "z"For more information, see the substitutions
option in Common Manifest File Options.
Mapping File Directories to the Amazon S3 Root
Amazon S3 performs well even when there are millions of files in the
same bucket. To import your data efficiently into Amazon S3 using AWS
Import/Export, you might decide to eliminate your subdirectories. If you
name your directories carefully, such that none of the names of the
directories are substrings of your file names, you can use the
substitutions manifest option to remove the
directory from the key name. The following example assumes you have a
directory structure that divides your data across the three
subdirectories, ZZ1, ZZ2,
ZZ3 in your file system.
ZZ1/ ZZ2/ ZZ3/
To remove the directory name from the Amazon S3 key names, define the
following substitutions option in your manifest
file:
substitutions:
"ZZ1/" : ""
"ZZ2/" : ""
"ZZ3/" : ""All of the files will be stored in the Amazon S3 bucket root.
Important
None of the files within the subdirectories should contain the
substitutions strings in their file names
(such as "ZZ1/",
"ZZ2/", or
"ZZ3/").
If two files have the same name, both files are uploaded to Amazon S3, but you will only retain the bytes of the last file transferred.
Use the forward slash (/) as the file
separator character. Don't use the back-slash
(\) or double back-slash
(\\).
