| « PreviousNext » | |
   | Did this page help you? Yes | No | Tell us about it... |
Managing File Exports
Topics
You may need to limit which files are exported from a bucket. For instance, you might not want all of the files in a bucket, or you may have too many files for a single device and need to export to multiple devices.
This section describes how to configure the operations
manifest file option to manage those tasks.
If you need to export files from multiple buckets, you can specify multiple exportBucket options in the same manifest file.
Operations Export Manifest Option
The operations manifest option specifies a list of one
or more export operations. AWS Import/Export exports keys in alphabetical order.
That is, it carries out the operations in the order listed and then in the
alphabetical order of the keys affected by each export operation. The files on
the storage device remain in strict alphabetical order even if the data to
export exceeds the capacity of the storage device; AWS Import/Export does not,
for example, squeeze in an extra file or two that are out of order just because
there is room on the storage device. This alphabetical order enables you to
continue the export easily: create a new export job using the key of the last
file exported as the beginMarker in your new export
job.
The operations subfields prefix,
beginMarker, and endMarker
limit the data exported from a bucket.
|
|
Restricts the keys exported to those located in the path
specified by |
|
|
Limits the keys exported to those that occur alphabetically after this option value. |
|
|
Limits the keys exported to those that occur alphabetically before and including this value. |
AWS Import/Export uses the entire key when naming the objects exported to your
storage device. For example, AWS Import/Export would map the key
starwars.s3.amazonaws.com/images/jedi.jpg to the
following file on your storage device
/starwars/images/jedi.jpg. For Microsoft Windows,
AWS Import/Export would map the same key to
\starwars\images\jedi.jpg.
The following sections talk in depth about some of the export fields you use
with operations.
Limiting the Exported Data
The following export manifest options enable you to export a subset of the
keys in a bucket (specified by exportBucket):
|
|
A subfield of |
|
|
A subfield of |
|
|
A subfield of |
The manifest options beginMarker, and
endMarker, and prefix can be
used together or separately.
To export keys occurring alphabetically between two key names
Use beginMarker and
endMarker to specify the beginning and the ending
of the keys to export.
The export job starts with the key after the
beginMarker. In other words, the
beginMarker is not included in the export job,
although the endMarker is included. For example, the
following options export all of the keys that occur alphabetically between
html/index.html and
images/kate.jpg.
The export will not include html/index.html, but will
include images/kate.jpg.
operations:
- exportBucket: mybucket
beginMarker: html/index.html
endMarker: images/kate.jpgNote
It’s possible to construct an export operation that includes no keys. For
example, if you specify image for prefix
and foo for endMarker, there would be no
matching keys because foo comes alphabetically before
image.
Exporting Data From Multiple Buckets
To export multiple subsets of data from one or more buckets, specify multiple
exportBucket operations followed by appropriate
export options. Each dash (-) after operations specifies
a new set of data to export. Each entry requires an
exportBucket manifest option. The following example
shows four data sets to export from three different buckets:
mybucket, lost, and
threescompany.
All buckets must be in the same Amazon S3 region and you must have read permissions on each bucket and each object.
To export data from more than one bucket
Use exportBucket to specify each bucket you want to
export data from.
In the following example, AWS Import/Export exports all keys under
/images in mybucket to
starwars/image-backup on the storage device.
AWS Import/Export also exports all images alphabetically between
html/ and images/kate.jpg
(including kate.jpg).
operations:
- exportBucket: mybucket
prefix: images
beginMarker: images/starwars/jedi.jpg
endMarker: images/starwars/masterwindoo.jpg
- exportBucket: mybucket
prefix: html
- exportBucket: lost
- exportBucket: threescompany
prefix: images
endMarker: images/characters/mrfurley.jpgThe order in which AWS Import/Export exports keys is the order of the operations
you specify in the operations manifest option, and then
within each operation, by the alphabetical order of the keys to be exported. In
the previous procedure, for example, AWS Import/Export exports all of the keys
from mybucket in alphabetical order, and then from
lostbucket in alphabetical order.
Note
AWS Import/Export exports keys in parallel, but the finished order of files on
the storage device is by operation and then alphabetical. This functionality
becomes important when the storage device has insufficient capacity to hold
all of the data that needs to be exported. This ordering enables you to
easily continue the export with a new export job using the key of the last
key exported as your beginMarker in your new export
job.
Specifying the Target Directory
AWS Import/Export gives you the ability to specify the location for the exported
files on your storage device using the export manifest options
operations and
targetDirectory.
To export data to a specific directory
Use the targetDirectory to specify the directory on
the storage device you want to export files into.
In the following example, AWS Import/Export exports the contents of the bucket
named bucket-data-comes-from to the directory,
myexport/stuff, on the storage device.
The default targetDirectory is the bucket name.
operations:
- exportBucket: bucket-data-comes-from
targetDirectory: myexport/stuff
