Managing File Exports
You may need to limit which files are exported from a bucket. For instance, you might not want all of the files in a bucket, or you may have too many files for a single device and need to export to multiple devices.
This section describes how to configure the
file option to manage those tasks.
If you need to export files from multiple buckets, you can specify multiple exportBucket options in the same manifest file.
Operations Export Manifest Option
operations manifest option specifies a list of one or more
export operations. AWS Import/Export Disk exports keys in alphabetical order. That is, it
carries out the operations in the order listed and then in the alphabetical order of the
keys affected by each export operation. The files on the storage device remain in strict
alphabetical order even if the data to export exceeds the capacity of the storage device;
Disk does not, for example, squeeze in an extra file or two that are out of
order just because there is room on the storage device. This alphabetical order enables
you to continue the export easily: create a new export job using the key of the last file
exported as the
beginMarker in your new export job.
endMarker limit the data
exported from a bucket.
Restricts the keys exported to those located in the path specified by
Limits the keys exported to those that occur alphabetically after this option value.
A subfield of
AWS Import/Export Disk uses the entire key when naming the objects exported to your storage
device. For example, AWS Import/Export Disk would map the key
starwars.s3.amazonaws.com/images/jedi.jpg to the following file on
your storage device
/starwars/images/jedi.jpg. For Microsoft Windows,
AWS Import/Export Disk would map the same key to
The following sections talk in depth about some of the export fields you use with
Limiting the Exported Data
endMarker to specify
the beginning and the ending of the keys to export.
The export job starts with the key after the
other words, the
beginMarker is not included in the export job,
endMarker is included. For example, the following
options export all of the keys that occur alphabetically between
The export will not include
html/index.html, will include
images/kate.jpg, and will not include
operations: - exportBucket: mybucket beginMarker: html/index.html endMarker: images/kate.jpg
It’s possible to construct an export operation that includes no keys. For example,
if you specify
endMarker, there would be no matching keys because
foo comes alphabetically before
Exporting Data From Multiple Buckets
To export multiple subsets of data from one or more buckets, specify multiple
exportBucket operations followed by appropriate export options.
Each dash (-) after
operations specifies a new set of data to
export. Each entry requires an
exportBucket manifest option. The
following example shows four data sets to export from three different buckets:
All buckets must be in the same Amazon S3 region and you must have read permissions on each bucket and each object.
To export data from more than one bucket
exportBucket to specify each bucket you want to export
In the following example, AWS Import/Export exports all keys under
starwars/image-backup on the storage device.
operations: - exportBucket: mybucket prefix: images beginMarker: images/starwars/jedi.jpg endMarker: images/starwars/masterwindoo.jpg - exportBucket: lost - exportBucket: threescompany prefix: images endMarker: images/characters/mrfurley.jpg
The order in which AWS Import/Export exports keys is the order of the operations you
specify in the
operations manifest option, and then within each
operation, by the alphabetical order of the keys to be exported. In the previous
procedure, for example, AWS Import/Export exports all of the keys from
mybucket in alphabetical order, and then from
AWS Import/Export exports keys in parallel, but the finished order of files on the
storage device is by operation and then alphabetical. This functionality becomes
important when the storage device has insufficient capacity to hold all of the data that
needs to be exported. This ordering enables you to easily continue the export with a new
export job using the key of the last key exported as your
beginMarker in your new export job.
Specifying the Target Directory
AWS Import/Export gives you the ability to specify the location for the exported files on
your storage device using the export manifest options
To export data to a specific directory
targetDirectory to specify the directory on the
storage device you want to export files into.
In the following example, AWS Import/Export exports the contents of the bucket named
bucket-data-comes-from to the directory,
myexport/stuff, on the storage device.
targetDirectory is the bucket name.
operations: - exportBucket: bucket-data-comes-from targetDirectory: myexport/stuff