AWS Import/Export
Developer Guide (API Version 2010-06-03)
« PreviousNext »
View the PDF for this guide.Go to the AWS Discussion Forum for this product.Go to the Kindle Store to download this guide in Kindle format.Did this page help you?  Yes | No |  Tell us about it...

Managing File Exports

You may need to limit which files are exported from a bucket. For instance, you might not want all of the files in a bucket, or you may have too many files for a single device and need to export to multiple devices.

This section describes how to configure the operations manifest file option to manage those tasks.

If you need to export files from multiple buckets, you can specify multiple exportBucket options in the same manifest file.

Operations Export Manifest Option

The operations manifest option specifies a list of one or more export operations. AWS Import/Export exports keys in alphabetical order. That is, it carries out the operations in the order listed and then in the alphabetical order of the keys affected by each export operation. The files on the storage device remain in strict alphabetical order even if the data to export exceeds the capacity of the storage device; AWS Import/Export does not, for example, squeeze in an extra file or two that are out of order just because there is room on the storage device. This alphabetical order enables you to continue the export easily: create a new export job using the key of the last file exported as the beginMarker in your new export job.

The operations subfields prefix, beginMarker, and endMarker limit the data exported from a bucket.

prefix

Restricts the keys exported to those located in the path specified by prefix.

beginMarker

Limits the keys exported to those that occur alphabetically after this option value.

endMarker

Limits the keys exported to those that occur alphabetically before and including this value.

AWS Import/Export uses the entire key when naming the objects exported to your storage device. For example, AWS Import/Export would map the key starwars.s3.amazonaws.com/images/jedi.jpg to the following file on your storage device /starwars/images/jedi.jpg. For Microsoft Windows, AWS Import/Export would map the same key to \starwars\images\jedi.jpg.

The following sections talk in depth about some of the export fields you use with operations.

Limiting the Exported Data

The following export manifest options enable you to export a subset of the keys in a bucket (specified by exportBucket):

prefix

A subfield of operations that restricts the keys exported to those located in the path specified by prefix.

beginMarker

A subfield of operations that limits the keys exported to those that occur alphabetically after, not including, this option value.

endMarker

A subfield of operations that limits the keys exported to those that occur alphabetically before and including this value.

The manifest options beginMarker, and endMarker, and prefix can be used together or separately.

To export keys occurring alphabetically between two key names

Use beginMarker and endMarker to specify the beginning and the ending of the keys to export.

The export job starts with the key after the beginMarker. In other words, the beginMarker is not included in the export job, although the endMarker is included. For example, the following options export all of the keys that occur alphabetically between html/index.html and images/kate.jpg.

The export will not include html/index.html, but will include images/kate.jpg.

operations:
  - exportBucket: mybucket
    beginMarker: html/index.html
    endMarker: images/kate.jpg

Note

It’s possible to construct an export operation that includes no keys. For example, if you specify image for prefix and foo for endMarker, there would be no matching keys because foo comes alphabetically before image.

Exporting Data From Multiple Buckets

To export multiple subsets of data from one or more buckets, specify multiple exportBucket operations followed by appropriate export options. Each dash (-) after operations specifies a new set of data to export. Each entry requires an exportBucket manifest option. The following example shows four data sets to export from three different buckets: mybucket, lost, and threescompany.

All buckets must be in the same Amazon S3 region and you must have read permissions on each bucket and each object.

To export data from more than one bucket

Use exportBucket to specify each bucket you want to export data from.

In the following example, AWS Import/Export exports all keys under /images in mybucket to starwars/image-backup on the storage device. AWS Import/Export also exports all images alphabetically between html/ and images/kate.jpg (including kate.jpg).

operations:
  - exportBucket: mybucket
    prefix: images
    beginMarker: images/starwars/jedi.jpg
    endMarker: images/starwars/masterwindoo.jpg
  - exportBucket: mybucket
    prefix: html
  - exportBucket: lost
  - exportBucket: threescompany
    prefix: images
    endMarker: images/characters/mrfurley.jpg

The order in which AWS Import/Export exports keys is the order of the operations you specify in the operations manifest option, and then within each operation, by the alphabetical order of the keys to be exported. In the previous procedure, for example, AWS Import/Export exports all of the keys from mybucket in alphabetical order, and then from lostbucket in alphabetical order.

Note

AWS Import/Export exports keys in parallel, but the finished order of files on the storage device is by operation and then alphabetical. This functionality becomes important when the storage device has insufficient capacity to hold all of the data that needs to be exported. This ordering enables you to easily continue the export with a new export job using the key of the last key exported as your beginMarker in your new export job.

Specifying the Target Directory

AWS Import/Export gives you the ability to specify the location for the exported files on your storage device using the export manifest options operations and targetDirectory.

To export data to a specific directory

Use the targetDirectory to specify the directory on the storage device you want to export files into.

In the following example, AWS Import/Export exports the contents of the bucket named bucket-data-comes-from to the directory, myexport/stuff, on the storage device.

The default targetDirectory is the bucket name.

operations:
  - exportBucket: bucket-data-comes-from
    targetDirectory: myexport/stuff