Configure an output location - Amazon EMR

Configure an output location

The most common output format of an Amazon EMR cluster is as text files, either compressed or uncompressed. Typically, these are written to an Amazon S3 bucket. This bucket must be created before you launch the cluster. You specify the S3 bucket as the output location when you launch the cluster.

For more information, see the following topics:

Create and configure an Amazon S3 bucket

Amazon EMR (Amazon EMR) uses Amazon S3 to store input data, log files, and output data. Amazon S3 refers to these storage locations as buckets. Buckets have certain restrictions and limitations to conform with Amazon S3 and DNS requirements. For more information, go to Bucket Restrictions and Limitations in the Amazon Simple Storage Service Developers Guide.

To create a an Amazon S3 bucket, follow the instructions on the Creating a bucket page in the Amazon Simple Storage Service Developers Guide.

Note

If you enable logging in the Create a Bucket wizard, it enables only bucket access logs, not cluster logs.

Note

For more information on specifying Region-specific buckets, refer to Buckets and Regions in the Amazon Simple Storage Service Developer Guide and Available Region Endpoints for the AWS SDKs .

After you create your bucket you can set the appropriate permissions on it. Typically, you give yourself (the owner) read and write access and authenticated users read access. See Bucket policies and user policies for instructions.

Required Amazon S3 buckets must exist before you can create a cluster. You must upload any required scripts or data referenced in the cluster to Amazon S3. The following table describes example data, scripts, and log file locations.

Information Example Location on Amazon S3
script or program s3://DOC-EXAMPLE-BUCKET1/script/MapperScript.py
log files s3://DOC-EXAMPLE-BUCKET1/logs
input data s3://DOC-EXAMPLE-BUCKET1/input
output data s3://DOC-EXAMPLE-BUCKET1/output