Organizing Files in Your Amazon S3 Bucket - Amazon Elastic Transcoder

Save costs and get more features with AWS Elemental MediaConvert

MediaConvert is a newer file-based video transcoding service that provides a comprehensive suite of advanced transcoding features, with on-demand rates starting at $0.0075/minute. Read more.

Already using Amazon Elastic Transcoder? It's simple to migrate to MediaConvert. For more information, see this overview which includes valuable information about the migration process and links to additional resources.

Organizing Files in Your Amazon S3 Bucket

Amazon Elastic Transcoder integrates with Amazon S3 to store input and output files. The 100-bucket storage limit in Amazon S3 affects how you organize and manage your Elastic Transcoder files.

When you work with Amazon S3, we recommend that you mimic a regular file system. You can do this by using the OutputKeyPrefix in Elastic Transcoder to add file paths to your output file. Mimicking a file system allows you to use the prefix and delimiter parameters in the Amazon S3 API to find your files.

For example, suppose you are creating a user-generated content (UGC) site that takes in videos for many customers and transcodes them for use on several common devices. To store the videos, you could organize your input and output buckets like this:

Input:   /DOC-EXAMPLE-BUCKET/input/movie.mp4
Output:  /DOC-EXAMPLE-BUCKET/output/movie/<format>/movie.mp4

This enables you to store separate formats so you can locate them easily. For example, say you have two customers using your service to transcode different movies into multiple formats. You can organize your bucket like this:

Customer 1:
/DOC-EXAMPLE-BUCKET/<1>/output/movie/avi/movie.avi
/DOC-EXAMPLE-BUCKET/<1>/output/movie/mkv/movie.mkv
/DOC-EXAMPLE-BUCKET/<1>/output/movie/hls400k/movie.m3u8
  - playlist file for the hls400k version of movie
/DOC-EXAMPLE-BUCKET/<1>/output/movie/hls400k/movie.ts
  - video file for the hls400k version of movie
/DOC-EXAMPLE-BUCKET/<1>/output/movie/hls400k/movie.png 
  - thumbnail file for the hls400k version of movie
/DOC-EXAMPLE-BUCKET/<1>/output/myOthermovie/hls1000k/movie.m3u8
  - playlist file for the hls1000k version of movie
/DOC-EXAMPLE-BUCKET/<1>/output/myOthermovie/hls1000k/movie.ts
  - video file for the hls1000k version of movie
/DOC-EXAMPLE-BUCKET/<1>/output/myOthermovie/hls1000k/movie.png
  - thumbnail file for the hls1000k version of movie
Customer 2:
/DOC-EXAMPLE-BUCKET/<2>/output/somemovie/avi/somemovie.avi
/DOC-EXAMPLE-BUCKET/<2>/output/somemovie/mkv/somemovie.mkv
/DOC-EXAMPLE-BUCKET/<2>/output/somemovie/hls400k/somemovie.m3u8
  - playlist file for the hls400k version of somemovie
/DOC-EXAMPLE-BUCKET/<2>/output/somemovie/hls400k/somemovie.ts
  - video file for the hls400k version of somemovie
/DOC-EXAMPLE-BUCKET/<2>/output/somemovie/hls400k/somemovie.png
  - thumbnail file for the hls400k version of somemovie
/DOC-EXAMPLE-BUCKET/<2>/output/myOthermovie/hls1000k/movie.m3u8
  - playlist file for the hls1000k version of movie
/DOC-EXAMPLE-BUCKET/<2>/output/myOthermovie/hls1000k/movie.ts
  - video file for the hls1000k version of movie
/DOC-EXAMPLE-BUCKET/<2>/output/myOthermovie/hls1000k/movie.png
  - thumbnail file for the hls1000k version of movie

We recommend that you end your OutputKeyPrefix with a '/' so that the last part of the OutputKeyPrefix and the OutputKey don't run together. Otherwise, your files will look like this:

/DOC-EXAMPLE-BUCKET/output/<movie title>/mkvmovie.mkv

We also recommend that you keep your input media in a single bucket per region, and that you keep transcoded media and thumbnails in a separate bucket in the same region. This approach helps you avoid cross-region transfer fees between your Amazon S3 bucket and Elastic Transcoder.

Note

If you expect to have more than a hundred requests per second accessing your Amazon S3 buckets, we recommend you follow Amazon S3 performance considerations when designing your system.