Improve Spark performance with Amazon S3

Amazon EMR offers features to help optimize performance when using Spark to query, read and write data saved in Amazon S3.

S3 Select can improve query performance for CSV and JSON files in some applications by "pushing down" processing to Amazon S3.

The EMRFS S3-optimized committer is an alternative to the OutputCommitter class, which uses the multipart uploads feature of EMRFS to improve performance when writing Parquet files to Amazon S3 using Spark SQL, DataFrames, and Datasets.

Topics

• Use S3 Select with Spark to improve query performance
• Use the EMRFS S3-optimized committer
• Use the EMRFS S3-optimized commit protocol
• Retry Amazon S3 requests with EMRFS