Menu
Amazon EMR
Developer Guide

Intermediate Compression (Hadoop 20.205)

Hadoop sends data between the mappers and reducers in its shuffle process. This network operation is a bottleneck for many clusters. To reduce this bottleneck, Amazon EMR enables intermediate data compression by default. Because it provides a reasonable amount of compression with only a small CPU impact, we use the Snappy codec.

You can modify the default compression settings with a bootstrap action. For more information about using bootstrap actions, see (Optional) Create Bootstrap Actions to Install Additional Software.

The following table presents the default values for the parameters that affect intermediate compression.

Parameter Value
mapred.compress.map.output true
mapred.map.output.compression.codec org.apache.hadoop.io.compress.SnappyCodec