10 – Choose the best-performing file format and partitioning
How do you select the best-performing file formats and partitioning? Selecting the best-performing file format and data partitioning for data-at-rest can have a large impact on the overall analytics workload efficiency.
| ID | Priority | Best practice |
|---|---|---|
|
☐ BP 10.1 |
Recommended | Select format based on data write frequency and patterns for append-only compared to in-place update. |
|
☐ BP 10.2 |
Recommended | Choose data formatting based on your data access pattern |
|
☐ BP 10.3 |
Recommended | Utilize compression techniques to both decrease storage requirements and enhance I/O efficiency. |
|
☐ BP 10.4 |
Recommended | Partition your data to enable efficient data pruning and reduce unnecessary file reads. |
For more details, refer to the following information:
-
Amazon Redshift Database Developer Guide: Creating data files for queries in Amazon Redshift Spectrum
-
Amazon EMR Release Guide: Hudi
-
AWS Big Data Blog: Apply record level changes from relational databases to Amazon S3 data lake
using Apache Hudi on Amazon EMR and AWS Database Migration service
