View a markdown version of this page

SUS04-BP02 Select appropriate file formats, compression, and software implementation - AWS Well-Architected Framework

SUS04-BP02 Select appropriate file formats, compression, and software implementation

Choose file formats, encodings, compression techniques, and software implementations that minimize the storage and processing resources your workload consumes while maintaining required functionality.

Desired outcome:

  • You minimize the resources used to store, transport, and process data.

  • You choose file formats and compression based on how the data is written, read, and consumed.

  • You store data in formats that balance storage efficiency, processing performance, and accessibility for your workload, reducing storage needs and making processing more efficient.

  • You process data efficiently, favoring efficient algorithms and considering how the data is consumed downstream.

Common anti-patterns:

  • Using general-purpose formats (such as CSV or JSON) for all data regardless of size or access patterns, and neglecting to consider file format and encoding options such as Parquet for analytical workloads [1].

  • Applying the same compression technique to all data without considering its characteristics and decompression volumes.

  • Ignoring column-oriented formats for analytical workloads.

  • Using inefficient serialization/deserialization methods that consume excessive CPU.

  • Failing to compress data before network transmission.

  • Ignoring code efficiency and the impact of frameworks and dependencies on resource consumption.

  • Overlooking algorithmic complexity (Big O notation) when implementing data processing tasks.

  • Choosing pre-built implementations without evaluating their efficiency for your specific use case.

  • Failing to conduct performance profiling and offset testing to identify optimization opportunities.

Benefits of establishing this best practice:

  • Lowers the required share on hardware related impacts, from raw material extraction to end of life treatment (Scope 3 emissions).

  • Lowers the required share on energy for operating infrastructure (Scope 2 emissions).

  • Minimizes data transfer volume and network resource requirements.

Level of risk exposed if this best practice is not established: Medium

Implementation guidance

Optimizing file formats and compression requires analyzing your data characteristics, access patterns, and processing needs. By selecting the right combination of formats and compression techniques, you can significantly reduce storage requirements and improve processing efficiency on the data.

Implementation steps

  1. Identify the most appropriate storage service for the intended use case.

  2. Analyze your data characteristics and access patterns. For data in S3 buckets, consider performing an S3 storage class analysis.

    1. Identify your primary data usage patterns (transactional, analytical, and archive).

    2. Assess your data structure and typical query operations.

    3. Evaluate your performance and availability requirements.

  3. Select appropriate file formats for your use cases.

    1. Evaluate columnar formats such as Apache Parquet or ORC for analytical workloads, which can significantly reduce storage requirements and improve query performance by allowing column-level operations.

    2. Consider binary formats such as protocol buffers or Apache Avro for data serialization.

    3. Use specialized formats for specific domains (for example, GLTF for 3D assets or WebP for images).

  4. Index your data appropriately for the access patterns.

    1. Choose the right Amazon DynamoDB partition key to support your query types.

    2. Design Amazon Aurora tables to best suit your usage patterns.

  5. Implement efficient compression techniques.

    1. Select compression algorithms based on data characteristics and access patterns.

    2. Consider column-level encoding for analytical formats.

    3. Use tiered compression strategies based on data age and access frequency.

  6. Optimize for AWS service integration.

    1. Use formats that work efficiently with Amazon Athena and AWS Glue.

    2. Implement S3 Transfer Acceleration for efficient data uploads.

    3. Configure Amazon CloudFront to serve compressed files.

  7. Take samples of data and process that data instead of processing all data where it fits the business case (for example, you can take samples with AWS X-Ray).

  8. Monitor and optimize data operations.

    1. Track storage usage and query performance across different formats.

    2. Evaluate compression ratios and processing overhead.

    3. Continuously refine your format and compression strategies based on results.

  9. Optimize code efficiency and implementation.

    1. Evaluate the impact of frameworks and dependencies on resource consumption.

    2. Consider algorithmic complexity (Big O notation) when implementing data processing tasks.

    3. Choose language implementations which maximize the level of parallelism.

    4. Use updated third-party libraries that use optimizations such as parallelism and hardware-level instructions.

    5. Conduct performance profiling to identify bottlenecks and optimization opportunities.

Resources

Related best practices:

Related documents:

Related videos:

Related examples: