Data engineering principles - AWS Prescriptive Guidance

Data engineering principles

We recommend that you adopt the principles in the following table when you build an architecture for a modern data pipeline.

Principle

Example

Use case

Flexibility

Use microservices

FastGo enjoys flexibility and scalability with a microservices architecture on AWS (AWS case study)

Reproducibility

Use infrastructure as code (IaC) to deploy your services

Part 3: How NatWest Group built auditable, reproducible, and explainable ML models with Amazon SageMaker (AWS Machine Learning Blog)

Reusability

Use libraries and references in a shared manner

Create and reuse governed datasets in Amazon QuickSight with new Dataset-as-a-Source feature (AWS Big Data Blog)

Scalability

Choose service configurations to accommodate any data load

Designing a data lake for growth and scale on the AWS Cloud (AWS Prescriptive Guidance)

Auditability

Keep an audit trail by using logs, versions, and dependencies

How Parametric Built Audit Surveillance using AWS Data Lake Architecture (AWS Architecture Blog)