View a markdown version of this page

Data mesh with Amazon DataZone - Data and Analytics Reference Architecture Diagrams
Data mesh with Amazon DataZone diagramFurther reading

Data mesh with Amazon DataZone

Publication date: November 20, 2024 (Diagram history)

Data mesh is a decentralized architectural and organizational framework that helps organizations accelerate innovation and drive business value. This architecture shows how to use Amazon DataZone to build a data mesh-based data solution. In this architecture, a producer account ingests and transforms raw data, then registers it as assets in the Amazon DataZone catalog. A central governance account hosts the Amazon DataZone domain and data portal, where consumers discover assets and request access through subscriptions. After the asset owner approves a subscription request, consumers access the data in the consumer account for analytics and machine learning.

Data mesh with Amazon DataZone diagram

Architecture diagram showing a data mesh using Amazon DataZone with Amazon S3, AWS Glue, Amazon Redshift, and SageMaker AI.

The following steps describe the architecture:

  1. Gather data from sources across the enterprise through databases, file shares, edge devices, logs, social networks, SaaS applications, and streaming media.

  2. Based on the source system and end user requirements, ingest raw data using Amazon AppFlow, AWS Database Migration Service, Amazon Kinesis, AWS IoT Core, and Amazon Managed Streaming for Apache Kafka.

  3. In the producer account, transform raw data using AWS Glue. Store metadata in AWS Glue Data Catalog, measure data quality using AWS Glue Data Quality, and register data in Amazon S3, Amazon Redshift, Amazon RDS, and third-party sources as assets in the Amazon DataZone catalog using data source jobs.

  4. The central governance account hosts the Amazon DataZone domain and the related data portal. Associate AWS accounts of data producers and consumers with the Amazon DataZone domain, and create projects under related domain units.

  5. End users log in to the Amazon DataZone data portal using IAM credentials or single sign-on (SSO) through https://docs.aws.amazon.com/singlesignon/latest/userguide/. They search, filter, and view asset information including data quality, business metadata, and technical metadata.

  6. When end users find assets of interest, they request access using the subscription feature of Amazon DataZone. The asset owner approves or rejects the request based on validity.

  7. After the subscription request is granted and fulfilled, access the asset in the consumer account for AI/ML model development using SageMaker AI, and for analytics and reporting use Athena, Amazon Redshift, and AWS Quick Sight.

Further reading

For additional information, refer to the following resources: