View a markdown version of this page

DSSEC07-BP03 Track data lineage - Digital Sovereignty Lens
Implementation guidanceResources

DSSEC07-BP03 Track data lineage

Organizations in highly regulated industries implement data lineage tracking to demonstrate that data localization, data residency, and data privacy requirements are met. Data lineage provides visibility of the complete journey of data from source to consumption. This capability becomes essential for demonstrating audit readiness and maintaining trust in data-driven decision making.

Desired outcome:

  • Data movement, transformations, and dependencies are tracked automatically across data pipelines through lineage capabilities.

  • Organizations maintain visibility into data flows and possess audit-ready documentation of data handling practices.

Common anti-patterns:

  • Manual lineage documentation that becomes outdated quickly and fails to capture real-time data transformations and dependencies.

  • Siloed lineage tracking that only covers specific tools or solutions without providing full visibility across the data estate.

  • Reactive lineage capture that attempts to reconstruct data flows after issues occur rather than proactively tracking lineage during data processing.

Benefits of establishing this best practice:

  • Enhanced regulatory adherence through detailed audit trails that demonstrate data handling practices and support regulatory reporting requirements.

  • Improved data governance by providing visibility into data quality, transformation logic, and impact analysis for schema changes and system modifications.

  • Increased data trust by enabling data consumers to understand data origins, transformation history, and quality measures before making business decisions.

  • Reduced compliance costs through automated documentation and audit trail generation that removes manual effort and reduces audit preparation time.

Level of risk exposed if this best practice is not established: Medium

Implementation guidance

Establish a data lineage tracking strategy. Implement automated lineage capture using AWS services. Use services that integrate with your existing data processing tools and solutions. Deploy a centralized lineage repository that can store, query, and visualize complex data relationships and integrate with governance tools.

Key AWS services for data lineage implementation include Amazon SageMaker AI Catalog — the governance capability of Amazon SageMaker AI Unified Studio, built on Amazon DataZone — for centralized governance and lineage visualization, and AWS Glue for automated lineage capture from extract, transform, and load (ETL) processes. The same lineage capability remains available directly in Amazon DataZone.

Implementation steps

  1. Assess current data environment and lineage requirements: Conduct an inventory of your data sources, processing systems, and consumption patterns. Identify critical data flows that require lineage tracking based on regulatory requirements, business criticality, and data sensitivity levels. Document data classification levels and associated lineage requirements:

    • Highly sensitive data: Full column-level lineage with transformation details

    • Regulated data: Table-level lineage with processing metadata

    • Internal data: Basic flow tracking with key transformation points

  2. Enable data lineage in Amazon SageMaker AI Catalog or Amazon DataZone: Amazon SageMaker AI Catalog (built on Amazon DataZone) and Amazon DataZone both provide OpenLineage-compatible lineage that you can capture and visualize across your data estate. This strengthens your data privacy posture in several ways. First, column-level lineage traces exactly where sensitive data (such as personally identifiable information (PII)) is stored and how it is processed in downstream activities, providing the transparency required to demonstrate compliance with data privacy legislation such as GDPR. Second, lineage versioning creates a historical audit trail of how data has been transformed over time, supporting data subject access requests and regulatory inquiries by documenting the chain of custody for personal data. Third, by visualizing upstream and downstream dependencies, lineage helps you assess the impact of changes to data pipelines on privacy-sensitive data flows before those changes are made, reducing the risk of unintended exposure. Follow the Amazon SageMaker AI Unified Studio or Amazon DataZone documentation to enable lineage. See the resources (Related documents) section for additional technical measures you can apply.

  3. Add cross-border data flow tracking: Use these data lineage capabilities to track and document cross-border data transfers. Lineage events captured through the PostLineageEvent API record the flow of data between source and target systems as standard OpenLineage run events. For AWS resources (such as AWS Glue tables or Amazon Redshift tables), the AWS Region is embedded in the resource ARN that is the lineage node's sourceIdentifier (for example, arn:aws:glue:<region>:<account-id>:table/<database>/<table-name>). This means the source and target regions can be derived directly from the upstream and downstream lineage nodes. The resulting lineage records can then be queried through the GetLineageNode and ListLineageNodeHistory APIs to support compliance analysis, for example, verifying that source and target resources reside within the same jurisdiction, or flagging transfers that cross regulatory boundaries.

  4. Protect lineage metadata with the same rigor as the data it describes: Lineage metadata might reveal sensitive information, including data relationships, transformation logic, and access patterns that are security-sensitive. Implement access controls on lineage data and encrypt lineage metadata at rest.

Resources

Related best practices:

Related documents:

Related videos:

Related services: