Have a clear understanding about where your organization’s data is coming from, how the data is transformed, who and what systems have access to the data, and how the data is used, is critical to increasing the business value of data. To achieve this goal, data lineage should be tracked, managed, and visualized.
Suggestion 7.3.1 – Track and control data lineage information
Data lineage information should include where data has come from, where the data is going, and who has access to the data. Data changes and the business logic used should also be tracked in the data lineage.
Suggestion 7.3.2 – Use visualization tools to investigate data lineage
Data lineage can become complicated when multiple systems are interacting with each other. Building a data lineage tool to visualize data lineage can reduce troubleshooting time and help identify downstream dependencies.
Suggestion 7.3.3 – Build a data lineage report to satisfy compliance and audit requirements
If some derestriction data lineage is required for compliance or audit purposes, your organization should either build a data lineage process using AWS services or investigate third-party applications.
For more details, refer to the following information:
-
AWS data lineage blog: Build data lineage for data lakes using AWS Glue, Amazon Neptune, and Spline
