Cross-Account Lineage Tracking
Amazon SageMaker supports tracking lineage entities from a different AWS account. Other AWS accounts can share their lineage entities with you and you can access these lineage entities through direct API calls or SageMaker lineage queries.
SageMaker uses AWS Resource Access Manager to help you securely share your lineage resources. You can
share your resources through the AWS RAM console
Set Up Cross-Account Lineage Tracking
You can group and share your Lineage Tracking Entities through a lineage group in Amazon SageMaker. SageMaker supports only one default lineage group per account. SageMaker creates the default lineage group whenever a lineage entity is created in your account. Every lineage entity owned by your account is assigned to this default lineage group. To share lineage entities with another account, you share this default lineage group with that account.
Note
You can share all lineage tracking entities in a lineage group or none.
Create a resource share for your lineage entities using AWS Resource Access Manager console. For more information, see Sharing your AWS resources in the AWS Resource Access Manager User Guide.
Note
After the resource share is created, it can take a few minutes for the resource and principal associations to complete. Once the association is set, the shared account receives an invitation to join the resource share. The shared account must accept the invite to gain access to shared resources. For more information on accepting a resource share invite in AWS RAM, see Using shared AWS resources in the AWS Resource Access Manager User Guide.
Your cross-account lineage tracking resource policy
Amazon SageMaker supports only one type of resource policy. The SageMaker resource policy must allow all of the following operations:
"sagemaker:DescribeAction" "sagemaker:DescribeArtifact" "sagemaker:DescribeContext" "sagemaker:DescribeTrialComponent" "sagemaker:AddAssociation" "sagemaker:DeleteAssociation" "sagemaker:QueryLineage"
Example The following is a SageMaker resource policy created using AWS Resource Access Manager for creating a resource share for an accounts lineage group.
{ "Version": "2012-10-17", "Statement": [ { "Sid": "FullLineageAccess", "Effect": "Allow", "Principal": { "AWS": "
123456789012
" #account-id }, "Action": [ "sagemaker:DescribeAction", "sagemaker:DescribeArtifact", "sagemaker:DescribeContext", "sagemaker:DescribeTrialComponent", "sagemaker:AddAssociation", "sagemaker:DeleteAssociation", "sagemaker:QueryLineage" ], "Resource": "arn:aws:sagemaker:us-west-2
:111111111111
:lineage-group/sagemaker-default-lineage-group" #Sample lineage group resource } ] }
Tracking Cross-Account Lineage Entities
With cross-account lineage tracking you can associate
lineage entities in different accounts using the same AddAssociation
API
action. When you associate two lineage entities, SageMaker validates if you have permissions to
perform the AddAssociation
API action on both lineage entities. SageMaker then
establishes the association. If you don’t have the permissions, SageMaker does
not create the association. Once the cross-account association is established,
you can access either lineage entity from the other through the QueryLineage
API action. For more information, see Querying Lineage Entities.
In addition to SageMaker automatically creating lineage entities, if you have cross-account
access, SageMaker connects artifacts that reference
the same object or data. If the data from one account is used in lineage tracking by
different accounts, SageMaker creates an artifact in each account to track that data.
With cross-account lineage, whenever SageMaker creates new artifacts, SageMaker checks if there
are other artifacts created for the same data that are also shared with you. SageMaker then
establishes associations between the newly created artifact and each of the artifacts
shared with you with the AssociationType
set to SameAs
.
You can then use the QueryLineage
API action to traverse the lineage entities
in your own account to lineage entities shared with you but owned by a different AWS
account. For more information, see Querying Lineage Entities
Topics
Accessing lineage resources from a different accounts
Once the cross-account access for sharing lineage has been set up, you can call the following SageMaker API actions directly with the ARN to describe the shared lineage entities from another account:
You can also manage Associations for lineage entities owned by different accounts that are shared with you, using the following SageMaker API actions:
For a notebook that demonstrates how to use SageMaker Lineage APIs to query lineage across accounts.,
see sagemaker-lineage-cross-account-with-ram.ipynb
Authorization for querying cross-account lineage entities
Amazon SageMaker must validate that you have permissions to perform the
QueryLineage
API action on the StartArns
. This is enforced through the
resource policy attached to the LineageGroup
. The result from this action includes all the lineage entities to which
you have access, whether they are owned by your account or shared by another account. For
more information, see Querying Lineage Entities.