View a markdown version of this page

Using third-party registries supported by pull through cache - AWS HealthOmics
Configuring pull through cache

Using third-party registries supported by pull through cache

You can use Amazon ECR pull through cache rules to synchronize repositories in a supported upstream registry with your Amazon ECR private repositories. For more information, see Sync an upstream registry in the Amazon Elastic Container Registry User Guide.

When you create the cache, pull through cache creates the image repository in your private registry and keeps it synchronized with the upstream image.

HealthOmics supports pull through cache for the following upstream registries:

  • Amazon ECR Public

  • Kubernetes container image registry

  • Quay

  • Docker Hub

  • Microsoft Azure Container Registry

  • GitHub Container Registry

  • GitLab Container Registry

For registries that pull through cache doesn't support, HealthOmics copies the images into your Amazon ECR private repositories. For more information, see Using third-party registries not supported by pull through cache.

Benefits of using Amazon ECR pull through cache include:

  1. You avoid having to manually migrate container images to Amazon ECR or to synchronize updates from the third party repository.

  2. Workflows access the synchronized container images in your private repository, which is more reliable than downloading content at run time from a public registry.

  3. Because Amazon ECR pull through caches use a predictable URI structure, the HealthOmics service can automatically map the Amazon ECR private URI with the upstream registry URI. You aren't required to update and replace URI values in the workflow definition.

Configuring pull through cache

Amazon ECR provides a registry for your AWS account in each Region. Make sure you create the Amazon ECR configuration in the same region where you plan to run the workflow.

Before you use containers from Amazon ECR pull through cache, create a pull through cache rule, set up permissions, and create repository templates. The following sections describe these configuration tasks.

Step 1: Creating a pull through cache rule

Create an Amazon ECR pull through cache rule for each upstream registry that has images you want to cache. A rule specifies a mapping between an upstream registry and the Amazon ECR private repository.

For an upstream registry that requires authentication, you provide your credentials using AWS Secrets Manager.

Note

Don't change a pull through cache rule while an active run is using the private repository. The run could fail or, more critically, result in your pipeline using unexpected images.

For more information, see Creating a pull through cache rule in the Amazon Elastic Container Registry User Guide.

Creating a pull through cache rule using the console

To configure pull through cache, follow these steps using the Amazon ECR console:

Part 1: Choosing the upstream registry

  1. Open the Amazon ECR console.

  2. In the navigation pane, under Private registry, expand Features & Settings, then choose Pull through cache.

  3. On the Pull through cache page, choose Add rule.

  4. In the Upstream registry panel, choose the upstream registry to sync with your private registry, then choose Next.

  5. If the upstream registry requires authentication, on the next page, specify the Secrets Manager secret that contains your credentials. Choose Next.

Part 2: Specifying namespaces and creating the rule

  1. Under Specify namespaces, in the Cache namespace panel, choose whether to create the private repositories using a specific repository prefix or with no prefix. If you choose to use a prefix, specify the prefix name in Cache repository prefix.

  2. In the Upstream namespace panel, choose whether to pull from upstream repositories using a specific repository prefix or with no prefix. If you choose to use a prefix, specify the prefix name in Upstream repository prefix.

    The Namespace example panel shows an example pull request, upstream URL, and the URL of the cache repository that is created.

  3. Choose Next.

  4. Review the configuration and choose Create to create the rule.

Creating a pull through cache rule using the AWS CLI

Use the Amazon ECR create-pull-through-cache-rule command to create a pull through cache rule. For upstream registries that require authentication, store the credentials in a Secrets Manager secret.

The following examples show how to create a rule for each supported upstream registry.

Amazon ECR Public

The following example creates a pull through cache rule for the Amazon ECR Public registry. It specifies a repository prefix of ecr-public, so each repository that the rule creates uses the naming scheme ecr-public/upstream-repository-name.

aws ecr create-pull-through-cache-rule \
     --ecr-repository-prefix ecr-public \
     --upstream-registry-url public.ecr.aws \
     --region us-east-1
Kubernetes container image registry

The following example creates a pull through cache rule for the Kubernetes public registry. It specifies a repository prefix of kubernetes, so each repository that the rule creates uses the naming scheme kubernetes/upstream-repository-name.

aws ecr create-pull-through-cache-rule \
     --ecr-repository-prefix kubernetes \
     --upstream-registry-url registry.k8s.io \
     --region us-east-1
Quay

The following example creates a pull through cache rule for the Quay public registry. It specifies a repository prefix of quay, so each repository that the rule creates uses the naming scheme quay/upstream-repository-name.

aws ecr create-pull-through-cache-rule \
     --ecr-repository-prefix quay \
     --upstream-registry-url quay.io \
     --region us-east-1
Docker Hub

The following example creates a pull through cache rule for the Docker Hub registry. It specifies a repository prefix of docker-hub, so each repository that the rule creates uses the naming scheme docker-hub/upstream-repository-name. You must specify the full Amazon Resource Name (ARN) of the secret containing your Docker Hub credentials.

aws ecr create-pull-through-cache-rule \
     --ecr-repository-prefix docker-hub \
     --upstream-registry-url registry-1.docker.io \
     --credential-arn arn:aws:secretsmanager:us-east-1:111122223333:secret:ecr-pullthroughcache/example1234 \
     --region us-east-1
GitHub Container Registry

The following example creates a pull through cache rule for the GitHub Container Registry. It specifies a repository prefix of github, so each repository that the rule creates uses the naming scheme github/upstream-repository-name. You must specify the full Amazon Resource Name (ARN) of the secret containing your GitHub Container Registry credentials.

aws ecr create-pull-through-cache-rule \
     --ecr-repository-prefix github \
     --upstream-registry-url ghcr.io \
     --credential-arn arn:aws:secretsmanager:us-east-1:111122223333:secret:ecr-pullthroughcache/example1234 \
     --region us-east-1
Microsoft Azure Container Registry

The following example creates a pull through cache rule for the Microsoft Azure Container Registry. It specifies a repository prefix of azure, so each repository that the rule creates uses the naming scheme azure/upstream-repository-name. You must specify the full Amazon Resource Name (ARN) of the secret containing your Microsoft Azure Container Registry credentials.

aws ecr create-pull-through-cache-rule \
     --ecr-repository-prefix azure \
     --upstream-registry-url myregistry.azurecr.io \
     --credential-arn arn:aws:secretsmanager:us-east-1:111122223333:secret:ecr-pullthroughcache/example1234 \
     --region us-east-1
GitLab Container Registry

The following example creates a pull through cache rule for the GitLab Container Registry. It specifies a repository prefix of gitlab, so each repository that the rule creates uses the naming scheme gitlab/upstream-repository-name. You must specify the full Amazon Resource Name (ARN) of the secret containing your GitLab Container Registry credentials.

aws ecr create-pull-through-cache-rule \
     --ecr-repository-prefix gitlab \
     --upstream-registry-url registry.gitlab.com \
     --credential-arn arn:aws:secretsmanager:us-east-1:111122223333:secret:ecr-pullthroughcache/example1234 \
     --region us-east-1

You can use the get-run-task AWS CLI command to retrieve information about the container image used for a specific task:

aws omics get-run-task --id run-id --task-id task-id

The output includes the following information about the container image:

"imageDetails": {
    "image": "string",
    "imageDigest": "string",
    "sourceImage": "string",
    ...
}

Step 2: Setting up permissions for the upstream registry

Use registry permissions to allow HealthOmics to use the pull through cache and to pull the container images into the Amazon ECR private registry. Add an Amazon ECR Registry policy to the registry that provides the containers used in runs.

The following policy grants permission for the HealthOmics service to create repositories with the specified pull through cache prefixes and to initiate upstream pulls into these repositories.

  1. Open the Amazon ECR console.

  2. In the navigation pane, under Private registry, expand Registry permissions, then choose Generate statement.

  3. On the top right side, choose JSON. Enter a policy similar to the following:

    JSON
    {
    "Version":"2012-10-17",
    "Statement": [
        {
            "Sid": "AllowPTCinRegPermissions",
            "Effect": "Allow",
            "Principal": {
                "Service": "omics.amazonaws.com"
            },
            "Action": [
                "ecr:CreateRepository",
                "ecr:BatchImportUpstreamImage"
            ],
            "Resource": [
                "arn:aws:ecr:us-east-1:123456789012:repository/ecr-public/*",
                "arn:aws:ecr:us-east-1:123456789012:repository/docker-hub/*"
            ] 
        }
    ]
}

For more information about the policies for Amazon ECR pull through cache, see Policies for Amazon ECR pull through cache.

Step 3: Creating repository templates

To use pull through caching in HealthOmics, the Amazon ECR repository must have a repository creation template. The template defines configuration settings for when you or Amazon ECR create a private repository for an upstream registry.

Each template contains a repository namespace prefix, which Amazon ECR uses to match new repositories to a specific template. Templates specify the configuration for all repository settings including resource-based access policies, tag immutability, encryption, and lifecycle policies.

For more information, see Repository creation templates in the Amazon Elastic Container Registry User Guide.

To create a repository creation template, follow these steps using the Amazon ECR console:

Part 1: Defining the template

  1. Open the Amazon ECR console.

  2. In the navigation pane, under Private registry, expand Features and settings, then choose Repository creation templates.

  3. Choose Create template.

  4. In Template details, choose Pull through cache.

  5. Choose whether to apply this template to a specific prefix or to all repositories that don't match another template.

    If you choose A specific prefix, enter the namespace prefix value in Prefix. You specified this prefix when you created the pull through cache rule.

  6. Choose Next.

Part 2: Adding the repository creation configuration

  1. On the Add repository creation configuration page, for Repository permissions, use one of the sample policy statements, or enter one similar to the following example:

    JSON
    {
    "Version":"2012-10-17",
    "Statement": [
        {
            "Sid": "PTCRepoCreationTemplate",
            "Effect": "Allow",
            "Principal": {
                "Service": "omics.amazonaws.com"
            },
            "Action": [
                "ecr:BatchGetImage",
                "ecr:GetDownloadUrlForLayer"
            ],
            "Resource": "*"
        }
    ]
}

  2. (Optional) Add repository settings such as lifecycle policy and tags. Amazon ECR applies these rules for all container images created for pull through cache that use the specified prefix.

  3. Choose Next.

  4. Review the configuration and choose Create to create the template.

After you configure pull through cache, you can create a new workflow or workflow version to use the containers from configured registries. For more information, see Adding private or third-party containers during workflow creation.