Using third-party registries not supported by pull through cache

Your workflow can use container images from upstream registries that Amazon ECR pull through cache (PTC) doesn't support. The first time a task from your run needs one of these images, HealthOmics copies it from the upstream registry into a private Amazon ECR repository in your account and starts the task from the copy. Later tasks and runs reuse the copy, so the download happens only once. If several runs need the same image at the same time, for example, runs in a batch, HealthOmics copies the image once and the runs wait for that copy. Your workflow definition keeps the original upstream image URIs, so you don't pull, push, or rewrite image URIs yourself.

We recommend that you reference these images by digest or version instead of by tag. For more information, see Use a digest or version instead of a tag for non-PTC images.

HealthOmics supports the following upstream registries that PTC doesn't support:

• Seqera Containers (community.wave.seqera.io and wave.seqera.io)

• NVIDIA NGC (nvcr.io)

• Google Container Registry (gcr.io)

• Google Artifact Registry (region-docker.pkg.dev). Use the full regional host, for example us-central1-docker.pkg.dev.

Configuring non-PTC container support

Before you use containers from registries that PTC doesn't support, store your registry credentials and set up permissions for HealthOmics.

Step 1: Store your registry credential in Secrets Manager

HealthOmics authenticates and copies container images from upstream registries using your credentials. You store your credentials as a secret in the same AWS Region as your HealthOmics runs. You can also use a secret from a different AWS account or share your secret with another account. When you use a secret from a different AWS account, the service role that you pass to StartRun must be allowed to read the secret. For more information, see Access Secrets Manager secrets from a different account in the Secrets Manager User Guide.

Every supported registry needs a credential, including registries that allow anonymous pulls, such as community.wave.seqera.io.

The following tabs show the credential format and an example for each supported registry.

Seqera Containers and Wave

A JSON object with a single accessToken key that holds a Seqera Platform access token.

The following AWS CLI command creates a secret for Seqera Containers:

aws secretsmanager create-secret \
     --name omics-registry/seqera \
     --secret-string '{"accessToken":"seqera-platform-access-token"}' \
     --region us-east-1

NVIDIA NGC

A JSON object with username set to the literal $oauthtoken and accessToken set to your NGC API key.

The following AWS CLI command creates a secret for NVIDIA NGC:

aws secretsmanager create-secret \
     --name omics-registry/nvidia-ngc \
     --secret-string '{"username":"$oauthtoken","accessToken":"ngc-api-key"}' \
     --region us-east-1

When you create a secret for NVIDIA NGC, keep the --secret-string value in single quotes so that your shell doesn't expand $oauthtoken.

Google Container Registry and Google Artifact Registry

Create a JSON key for a Google Cloud service account that can read images from your registry, and save the key file (for example, key.json). For instructions, see Create and delete service account keys in the Google Cloud documentation.

Set username to _json_key and accessToken to the contents of the key file as a single JSON string. The following AWS CLI command creates the secret and uses the jq tool to build the JSON string:

aws secretsmanager create-secret \
     --name omics-registry/google \
     --secret-string "$(jq -n --rawfile key key.json '{username: "_json_key", accessToken: $key}')" \
     --region us-east-1

You can record the Amazon Resource Name (ARN) of the secret from the response and use it as the credentialArn value in the registry mapping.

Step 2: Grant permissions to the service role

HealthOmics uses the service role that you pass to StartRun to read the secret, create the destination repository, and push the copied image. The service role must be in the same AWS account as the run. Otherwise, HealthOmics rejects the role with the error IAM role does not match the requesting AWS account.

For the policy, see Service role policy for non-PTC container support.

After you grant these permissions, add a registry mapping with the secret ARN as the credentialArn. For more information, see Registry mappings.

Destination repository naming

You specify registry mappings when you create a workflow, and HealthOmics copies the images during a run. For more information, see Adding private or third-party containers during workflow creation.

Each copied image lands at a URI that you can compute from the registry mapping:

account-id.dkr.ecr.region.amazonaws.com/ecrRepositoryPrefix/image-path:tag

image-path is the upstream image URI without the registry host, and without upstreamRepositoryPrefix if you set one. Keep upstreamRepositoryPrefix unset for Wave (wave.seqera.io). The account is the workflow's account, unless you set ecrAccountId. When you set ecrAccountId, HealthOmics creates the repository and pushes the copy in that account using your service role. The service role then needs the same Amazon ECR permissions on that account's repositories, and that account's registry and repository policies must allow it.

For example, community.wave.seqera.io/library/samtools:1.22.1--eccb42ff8fb55509 with prefix wave-seqera lands at 111122223333.dkr.ecr.us-east-1.amazonaws.com/wave-seqera/library/samtools:1.22.1--eccb42ff8fb55509.

To confirm that a copy succeeded, check that the image exists at its destination URI, for example with the following AWS CLI command:

aws ecr describe-images \
     --repository-name wave-seqera/library/samtools \
     --image-ids imageTag=1.22.1--eccb42ff8fb55509 \
     --region us-east-1

If the repository doesn't exist in the workflow's account, HealthOmics creates it. If the repository has no repository policy, HealthOmics attaches one that lets HealthOmics pull the image. If the repository already has a policy, HealthOmics leaves it unchanged, so include a statement that lets HealthOmics pull images. For an example, see Create a resource policy for the Amazon ECR repository.

ecrRepositoryPrefix/image-path becomes an Amazon ECR repository name, so the prefix must use lowercase letters, numbers, and the characters /, -, _, and ..

If your workflows in the same account share an ecrRepositoryPrefix, they share the copied images. If you delete a copied image, it affects every workflow that uses that image with that prefix.

Considerations for non-PTC container support

Consider the following when you use non-PTC container support:

• Architecture – HealthOmics copies x86_64 images only. ARM-based and multi-architecture images aren't supported.

• Workflow sharing – Non-PTC container support isn't supported for shared workflows. When another account runs a workflow that you shared, HealthOmics ignores registry mappings that include credentialArn.

• Image tags – HealthOmics copies the image that a tag, such as latest, points to and doesn't refresh the copy when the upstream content changes. To pick up the new content, delete the copied image from Amazon ECR. After you delete it, HealthOmics copies the image again from the upstream registry on the next run that needs it. If your workflow references a new tag or digest, HealthOmics copies that image the first time a task needs it.

• Copy failures – If the registry credential is invalid, or the service role doesn't have the permissions that HealthOmics needs, the image copy fails, and the task and the run fail. If a copy of the image already exists in Amazon ECR, HealthOmics uses that copy and doesn't check the registry credential.