Step 1: Set up access control Step 2: Install and configure the ASCP Step 3: Identify which secrets to mount Step 4: Mount the secrets as files in the Amazon EKS pod Troubleshoot

Use AWS Secrets Manager secrets in Amazon Elastic Kubernetes Service

To show secrets from Secrets Manager as files mounted in Amazon EKS pods, you can use the AWS Secrets and Configuration Provider (ASCP) for the Kubernetes Secrets Store CSI Driver. The ASCP works with Amazon Elastic Kubernetes Service (Amazon EKS) 1.17+ running an Amazon EC2 node group. AWS Fargate node groups are not supported. With the ASCP, you can store and manage your secrets in Secrets Manager and then retrieve them through your workloads running on Amazon EKS. If your secret contains multiple key/value pairs in JSON format, you can choose which ones to mount in Amazon EKS. The ASCP uses JMESPath syntax to query the key/value pairs in your secret. The ASCP also works with Parameter Store parameters.

If you use a private Amazon EKS cluster, ensure that the VPC that the cluster is in has a Secrets Manager endpoint. The Secrets Store CSI Driver uses the endpoint to make calls to Secrets Manager. For information about creating an endpoint in a VPC, see VPC endpoint.

If you use Secrets Manager automatic rotation for your secrets, you can also use the Secrets Store CSI Driver rotation reconciler feature to ensure you are retrieving the latest secret from Secrets Manager. For more information, see Auto rotation of mounted contents and synced Kubernetes Secrets.

Topics

Step 1: Set up access control
Step 2: Install and configure the ASCP
Step 3: Identify which secrets to mount
Step 4: Mount the secrets as files in the Amazon EKS pod
Troubleshoot
SecretProviderClass

Step 1: Set up access control

The ASCP retrieves the Amazon EKS pod identity and exchanges it for an IAM role. You set permissions in an IAM policy for that IAM role. When the ASCP assumes the IAM role, it gets access to the secrets you authorized. Other containers can't access the secrets unless you also associate them with the IAM role.

If calls from the ASCP to look up the Region and IAM role associated with the pod are throttled by Kubernetes, you can change the throttling quotas using helm install, as shown in Step 2.

To grant your Amazon EKS pod access to secrets in Secrets Manager

Create a permissions policy that grants secretsmanager:GetSecretValue and secretsmanager:DescribeSecret permission to the secrets that the pod needs to access. For an example policy, see Example: Permission to read and describe individual secrets.
Create an IAM OpenID Connect (OIDC) provider for the cluster if you don't already have one. For more information, see Create an IAM OIDC provider for your cluster in the Amazon EKS User Guide.
Create an IAM role for service account and attach the policy to it. For more information, see Create an IAM role for a service account in the Amazon EKS User Guide.
If you use a private Amazon EKS cluster, ensure that the VPC that the cluster is in has an AWS STS endpoint. For information about creating an endpoint, see Interface VPC endpoints in the AWS Identity and Access Management User Guide.

Step 2: Install and configure the ASCP

The ASCP is available on GitHub in the secrets-store-csi-provider-aws repository. The repo also contains example YAML files for creating and mounting a secret.

During installation, you can configure the ASCP to use a FIPS endpoint. For a list of endpoints, see AWS Secrets Manager endpoints.

To install the ASCP by using Helm

To ensure the repo is pointing to the latest charts, use helm repo update.

Add the Secrets Store CSI Driver chart.


helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts

Install the chart. To configure throttling, add the following flag: --set-json 'k8sThrottlingParams={"qps": "<number of queries per second>", "burst": "<number of queries per second>"}'
```
helm install -n kube-system csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver
```

Add the ASCP chart.


helm repo add aws-secrets-manager https://aws.github.io/secrets-store-csi-driver-provider-aws

Install the chart. To use a FIPS endpoint, add the following flag: --set useFipsEndpoint=true


helm install -n kube-system secrets-provider-aws aws-secrets-manager/secrets-store-csi-driver-provider-aws

To install by using the YAML in the repo

Use the following commands.


helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
helm install -n kube-system csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver
kubectl apply -f https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/deployment/aws-provider-installer.yaml

Step 3: Identify which secrets to mount

To determine which secrets the ASCP mounts in Amazon EKS as files on the filesystem, you create a SecretProviderClass YAML file. The SecretProviderClass lists the secrets to mount and the file name to mount them as. The SecretProviderClass must be in the same namespace as the Amazon EKS pod it references.

The following examples show how to use SecretProviderClass to describe the secrets you want to mount and what to name the files mounted in the Amazon EKS pod.

Examples:

Mount secrets by name or ARN
Mount key/value pairs from a secret
Define a failover Region for a multi-Region secret
Choose a failover secret to mount

Example: Mount secrets by name or ARN

The following example shows a SecretProviderClass that mounts three files in Amazon EKS:

A secret specified by full ARN.
A secret specified by name.
A specific version of a secret.


apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: aws-secrets
spec:
  provider: aws
  parameters:
    objects: |
        - objectName: "arn:aws:secretsmanager:us-east-2:111122223333:secret:MySecret2-d4e5f6"
        - objectName: "MySecret3"
          objectType: "secretsmanager"
        - objectName: "MySecret4"
          objectType: "secretsmanager"
          objectVersionLabel: "AWSCURRENT"

Example: Mount key/value pairs from a secret

The following example shows a SecretProviderClass that mounts three files in Amazon EKS:

A secret specified by full ARN.
The username key/value pair from the same secret.
The password key/value pair from the same secret.


apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: aws-secrets
spec:
  provider: aws
  parameters:
    objects: |
      - objectName: "arn:aws:secretsmanager:us-east-2:111122223333:secret:MySecret-a1b2c3"
        jmesPath: 
            - path: username
              objectAlias: dbusername
            - path: password
              objectAlias: dbpassword

Example: Define a failover Region for a multi-Region secret

To provide availability during connectivity outages or for disaster recovery configurations, the ASCP supports an automated failover feature to retrieve secrets from a secondary region.

The following example shows a SecretProviderClass that retrieves a secret that is replicated to multiple Regions. In this example, the ASCP tries to retrieve the secret from both us-east-1 and us-east-2. If either Region returns a 4xx error, for example for an authentication issue, the ASCP does not mount either secret. If the secret is retrieved successfully from us-east-1, then the ASCP mounts that secret value. If the secret is not retrieved successfully from us-east-1, but it is retrieved successfully from us-east-2, then the ASCP mounts that secret value.


apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: aws-secrets
spec:
  provider: aws
  parameters:
    region: us-east-1
    failoverRegion: us-east-2
    objects: |
      - objectName: "MySecret"

Example: Choose a failover secret to mount

The following example shows a SecretProviderClass that specifies which secret to mount in case of failover. The failover secret isn't a replica. In this example, the ASCP tries to retrieve the two secrets specified by objectName. If either returns a 4xx error, for example for an authentication issue, the ASCP does not mount either secret. If the secret is retrieved successfully from us-east-1, then the ASCP mounts that secret value. If the secret is not retrieved successfully from us-east-1, but it is retrieved successfully from us-east-2, then the ASCP mounts that secret value. The mounted file in Amazon EKS is named MyMountedSecret.


apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: aws-secrets
spec:
  provider: aws
  parameters:
    region: us-east-1
    failoverRegion: us-east-2
    objects: |
      - objectName: "arn:aws:secretsmanager:us-east-1:111122223333:secret:MySecret-a1b2c3"
        objectAlias: "MyMountedSecret"
        failoverObject: 
          - objectName: "arn:aws:secretsmanager:us-east-2:111122223333:secret:MyFailoverSecret-d4e5f6"

Step 4: Mount the secrets as files in the Amazon EKS pod

The following instructions show how to mount secrets as files using example YAML files ExampleSecretProviderClass.yaml and ExampleDeployment.yaml.

To mount secrets in Amazon EKS

Apply the SecretProviderClass to the pod with the command kubectl apply -f ExampleSecretProviderClass.yaml.
Deploy your pod with the command kubectl apply -f ExampleDeployment.yaml.
The ASCP mounts the files.

Troubleshoot

You can view most errors by describing the pod deployment.

To see error messages for your container

Get a list of pod names with the following command. If you aren't using the default namespace, use -n <NAMESPACE>.
```
kubectl get pods
```
To describe the pod, in the following command, for <PODID> use the pod ID from the pods you found in the previous step. If you aren't using the default namespace, use -n <NAMESPACE>.
```
kubectl describe pod/<PODID>
```

To see errors for the ASCP

To find more information in the provider logs, in the following command, for <PODID> use the ID of the csi-secrets-store-provider-aws pod.
```
kubectl -n kube-system get pods
kubectl -n kube-system logs pod/<PODID>
```

Warning Javascript is disabled or is unavailable in your browser.

To use the Amazon Web Services Documentation, Javascript must be enabled. Please refer to your browser's Help pages for instructions.

Document Conventions

AWS Lambda

SecretProviderClass