Sign-in credentials authentication with AWS Secrets Manager - Amazon Managed Streaming for Apache Kafka

credentials authentication with AWS Secrets Manager

You can control access to your Amazon MSK clusters using sign-in credentials that are stored and secured using AWS Secrets Manager. Storing user credentials in Secrets Manager reduces the overhead of cluster authentication such as auditing, updating, and rotating credentials. Secrets Manager also lets you share user credentials across clusters.

How it works

Sign-in credentials authentication for Amazon MSK uses SASL/SCRAM (Simple Authentication and Security Layer/ Salted Challenge Response Mechanism) authentication. To set up sign-in credentials authentication for a cluster, you create a Secret resource in AWS Secrets Manager, and associate sign-in credentials with that secret.

SASL/SCRAM is defined in RFC 5802. SCRAM uses secured hashing algorithms, and does not transmit plaintext sign-in credentials between client and server.


When you set up SASL/SCRAM authentication for your cluster, Amazon MSK turns on TLS encryption for all traffic between clients and brokers.

Setting up SASL/SCRAM authentication for an Amazon MSK cluster

To set up a secret in AWS Secrets Manager, follow the Creating and Retrieving a Secret tutorial in the AWS Secrets Manager User Guide.

Note the following requirements when creating a secret for an Amazon MSK cluster:

  • Choose Other type of secrets (e.g. API key) for the secret type.

  • Your secret name must begin with the prefix AmazonMSK_.

  • You must either use an existing custom AWS KMS key or create a new custom AWS KMS key for your secret. Secrets Manager uses the default AWS KMS key for a secret by default.


    A secret created with the default AWS KMS key cannot be used with an Amazon MSK cluster.

  • Your sign-in credential data must be in the following format to enter key-value pairs using the Plaintext option.

    { "username": "alice", "password": "alice-secret" }
  • Record the ARN (Amazon Resource Name) value for your secret.

  • Important

    You can't associate a Secrets Manager secret with a cluster that exceeds the limits described in Right-size your cluster: Number of partitions per broker.

  • If you use the AWS CLI to create the secret, specify a key ID or ARN for the kms-key-id parameter. Don't specify an alias.

  • To associate the secret with your cluster, use either the Amazon MSK console, or the BatchAssociateScramSecret operation.


    When you associate a secret with a cluster, Amazon MSK attaches a resource policy to the secret that allows your cluster to access and read the secret values that you defined. You should not modify this resource policy. Doing so can prevent your cluster from accessing your secret.

    The following example JSON input for the BatchAssociateScramSecret operation associates a secret with a cluster:

    { "clusterArn" : "arn:aws:kafka:us-west-2:0123456789019:cluster/SalesCluster/abcd1234-abcd-cafe-abab-9876543210ab-4", "secretArnList": [ "arn:aws:secretsmanager:us-west-2:0123456789019:secret:AmazonMSK_MyClusterSecret" ] }

Connecting to your cluster with sign-in credentials

After you create a secret and associate it with your cluster, you can connect your client to the cluster. The following example steps demonstrate how to connect a client to a cluster that uses SASL/SCRAM authentication, and how to produce to and consume from an example topic.

  1. Retrieve your cluster details with the following command. Replace ClusterArn with the Amazon Resource Name (ARN) of your cluster:

    aws kafka describe-cluster --cluster-arn "ClusterArn"

    From the JSON result of the command, save the value associated with the string named ZookeeperConnectString.

  2. To create an example topic, run the following command on your client machine. Replace ZookeeperConnectString with the string you recorded in the previous step.

    <path-to-your-kafka-installation>/bin/ --create --zookeeper ZookeeperConnectString --replication-factor 3 --partitions 1 --topic ExampleTopicName
  3. On your client machine, create a JAAS configuration file that contains the user credentials stored in your secret. For example, for the user alice, create a file called users_jaas.conf with the following content.

    KafkaClient { required username="alice" password="alice-secret"; };
  4. Use the following command to export your JAAS config file as a KAFKA_OPTS environment parameter.

  5. Create a file named kafka.client.truststore.jks in a ./tmp directory.

  6. Use the following command to copy the JDK key store file from your JVM cacerts folder into the kafka.client.truststore.jks file that you created in the previous step. Replace JDKFolder with the name of the JDK folder on your instance. For example, your JDK folder might be named java-1.8.0-openjdk-

    cp /usr/lib/jvm/JDKFolder/jre/lib/security/cacerts /tmp/kafka.client.truststore.jks
  7. In the bin directory of your Apache Kafka installation, create a client properties file called with the following contents. This file defines the SASL mechanism and protocol.

    security.protocol=SASL_SSL sasl.mechanism=SCRAM-SHA-512 ssl.truststore.location=<path-to-keystore-file>/kafka.client.truststore.jks
  8. Retrieve your bootstrap brokers string with the following command. Replace ClusterArn with the Amazon Resource Name (ARN) of your cluster:

    aws kafka get-bootstrap-brokers --cluster-arn ClusterArn

    From the JSON result of the command, save the value associated with the string named BootstrapBrokerStringSaslScram.

  9. To produce to the example topic that you created, run the following command on your client machine. Replace BootstrapBrokerStringSaslScram with the value that you retrieved in the previous step.

    <path-to-your-kafka-installation>/bin/ --broker-list BootstrapBrokerStringSaslScram --topic ExampleTopicName --producer.config
  10. To consume from the topic you created, run the following command on your client machine. Replace BootstrapBrokerStringSaslScram with the value that you obtained previously.

    <path-to-your-kafka-installation>/bin/ --bootstrap-server BootstrapBrokerStringSaslScram --topic ExampleTopicName --from-beginning --consumer.config

Working with users

Creating users: You create users in your secret as key-value pairs. When you use the Plaintext option in the Secrets Manager console, you should specify sign-in credential data in the following format.

{ "username": "alice", "password": "alice-secret" }

Revoking user access: To revoke a user's credentials to access a cluster, we recommend that you first remove or enforce an ACL on the cluster, and then disassociate the secret. This is because of the following:

  • Removing a user does not close existing connections.

  • Changes to your secret take up to 10 minutes to propagate.

For information about using an ACL with Amazon MSK, see Apache Kafka ACLs.

We recommend that you restrict access to your zookeeper nodes to prevent users from modifying ACLs. For more information, see Controlling access to Apache ZooKeeper.


Note the following limitations when using SCRAM secrets:

  • Amazon MSK only supports SCRAM-SHA-512 authentication.

  • An Amazon MSK cluster can have up to 1000 users.

  • You must use an AWS KMS key with your Secret. You cannot use a Secret that uses the default Secrets Manager encryption key with Amazon MSK. For information about creating a KMS key, see Creating symmetric encryption KMS keys.

  • You can't use an asymmetric KMS key with Secrets Manager.

  • You can associate up to 10 secrets with a cluster at a time using the BatchAssociateScramSecret operation.

  • The name of secrets associated with an Amazon MSK cluster must have the prefix AmazonMSK_.

  • Secrets associated with an Amazon MSK cluster must be in the same Amazon Web Services account and AWS region as the cluster.