Set up prerequisites for MSK Replicator with self-managed Apache Kafka clusters

Create an IAM execution role

Create an IAM role with a trust policy for kafka.amazonaws.com. Attach the AWSMSKReplicatorExecutionRole managed policy. The managed policy grants the cluster-, topic-, and consumer-group-level Kafka permissions the Replicator needs, but it does not include AWS Secrets Manager or AWS KMS permissions, which are required for SASL/SCRAM authentication and CMK-encrypted credentials. For the inline policy snippets to add, see Additional SER permissions for SASL/SCRAM, mTLS, SASL/OAUTHBEARER, and customer managed keys.

Example trust policy:

{
  "Statement": [{
    "Effect": "Allow",
    "Principal": {"Service": "kafka.amazonaws.com"},
    "Action": "sts:AssumeRole"
  }]
}

Configure SASL/SCRAM user and ACL permissions

Create a dedicated SCRAM user on your self-managed Kafka cluster. The following ACL permissions are required:

1. Read, Describe on all topics

2. Read, Describe on all consumer groups

3. Describe on cluster resource

Example kafka-acls.sh commands:

# Grant Read and Describe on all topics
kafka-acls.sh --bootstrap-server <broker>:9092 \
  --add --allow-principal User:msk-replicator \
  --operation Read --operation Describe \
  --topic '*'

# Grant Read and Describe on all consumer groups
kafka-acls.sh --bootstrap-server <broker>:9092 \
  --add --allow-principal User:msk-replicator \
  --operation Read --operation Describe \
  --group '*'

# Grant Describe on cluster
kafka-acls.sh --bootstrap-server <broker>:9092 \
  --add --allow-principal User:msk-replicator \
  --operation Describe --cluster

Configure mTLS on self-managed cluster

Configure an SSL listener on your self-managed Kafka brokers with ssl.client.auth=required. The broker's truststore must contain the CA certificate that signed the client certificate you will use for MSK Replicator.

Grant ACL permissions to the Kafka principal derived from the client certificate's Distinguished Name (DN). The required permissions are: Read and Describe on all topics, Read and Describe on all consumer groups, and Describe on the cluster resource.

Configure SASL/OAUTHBEARER (OAuth) on self-managed cluster

With SASL/OAUTHBEARER, MSK Replicator obtains an access token from your identity provider (IDP) and presents it to your self-managed Kafka cluster during the SASL/OAUTHBEARER handshake (RFC 7628). Configure your brokers with a SASL_SSL listener that has OAUTHBEARER enabled in sasl.enabled.mechanisms, and configure the broker-side validation of tokens issued by your IDP.

Grant the Kafka principal that your IDP maps the access token to the ACL permissions that MSK Replicator requires on the source cluster.

MSK Replicator supports the following mechanisms for acquiring an access token. You choose one when you create the Replicator (see CreateReplicator API examples for self-managed Kafka clusters).

• Client credentials — The standard client_credentials grant (RFC 6749 §4.4). You provide a client_id and client_secret in AWS Secrets Manager. Use this mechanism with IDPs such as Okta, Microsoft Entra ID, Keycloak, PingFederate, and Google.

• IAM JWT bearer — The JWT bearer assertion grant (RFC 7523). MSK Replicator uses the service execution role's AWS identity to obtain a signed JWT that is sent to the token endpoint as the assertion. No shared secret is required, though you can optionally supply client credentials if your IDP requires the client to also authenticate.

• Client credentials assertion — The client_credentials grant with a JWT client assertion (RFC 7521/7523 §2.2). The service execution role's signed JWT is used as the client_assertion to authenticate the client, without a shared secret.

The following requirements apply to the token endpoint:

• The tokenEndpointUrl must use the HTTPS scheme and specify a hostname (IP address literals are not allowed, so that TLS hostname verification can be performed).

• The token endpoint must be reachable from the VPC subnets you provide for the Replicator. See Configure network connectivity.

• If your IDP presents a certificate issued by a private CA, store the CA certificate in AWS Secrets Manager and reference it with tokenEndpointTlsCertificateArn when you create the Replicator.

Configure SSL on self-managed cluster

Configure SSL listeners on your brokers. For publicly trusted certificates, no additional configuration is required. For private or self-signed certificates, include the full CA certificate chain in the secret stored in AWS Secrets Manager.

Store credentials in AWS Secrets Manager

Create a secret of type Other (not RDS/Redshift) in AWS Secrets Manager with the appropriate key-value pairs for your authentication type.

For SASL/SCRAM:

1. username — SCRAM username for the self-managed cluster

2. password — SCRAM password for the self-managed cluster

3. certificate — CA certificate chain (PEM format; required for private/self-signed certs)

For mTLS:

1. certificate — PEM-encoded client certificate chain

2. privateKey — PEM-encoded private key

3. privateKeyPassword — (Optional) Passphrase for the private key, required only for encrypted PKCS8 keys

For SASL/OAUTHBEARER:

A secret is required for the client credentials mechanism and is optional for the IAM JWT bearer and client credentials assertion mechanisms (supply it only if your IDP requires the client to also authenticate). The secret is a flat JSON object of key-value pairs. MSK Replicator recognizes the following keys:

• client_id — The OAuth client identifier. Required for the client credentials mechanism.

• client_secret — The OAuth client secret. Required for the client credentials mechanism.

• custom_param.<name> — (Optional) An additional form parameter appended to the token request, for IDPs that require parameters beyond the standard OAuth set. Add one key per parameter (for example, custom_param.resource).

• custom_header.<name> — (Optional) An additional HTTP header sent with the token request. Add one key per header (for example, custom_header.X-Custom).

• extension.<name> — (Optional) A SASL extension sent to the Kafka broker during the SASL/OAUTHBEARER handshake, for Kafka providers that require additional key-value pairs during authentication. Add one key per extension.

Keys other than client_id and client_secret that do not use one of these prefixes are ignored. The following is an example secret value for the client credentials mechanism:

{
  "client_id": "my-oauth-client",
  "client_secret": "example-client-secret",
  "custom_param.resource": "urn:example:kafka"
}

Note

MSK Replicator rejects custom_param. entries whose parameter name conflicts with a standard OAuth parameter (for example, grant_type, client_id, client_secret, client_assertion, client_assertion_type, assertion, and scope). It also rejects restricted custom_header. entries such as Host, Authorization, and Content-Type.

Configure network connectivity

MSK Replicator requires network connectivity to your self-managed Kafka cluster. Supported options:

• AWS Site-to-Site VPN — Connect on-premises networks to your VPC over the internet.

• AWS Direct Connect — Establish a dedicated private network connection from your premises to AWS.

If you use SASL/OAUTHBEARER, the token endpoint must also be reachable from the VPC subnets you provide for the Replicator. For an internet-hosted IDP, this typically requires an internet gateway, NAT gateway, and route table entries; for an on-premises or private IDP, use AWS Site-to-Site VPN or AWS Direct Connect. The token endpoint must not resolve to a loopback, link-local, or AWS metadata address.

Configure security groups

Ensure security groups allow traffic between MSK Replicator and the self-managed cluster on the port used by your authentication listener. Update both inbound rules on VPC security groups and outbound rules on the self-managed cluster firewall.