Amazon MSK multi-VPC private connectivity in a single Region - Amazon Managed Streaming for Apache Kafka

Amazon MSK multi-VPC private connectivity in a single Region

Multi-VPC private connectivity (powered by AWS PrivateLink) for Amazon Managed Streaming for Apache Kafka (Amazon MSK) clusters is a feature that enables you to more quickly connect Kafka clients hosted in different Virtual Private Clouds (VPCs) and AWS accounts to an Amazon MSK cluster.

Multi-VPC private connectivity is a managed solution that simplifies the networking infrastructure for multi-VPC and cross-account connectivity. Clients can connect to the Amazon MSK cluster over PrivateLink while keeping all traffic within the AWS network. Multi-VPC private connectivity for Amazon MSK clusters is available in all AWS Regions where Amazon MSK is available.

What is multi-VPC private connectivity?

Multi-VPC private connectivity for Amazon MSK is a connectivity option that enables you to connect Apache Kafka clients that are hosted in different Virtual Private Clouds (VPCs) and AWS accounts to a MSK cluster.

Amazon MSK simplifies cross-account access with cluster policies. These policies allow the cluster owner to grant permissions for other AWS accounts to establish private connectivity to the MSK cluster.

Benefits of multi-VPC private connectivity

Multi-VPC private connectivity has several advantages over other connectivity solutions:

  • It automates operational management of the AWS PrivateLink connectivity solution.

  • It allows overlapping IPs across connecting VPCs, eliminating the need to maintain non-overlapping IPs, complex peering, and routing tables associated with other VPC connectivity solutions.

You use a cluster policy for your MSK cluster to define which AWS accounts have permissions to set up cross-account private connectivity to your MSK cluster. The cross-account admin can delegate permissions to appropriate roles or users. When used with IAM client authentication, you can also use the cluster policy to define Kafka data plane permissions on a granular basis for the connecting clients.

Requirements and limitations for multi-VPC private connectivity

Note these MSK cluster requirements for running multi-VPC private connectivity:

  • Multi-VPC private connectivity is supported only on Apache Kafka 2.7.1 or higher. Make sure that any clients that you use with the MSK cluster are running Apache Kafka versions that are compatible with the cluster.

  • Multi-VPC private connectivity supports auth types IAM, TLS and SASL/SCRAM. Unauthenticated clusters can't use multi-VPC private connectivity.

  • If you are using the SASL/SCRAM or mTLS access-control methods, you must set Apache Kafka ACLs for your cluster. First, set the Apache Kafka ACLs for your cluster. Then, update the cluster's configuration to have the property allow.everyone.if.no.acl.found set to false for the cluster. For information about how to update the configuration of a cluster, see Amazon MSK configuration operations. If you are using IAM access control and want to apply authorization policies or update your authorization policies, see IAM access control. For information about Apache Kafka ACLs, see Apache Kafka ACLs.

  • Multi-VPC private connectivity doesn’t support the t3.small instance type.

  • Multi-VPC private connectivity isn’t supported across AWS Regions, only on AWS accounts within the same Region.

  • Amazon MSK doesn't support multi-VPC private connectivity to Zookeeper nodes.

Getting started using multi-VPC private connectivity

This tutorial uses a common use case as an example of how you can use multi-VPC connectivity to privately connect an Apache Kafka client to an MSK cluster from inside AWS, but outside VPC of the cluster. This process requires the cross-account user to create a MSK managed VPC connection and configuration for each client, including required client permissions. The process also requires the MSK cluster owner to enable PrivateLink connectivity on the MSK cluster and select authentication schemes to control access to the cluster.

In different parts of this tutorial, we choose options that apply to this example. This doesn't mean that they're the only options that work for setting up an MSK cluster or client instances.

The network configuration for this use case is as follows:

  • A cross-account user (Kafka client) and an MSK cluster are in the same AWS network/Region, but in different accounts:

    • MSK cluster in Account A

    • Kafka client in Account B

  • The cross-account user will connect privately to the MSK cluster using IAM auth scheme.

This tutorial assumes that there is a provisioned MSK cluster created with Apache Kafka version 2.7.1 or higher. The MSK cluster must be in an ACTIVE state before beginning the configuration process. To avoid potential data loss or downtime, clients that will use multi-VPC private connection to connect to the cluster should use Apache Kafka versions that are compatible with the cluster.

The following diagram illustrates the architecture of Amazon MSK multi-VPC connectivity connected to a client in a different AWS account.


                        Multi-vpc network diagram in a single Region

Step 1: On the MSK cluster in Account A, turn on multi-VPC connectivity for IAM auth scheme on the cluster

The MSK cluster owner needs to make configuration settings on the MSK cluster after the cluster is created and in an ACTIVE state.

The cluster owner turns on multi-VPC private connectivity on the ACTIVE cluster for any auth schemes that will be active on the cluster. This can be done using the UpdateSecurity API or MSK console. The IAM, SASL/SCRAM, and TLS auth schemes support multi-VPC private connectivity. Multi-VPC private connectivity can’t be enabled for unauthenticated clusters.

For this use case, you’ll configure the cluster to use the IAM auth scheme.

Note

If you are configuring your MSK cluster to use SASL/SCRAM auth scheme, the Apache Kafka ACLs property "allow.everyone.if.no.acl.found=false" is mandatory. See Apache Kafka ACLs.

When you update multi-VPC private connectivity settings, Amazon MSK starts a rolling reboot of broker nodes that updates the broker configurations. This can take up to 30 minutes or more to complete. You can’t make other updates to the cluster while connectivity is being updated.

Turn on multi-VPC for selected auth schemes on the cluster in Account A using the console
  1. Open the Amazon MSK console at https://console.aws.amazon.com/msk/ for the account where the cluster is located.

  2. In the navigation pane, under MSK Clusters, choose Clusters to display the list of clusters in the account.

  3. Select the cluster to configure for multi-VPC private connectivity. The cluster must be in an ACTIVE state.

  4. Select the cluster Properties tab, and then go to Network settings.

  5. Select the Edit drop down menu and select Turn on multi-VPC connectivity.

  6. Select one or more authentication types you want turned on for this cluster. For this use case, select IAM role-based authentication.

  7. Select Save changes.

Example - UpdateConnectivity API that turns on Multi-VPC private connectivity auth schemes on a cluster

As an alternative to the MSK console, you can use the UpdateConnectivity API to turn on multi-VPC private connectivity and configure auth schemes on an ACTIVE cluster. The following example shows the IAM auth scheme turned on for the cluster.

{ "currentVersion": "K3T4TT2Z381HKD", "connectivityInfo": { "vpcConnectivity": { "clientAuthentication": { "sasl": { "iam": { "enabled": TRUE } } } } } }

Amazon MSK creates the networking infrastructure required for private connectivity. Amazon MSK also creates a new set of bootstrap broker endpoints for each auth type that requires private connectivity. Note that the plaintext auth scheme does not support multi-VPC private connectivity.

Step 2: Attach a cluster policy to the MSK cluster

The cluster owner can attach a cluster policy (also known as a resource-based policy) to the MSK cluster where you will turn on multi-VPC private connectivity. The cluster policy gives the clients permission to access the cluster from another account. Before you can edit the cluster policy, you need the account ID(s) for the accounts that should have permission to access the MSK cluster. See How Amazon MSK works with IAM.

The cluster owner must attach a cluster policy to the MSK cluster that authorizes the cross-account user in Account B to get bootstrap brokers for the cluster and to authorize the following actions on the MSK cluster in Account A:

  • CreateVpcConnection

  • GetBootstrapBrokers

  • DescribeCluster

  • DescribeClusterV2

For reference, the following is an example of the JSON for a basic cluster policy, similar to the default policy shown in the MSK console IAM policy editor.

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": [ "123456789012" ] }, "Action": [ "kafka:CreateVpcConnection", "kafka:GetBootstrapBrokers", "kafka:DescribeCluster", "kafka:DescribeClusterV2" ], "Resource": "arn:aws:kafka:us-east-1:123456789012:cluster/testing/de8982fa-8222-4e87-8b20-9bf3cdfa1521-2" } ] }
Attach a cluster policy to the MSK cluster
  1. In the Amazon MSK console, under MSK Clusters, choose Clusters.

  2. Scroll down to Security settings and select Edit cluster policy.

  3. In the console, on the Edit Cluster Policy screen, select Basic policy for multi-VPC connectivity.

  4. In the Account ID field, enter the account ID for each account that should have permission to access this cluster. As you type the ID, it is automatically copied over into the displayed policy JSON syntax. In our example cluster policy, the Account ID is 123456789012.

  5. Select Save changes.

For information about cluster policy APIs, see Amazon MSK resource-based policies.

Step 3: Cross-account user actions to configure client-managed VPC connections

To set up multi-VPC private connectivity between a client in a different account from the MSK cluster, the cross-account user creates a managed VPC connection for the client. Multiple clients can be connected to the MSK cluster by repeating this procedure. For the purposes of this use case, you’ll configure just one client.

Clients can use the supported auth schemes IAM, SASL/SCRAM, or TLS. Each managed VPC connection can have only one auth scheme associated with it. The client auth scheme must be configured on the MSK cluster where the client will connect.

For this use case, configure the client auth scheme so that the client in Account B uses the IAM auth scheme.

Prerequisites

This process requires the following items:

  • The previously created cluster policy that grants the client in Account B permission to perform actions on the MSK cluster in Account A.

  • An identity policy attached to the client in Account B that grants permissions for kafka:CreateVpcConnection, ec2:CreateTags, ec2:CreateVPCEndpoint and ec2:DescribeVpcAttribute action.

For reference, the following is an example of the JSON for a basic client identity policy.

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "kafka:CreateVpcConnection", "ec2:CreateTags", "ec2:CreateVPCEndpoint", "ec2:DescribeVpcAttribute" ], "Resource": "*" } ] }
To create a managed VPC connection for a client in Account B
  1. From the cluster administrator, get the Cluster ARN of the MSK cluster in Account A that you want the client in Account B to connect to. Make note of the cluster ARN to use later.

  2. In the MSK console for the client Account B, choose Managed VPC connections, and then choose Create connection.

  3. In the Connection settings pane, paste the cluster ARN into the cluster ARN text field, and then choose Verify.

  4. Select the Authentication type for the client in Account B. For this use case, choose IAM when creating the client VPC connection.

  5. Choose the VPC for the client.

  6. Choose at least two availability Zones and associated Subnets. You can get the availability zone IDs from the AWS Management Console cluster details or by using the DescribeCluster API or the describe-cluster AWS CLI command. The zone IDs that you specify for the client subnet must match those of the cluster subnet. If the values for a subnet are missing, first create a subnet with the same zone ID as your MSK cluster.

  7. Choose a Security group for this VPC connection. You can use the default security group. For more information on configuring a security group, see Control traffic to resources using security groups.

  8. Select Create connection.

  9. To get the list of new bootstrap broker strings from the cross-account user’s MSK console (Cluster details > Managed VPC connection), see the bootstrap broker strings shown under “Cluster connection string.” From the client Account B, the list of bootstrap brokers can be viewed by calling the GetBootstrapBrokers API or by viewing the list of bootstrap brokers in the console cluster details.

  10. Update the security groups associated with the VPC connections as follows:

    1. Set inbound rules for the PrivateLink VPC to allow all traffic for the IP range from the Account B network.

    2. [Optional] Set Outbound rules connectivity to the MSK cluster. Choose the Security Group in the VPC console, Edit Outbound Rules, and add a rule for Custom TCP Traffic for port ranges 14001-14100. The multi-VPC network load balancer is listening on the 14001-14100 port ranges. See Network Load Balancers.

  11. Configure the client in Account B to use the new bootstrap brokers for multi-VPC private connectivity to connect to the MSK cluster in Account A. See Produce and consume data.

After authorization is complete, Amazon MSK creates a managed VPC connection for each specified VPC and auth scheme. The chosen security group is associated with each connection. This managed VPC connection is configured by Amazon MSK to connect privately to the brokers. You can use the new set of bootstrap brokers to connect privately to the Amazon MSK cluster.