Accessing an Amazon MSK Cluster - Amazon Managed Streaming for Apache Kafka

Accessing an Amazon MSK Cluster

To access your Amazon MSK cluster from a client that's in the same Amazon VPC as the cluster, make sure the cluster's security group has an inbound rule that accepts traffic from the client's security group. For information about setting up these rules, see Security Group Rules. For an example of how to access a cluster from an Amazon EC2 instance that's in the same VPC as the cluster, see Getting Started Using Amazon MSK.

To access an MSK cluster from outside its Amazon VPC, the following options exist.

Amazon VPC Peering

To access your MSK cluster from a VPC that's different from the cluster's VPC, you can create a peering connection between the two VPCs. For information about VPC peering, see the Amazon VPC Peering Guide.

AWS Direct Connect

AWS Direct Connect links your internal network to an AWS Direct Connect location over a standard 1 gigabit or 10 gigabit Ethernet fiber-optic cable. One end of the cable is connected to your router, the other to an AWS Direct Connect router. With this connection in place, you can create virtual interfaces directly to the AWS cloud and Amazon VPC, bypassing Internet service providers in your network path. For more information, see AWS Direct Connect.

AWS Transit Gateway

AWS Transit Gateway is a service that enables you to connect your VPCs and your on-premises networks to a single gateway. For information about how to use AWS Transit Gateway, see AWS Transit Gateway.

VPN Connections

You can connect your MSK cluster's VPC to remote networks and users using the VPN connectivity options described in the following topic: VPN Connections.

REST Proxies

You can install a REST proxy on an instance running within your cluster's Amazon VPC. REST proxies enable your producers and consumers to communicate with the cluster through HTTP API requests.

Multiple Region Multi-VPC Connectivity

The following document describes connectivity options for multiple VPCs that reside in different AWS Regions: Multiple Region Multi-VPC Connectivity.

EC2-Classic

Use the following procedure to access your cluster from an EC2-Classic instance.

  1. Follow the guidance described in ClassicLink to connect your EC2-Classic instance to your cluster's VPC.

  2. Find and copy the private IP associated with your EC2-Classic instance.

  3. Using the AWS CLI, run the following command, replacing ClusterArn with the Amazon Resource Name (ARN) for your MSK cluster.

    aws kafka describe-cluster --region us-east-1 --cluster-arn "ClusterArn"
  4. In the output of the describe-cluster command, look for SecurityGroups and save the ID of the security group for your MSK cluster.

  5. Open the Amazon VPC console at https://console.aws.amazon.com/vpc/.

  6. In the left pane, choose Security Groups.

  7. Choose the security group whose ID you saved after you ran the describe-cluster command. Select the box at the beginning of the row corresponding to this security group.

  8. In the lower half of the page, choose Inbound Rules.

  9. Choose Edit rules, then choose Add Rule.

  10. For the Type field, choose All traffic in the drop-down list.

  11. Leave the Source set to Custom and enter the private IP of your EC2-Classic instance, followed immediately by /32 with no intervening spaces.

  12. Choose Save rules.

Port Information

The following list provides the numbers of the ports that Amazon MSK uses to communicate with client machines.

  • To communicate with producers and consumers in plaintext, brokers use port 9092.

  • To communicate with producers and consumers in TLS, brokers use port 9094.

  • Apache ZooKeeper nodes use port 2181.