Step 3: Create an Amazon MSK Cluster

In this step of Getting Started Using Amazon MSK, you create an Amazon MSK cluster in the VPC.

To create an Amazon MSK cluster using the AWS CLI

1. Open the Amazon VPC console at https://console.aws.amazon.com/vpc/.

2. In the navigation pane, choose Subnets, and then copy the subnet IDs of the three subnets you created previously. Save these IDs because you need them later in this procedure.

3. In the navigation pane, choose Security Groups. Then in the table of security groups, find the group for which the VPC ID column has the ID you saved for AWSKafkaTutorialVPC. Copy the ID of this security group and save it because you need it later in this procedure.

4. Copy the following JSON and save it to a file. Name the file clusterinfo.json.

   {
     "BrokerNodeGroupInfo": {
       "BrokerAZDistribution": "DEFAULT",
       "InstanceType": "kafka.m5.large",
       "ClientSubnets": [
         "AWSKafkaTutorialSubnet-1 Subnet ID",
         "AWSKafkaTutorialSubnet-2 Subnet ID",
         "AWSKafkaTutorialSubnet-3 Subnet ID"
       ],
       "SecurityGroups": [
         "AWSKafkaTutorialVPC Security Group ID"
       ]
     },
     "ClusterName": "AWSKafkaTutorialCluster",
     "EncryptionInfo": {
       "EncryptionAtRest": {
         "DataVolumeKMSKeyId": "your-CMK"
       },
       "EncryptionInTransit": {
         "InCluster": true,
         "ClientBroker": "TLS"
       }
     },
     "EnhancedMonitoring": "PER_TOPIC_PER_BROKER",
     "KafkaVersion": "2.2.1",
     "NumberOfBrokerNodes": 3
   }

5. In the clusterinfo.json file, replace the placeholders for the three subnet IDs and the security group ID with the values that you saved in previous steps.

6. In clusterinfo.json, replace your-CMK with a customer managed CMK. You can also remove EncryptionAtRest and let Amazon MSK create a CMK and use it on your behalf. Setting InCluster to true means that you want Amazon MSK to encrypt your data as it travels between brokers within the cluster. For ClientBroker you can choose one of the following settings: TLS, TLS_PLAINTEXT, or PLAINTEXT. In this exercise, we use TLS to indicate that we want data to be encrypted as it travels between clients and brokers. For more information about encryption settings, see Amazon MSK Encryption.

7. Upgrade your AWS CLI to the latest version to ensure that it has support for Amazon MSK. For detailed instructions on how to upgrade the AWS CLI, see Installing the AWS Command Line Interface.

8. Run the following AWS CLI command in the directory where you saved the clusterinfo.json file.

   aws kafka create-cluster --cli-input-json file://clusterinfo.json

   The output of the command looks like the following JSON:

   {
       "ClusterArn": "...",
       "ClusterName": "AWSKafkaTutorialCluster",
       "State": "CREATING"
   }

9. Save the value of the ClusterArn key because you need it later.

Important

Ensure that you saved ClusterArn before you proceed.

Next Step

Step 4: Create a Client Machine