Step 3: Create an Amazon MSK Cluster
In this step of Getting Started Using Amazon MSK, you create an Amazon MSK cluster in the VPC.
To create an Amazon MSK cluster using the AWS CLI
-
Open the Amazon VPC console at https://console.aws.amazon.com/vpc/
. -
In the navigation pane, choose Subnets, and then copy the subnet IDs of the three subnets you created previously. Save these IDs because you need them later in this procedure.
-
In the navigation pane, choose Security Groups. Then in the table of security groups, find the group for which the VPC ID column has the ID you saved for AWSKafkaTutorialVPC. Copy the ID of this security group and save it because you need it later in this procedure.
-
Copy the following JSON and save it to a file. Name the file
clusterinfo.json
.{ "BrokerNodeGroupInfo": { "BrokerAZDistribution": "DEFAULT", "InstanceType": "kafka.m5.large", "ClientSubnets": [ "
AWSKafkaTutorialSubnet-1 Subnet ID
", "AWSKafkaTutorialSubnet-2 Subnet ID
", "AWSKafkaTutorialSubnet-3 Subnet ID
" ], "SecurityGroups": [ "AWSKafkaTutorialVPC Security Group ID
" ] }, "ClusterName": "AWSKafkaTutorialCluster", "EncryptionInfo": { "EncryptionAtRest": { "DataVolumeKMSKeyId": "your-CMK
" }, "EncryptionInTransit": { "InCluster": true, "ClientBroker": "TLS" } }, "EnhancedMonitoring": "PER_TOPIC_PER_BROKER", "KafkaVersion": "2.2.1", "NumberOfBrokerNodes": 3 } -
In the
clusterinfo.json
file, replace the placeholders for the three subnet IDs and the security group ID with the values that you saved in previous steps. -
In
clusterinfo.json
, replaceyour-CMK
with a customer managed CMK. You can also removeEncryptionAtRest
and let Amazon MSK create a CMK and use it on your behalf. SettingInCluster
totrue
means that you want Amazon MSK to encrypt your data as it travels between brokers within the cluster. ForClientBroker
you can choose one of the following settings:TLS
,TLS_PLAINTEXT
, orPLAINTEXT
. In this exercise, we useTLS
to indicate that we want data to be encrypted as it travels between clients and brokers. For more information about encryption settings, see Amazon MSK Encryption. -
Upgrade your AWS CLI to the latest version to ensure that it has support for Amazon MSK. For detailed instructions on how to upgrade the AWS CLI, see Installing the AWS Command Line Interface.
-
Run the following AWS CLI command in the directory where you saved the
clusterinfo.json
file.aws kafka create-cluster --cli-input-json file://clusterinfo.json
The output of the command looks like the following JSON:
{ "ClusterArn": "...", "ClusterName": "AWSKafkaTutorialCluster", "State": "CREATING" }
-
Save the value of the
ClusterArn
key because you need it later.
Ensure that you saved ClusterArn
before you proceed.
Next Step
Step 4: Create a Client Machine