Step 5: Create a Topic
In this step of Getting Started Using Amazon MSK, you install Apache Kafka client libraries and tools on the client machine, and then you create a topic.
To create a topic on the client machine
-
Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/
. -
In the navigation pane, choose Instances, and then choose AWSKafkaTutorialClient by selecting the check box next to it.
-
Choose Actions, and then choose Connect. Follow the instructions to connect to the client machine AWSKafkaTutorialClient.
-
Install Java on the client machine by running the following command:
sudo yum install java-1.8.0
-
Run the following command to download Apache Kafka.
wget https://archive.apache.org/dist/kafka/2.2.1/kafka_2.12-2.2.1.tgz
Note If you want to use a mirror site other than the one used in this command, you can choose a different one on the Apache
website. -
Run the following command in the directory where you downloaded the TAR file in the previous step.
tar -xzf kafka_2.12-2.2.1.tgz
-
Go to the kafka_2.12-2.2.1 directory.
-
Cluster creation can take a few minutes. To find out whether the cluster you created is ready, run the following command, replacing
ClusterArn
with the Amazon Resource Name (ARN) that you obtained at the end of Step 3: Create an Amazon MSK Cluster.aws kafka describe-cluster --region us-east-1 --cluster-arn "
ClusterArn
"The result of running this command looks like the following JSON:
{ "ClusterInfo": { "BrokerNodeGroupInfo": { "BrokerAZDistribution": "DEFAULT", "ClientSubnets": [ "subnet-0d44a1567c2ce409a", "subnet-051201cac65561565", "subnet-08b4eceb2bd3bd8c2" ], "InstanceType": "kafka.m5.large", "SecurityGroups": [ "sg-041e78b0a8ba7f834" ], "StorageInfo": { "EbsStorageInfo": { "VolumeSize": 1000 } } }, "ClusterArn": "...", "ClusterName": "AWSKafkaTutorialCluster", "CreationTime": "2018-11-06T01:36:57.451Z", "CurrentBrokerSoftwareInfo": { "KafkaVersion": "2.2.1" }, "CurrentVersion": "K3UN6WX5RRO2AG", "EncryptionInfo": { "EncryptionAtRest": { "DataVolumeKMSKeyId": "arn:aws:kms:us-east-1:012345678901:key/a7de6539-7d2e-4e71-a279-aaaa5555878" } }, "EnhancedMonitoring": "DEFAULT", "NumberOfBrokerNodes": 3, "State": "CREATING" } }
If the output of the command shows that the state of the cluster is still
CREATING
, wait a few minutes, and then run the command again. Keep running this command every few minutes until the state turns toACTIVE
. When the state isACTIVE
, the output of thisdescribe-cluster
command includes an additional key namedZookeeperConnectString
. Copy the entire value associated with this key because you need it to create an Apache Kafka topic in the following command. -
Run the following command, replacing
ZookeeperConnectString
with the value that you saved after you ran thedescribe-cluster
command.bin/kafka-topics.sh --create --zookeeper
ZookeeperConnectString
--replication-factor 3 --partitions 1 --topic AWSKafkaTutorialTopicIf the command succeeds, you see the following message:
Created topic AWSKafkaTutorialTopic.
Next Step
Step 6: Produce and Consume Data