Step 6: Produce and Consume Data - Amazon Managed Streaming for Apache Kafka

Step 6: Produce and Consume Data

In this step of Getting Started Using Amazon MSK, you produce and consume data.

To produce and consume messages

  1. In this example we use the JVM truststore to talk to the MSK cluster. To do this, first create a folder named /tmp on the client machine. Then, go to the bin folder of the Apache Kafka installation and run the following command, replacing JDKFolder with the name of your JDK folder. For example, the name of the JDK folder on your instance might be java-1.8.0-openjdk-1.8.0.201.b09-0.amzn2.x86_64. 

    cp /usr/lib/jvm/JDKFolder/jre/lib/security/cacerts /tmp/kafka.client.truststore.jks

  2. While still in the bin folder of the Apache Kafka installation on the client machine, create a text file named client.properties with the following contents. 

    security.protocol=SSL
ssl.truststore.location=/tmp/kafka.client.truststore.jks

  3. Run the following command, replacing ClusterArn with the Amazon Resource Name (ARN) that you obtained at the end of Step 3: Create an Amazon MSK Cluster. 

    aws kafka get-bootstrap-brokers --region us-east-1 --cluster-arn ClusterArn

    From the JSON result of the command, save the value associated with the string named "BootstrapBrokerString" because you need it in the following commands.

  4. Run the following command, replacing BootstrapBrokerString with the value that you obtained when you ran the previous command. 

    kafka-console-producer.sh --broker-list BootstrapBrokerString --producer.config client.properties --topic AWSKafkaTutorialTopic

  5. Enter any message that you want, and press Enter. Repeat this step two or three times. Every time you enter a line and press Enter, that line is sent to your Apache Kafka cluster as a separate message.

  6. Keep the connection to the client machine open, and then open a second, separate connection to that machine in a new window.

  7. In the following command, replace BootstrapBrokerString with the value that you saved earlier. Then, go to the bin folder and run the command using your second connection to the client machine. 

    kafka-console-consumer.sh --bootstrap-server BootstrapBrokerString --consumer.config client.properties --topic AWSKafkaTutorialTopic --from-beginning

    You start seeing the messages you entered earlier when you used the console producer command. These messages are TLS encrypted in transit.

  8. Enter more messages in the producer window, and watch them appear in the consumer window.

Next Step

Step 7: Use Amazon CloudWatch to View Amazon MSK Metrics