Option 3 - AWS Streaming Data Solution for Amazon MSK

Option 3

Deploying the aws-streaming-data-solution-for-msk-using-aws-lambda-and-kinesis-data-firehose AWS CloudFormation template builds the following environment in the AWS Cloud.

          AWS CloudFormation template using Amazon MSK, Lambda, and Amazon Kinesis Data Firehose reference

Figure 3: AWS CloudFormation template using Kinesis Data Streams, Kinesis Data Firehose, and S3 reference architecture

This AWS CloudFormation template deploys a reference architecture that does the following:

  1. A Lambda function that processes process records in a Kafka topic.

  2. A Kinesis Data Firehose delivery stream that buffers data before delivering it to the destination.

  3. An Amazon S3 bucket that stores all original events from the MSK cluster.