Option 3 - AWS Streaming Data Solution for Amazon MSK

Option 3

Option 3: Deploy the aws-streaming-data-solution-for-msk-using-aws-lambda-and-kenisis-data-firehose CloudFormation template

Before you launch this template, review the architecture and other considerations in this guide. Follow the step-by-step instructions in this section to configure and deploy the solution into your account.

Time to deploy: Approximately 10 minutes

Launch the Stack

Note

You are responsible for the cost of the AWS services used while running this solution. Refer to the Cost section for more details. For full details, refer to the pricing webpage for each AWS service used in this solution.

  1. Sign in to the AWS Management Console and use the button below to launch the aws-streaming-data-solution-for-msk-using-aws-lambda-and-kinesis-data-firehose AWS CloudFormation template.

    
                                    AWS Streaming Data Solution for Amazon MSK using AWS Lambda
                                        and Kinesis Data Firehose launch button

    Alternatively, you can download the template as a starting point for your own implementation.

  2. The template launches in the US East (N. Virginia) Region by default. To launch this solution in a different AWS Region, use the Region selector in the console navigation bar.

  3. On the Create stack page, verify that the correct template URL shows in the Amazon S3 URL text box and choose Next.

  4. On the Specify stack details page, assign a name to your solution stack. For information about naming character limitations, refer to IAM and STS Limits in the AWS Identity and Access Management User Guide.

  5. Under Parameters, review the parameters for the template and modify them as necessary. This solution uses the following default values.

    Parameter Default Description
    AWS Lambda consumer configuration
    ARN of the MSK cluster

    (ClusterArn)

    <Requires input>

    ARN of the Amazon MSK cluster.

    Maximum number of items to retrieve in a single batch

    (BatchSize)

    100

    The maximum number of records to retrieve in a single batch. The allowed range is from 1 to 10000 hours.

    Name of a Kafka topic to consume

    (TopicName)

    <Requires input>

    The name of the Apache Kafka topic to consume.

    Amazon Kinesis Data Firehose configuration
    Size of the buffer (in MBs) that incoming data is buffered before delivery

    (BufferingSize)

    5

    The size to buffer incoming data before delivering to S3. The allowed range is from 1 to 128.

    Length of time (in seconds) that incoming data is buffered before delivery

    (BufferingInterval)

    300

    The amount of time to buffer incoming data before delivering to S3. The allowed range is from 60 to 900.

    Compression format for delivered data in Amazon S3

    (CompressionFormat)

    GZIP

    The format of data once it’s delivered to S3. Allowed values are GZIP, HADOOP_SNAPPY, Snappy, UNCOMPRESSED, and ZIP.

  6. Choose Next.

  7. On the Configure stack options page, choose Next.

  8. On the Review page, review and confirm the settings. Check the box acknowledging that the template will create AWS Identity and Access Management (IAM) resources.

  9. Choose Create stack to deploy the stack.

    You can view the status of the stack in the AWS CloudFormation console in the Status column. You should receive a CREATE_COMPLETE status in approximately ten minutes.