Option 4 - AWS Streaming Data Solution for Amazon MSK

Option 4

Option 4: Deploy the aws-streaming-data-solution-for-msk-using-kinesis-data-analytics-and-amazon-s3 CloudFormation template

Before you launch this template, review the architecture and other considerations in this guide. Follow the step-by-step instructions in this section to configure and deploy the solution into your account.

Time to deploy: Approximately 10 minutes

Launch the Stack

Note

You are responsible for the cost of the AWS services used while running this solution. Refer to the Cost section for more details. For full details, refer to the pricing webpage for each AWS service used in this solution.

  1. Sign in to the AWS Management Console and use the button below to launch the aws-streaming-data-solution-for-msk-using-kinesis-data-analytics-and-amazon-s3 AWS CloudFormation template.

    
                                AWS Streaming Data Solution for Amazon MSK using Kinesis Data Analytics and Amazon S3 launch template button

    Alternatively, you can download the template as a starting point for your own implementation.

  2. The template launches in the US East (N. Virginia) Region by default. To launch this solution in a different AWS Region, use the Region selector in the console navigation bar.

  3. On the Create stack page, verify that the correct template URL shows in the Amazon S3 URL text box and choose Next.

  4. On the Specify stack details page, assign a name to your solution stack. For information about naming character limitations, refer to IAM and STS Limits in the AWS Identity and Access Management User Guide.

  5. Under Parameters, review the parameters for the template and modify them as necessary. This solution uses the following default values.

    Parameter Default Description
    Amazon MSK cluster configuration
    ARN of the MSK cluster

    (ClusterArn)

    <Requires input>

    ARN of the Amazon MSK cluster.

    Name of a Kafka topic to consume (TopicName) <Requires input>

    The name of the Apache Kafka topic to consume.

    Amazon Kinesis Data Analytics configuration
    Monitoring log level (LogLevel) INFO

    The level of detail of the CloudWatch logs for an application. The available options include DEBUG, ERROR, INFO, and WARN. For information about choosing a log level, refer to Application Monitoring Levels in the Amazon Kinesis Data Analytics Developer Guide.

    Monitoring metrics level (MetricsLevel) TASK

    The granularity of the CloudWatch logs for an application. The available options include APPLICATION, OPERATOR, PARALLELISM, and TASK. For information about choosing a metrics level, refer to Application Monitoring Levels in the Amazon Kinesis Data Analytics Developer Guide.

    Enable service-triggered snapshots (EnableSnapshots) True

    Choose to enable or disable snapshots for an application. Options are true or false.

    Enable automatic scaling (EnableAutoScaling) True

    Choose to increase the Kinesis Data Analytics service the parallelism of the application in response to increased throughput. Options are true or false.

  6. Choose Next.

  7. On the Configure stack options page, choose Next.

  8. On the Review page, review and confirm the settings. Check the box acknowledging that the template will create AWS Identity and Access Management (IAM) resources.

  9. Choose Create stack to deploy the stack.

    You can view the status of the stack in the AWS CloudFormation console in the Status column. You should receive a CREATE_COMPLETE status in approximately ten minutes.

Post-configuration tasks

By default, the demo producer and demo consumer applications will not run after the stacks are created. Use the following process to enable them.

  1. Sign in to the Amazon Kinesis console and, from the left menu pane, select Analytics applications.

  2. On the Amazon Kinesis Data Analytics page, select Kda<application-name>.

  3. Select Actions.

  4. Choose Run application.