Streaming Analytics Pipeline
Streaming Analytics Pipeline

Step 1. Launch the Stack

This automated AWS CloudFormation template deploys Streaming Analytics Pipeline on the AWS Cloud. Please make sure that you’ve configured your Amazon Redshift cluster or Amazon Elasticsearch Service domain before launching the stack, if you chose one of those as your destination.


You are responsible for the cost of the AWS services used while running this solution. See the Cost section for more details. For full details, see the pricing webpage for each AWS service you will be using in this solution.

  1. Log in to the AWS Management Console and click the button below to launch the streaming-analytics-pipeline AWS CloudFormation template.

                            Streaming Analytics Pipeline launch button

    You can also download the template as a starting point for your own implementation.

  2. The template is launched in the US East (N. Virginia) Region by default. To launch this solution in a different AWS Region, use the region selector in the console navigation bar.


    This solution uses AWS Lambda and Amazon Kinesis Data Analytics, which are currently available in the US East (N. Virginia) Region, the US West (Oregon) Region, and the EU (Ireland) Region. Therefore, you must launch this solution one of those regions. For the most current AWS Lambda availability by region, see the AWS service offerings by region.

  3. On the Select Template page, verify that you selected the correct template and choose Next.

  4. On the Specify Details page, assign a name to your Streaming Analytics Pipeline solution stack.

  5. Under Parameters, review the parameters for the template, and modify them as necessary. This solution uses the following default values.

    Parameter Default Description
    New or Existing Stream New Kinesis Stream The source Amazon Kinesis stream. Create a new stream or choose an existing stream.
    New Stream Shard Count <Requires input> The number of shards to allot to your new stream


    If you use an existing stream, leave this parameter blank.

    Existing Stream Name <Requires input> The name of an existing stream in the same AWS Region where you launch the solution


    If you use a new stream, leave this parameter blank.

    External Destination Amazon S3 The destination for your analyzed data. Select Amazon S3 (default), Amazon Redshift, Amazon Elasticsearch Service, or Kinesis stream.


    If you choose Amazon Redshift, Amazon Elasticsearch Service, or Kinesis stream, you must configure the destination. See Appendix A for steps to configure the destination.

    Configuration File Location <Requires input>

    The Amazon S3 bucket and key where the completed YAML configuration file is stored. For example, <bucket-name>/<key>.

    For information about the YAML file configuration, see Appendix B.

    Encrypt Data at Rest? Yes Specify whether or not the solution will create an AWS KMS encryption key, and encrypt raw and analyzed data in Amazon S3
    Persist Raw Source Data? Yes Specify whether or not the solution will persist raw streaming data from your source Kinesis stream to Amazon S3
    Destination Prefix AggregateData The prefix name that will be created in the Amazon S3 bucket


    Use this parameter only if you choose the default option (Amazon S3) as your destination.

    Buffer Interval 300 Specify the number of seconds (60-900) that Amazon Kinesis Data Firehose should buffer data before loading it to Amazon S3
    Buffer Size 5 Specify the size of data in MB (1-128) that Amazon Kinesis Data Firehose should buffer before loading it to Amazon S3
    Send Anonymous Usage Data Yes Send anonymous data to AWS to help us understand usage across our customer base as a whole. To opt out of this feature, select No. For more information, see Appendix D.
  6. Verify that you modified the correct parameters for your chosen destination.

  7. Choose Next.

  8. On the Options page, choose Next.

  9. On the Review page, review and confirm the settings. Be sure to check the box acknowledging that the template will create AWS Identity and Access Management (IAM) resources.

  10. Choose Create to deploy the stack.

    You can view the status of the stack in the AWS CloudFormation console in the Status column. You should see a status of CREATE_COMPLETE in roughly five (5) minutes.