Streaming Analytics Pipeline
Streaming Analytics Pipeline

Architecture Overview

Deploying this solution with the default parameters builds the following environment in the AWS Cloud.

        Streaming Analytics Pipeline architectural overview

Figure 1: Streaming Analytics Pipeline default architecture on AWS

By default, the AWS CloudFormation template creates a new Kinesis stream with two shards, an Amazon Kinesis Firehose delivery stream that encrypts data with AWS Key Management Service, an Amazon Simple Storage Service (Amazon S3) bucket to store raw and analyzed data, and an AWS Identity and Access Management (IAM) role with least-privilege access permissions. The template also launches an AWS Lambda custom resource that creates an Amazon Kinesis Analytics application based on settings you specify in a YAML configuration file. For more information, see Appendix B. The application consumes records from the source Kinesis stream and puts records into the Amazon Kinesis Firehose delivery stream.


If you do not specify a YAML configuration file, the Amazon Kinesis Analytics application will require further modification through the AWS Management Console and/or the service API to efficiently analyze your data.

If you choose to persist raw data, an AWS Lambda function is deployed. The Lambda function gets raw records from the source Kinesis stream, decodes the Base64-encoded data, batches the records, and puts them into another Amazon Kinesis Firehose delivery stream for delivery to Amazon S3.

The Streaming Analytics Pipeline can be customized to fit your needs. When you deploy the solution, you can specify an existing Kinesis stream, a configuration for an Amazon Kinesis Analytics application, whether or not to encrypt the data, and whether or not to persist raw data from your source Kinesis stream to Amazon S3. You can also choose from four destinations for your analyzed data: an Amazon S3 bucket (default), a pre-configured Amazon Redshift cluster, a pre-configured Amazon Elasticsearch Service domain, or an existing Kinesis stream.

        Streaming Analytics Pipeline architectural overview

Figure 2: Streaming Analytics Pipeline architecture on AWS