Menu
Streaming Analytics Pipeline
Streaming Analytics Pipeline

Overview

Many Amazon Web Services (AWS) customers use streaming data to gain real-time insight into customer activity and immediate business trends. Streaming data is generated continuously from thousands of data sources, and this information can help companies make well-informed decisions and proactively respond to changing conditions.

Amazon Kinesis, a platform for streaming data on AWS, offers powerful services that make it easier to build data processing applications, load massive volumes of data from hundreds of thousands of sources, and analyze streaming data in real time.

Kinesis services include Amazon Kinesis Streams which enables you to build your own custom applications that process or analyze streaming data, Amazon Kinesis Firehose which captures and automatically loads streaming data into Amazon Simple Storage Service (Amazon S3) and Amazon Redshift for near real-time analytics with your existing business intelligence tools, and Amazon Kinesis Analytics which processes streaming data in real time with standard SQL.

To help customers more easily configure a streaming data architecture, AWS offers the Streaming Analytics Pipeline. This solution automatically provisions and configures the AWS services necessary to start consuming and analyzing streaming data in minutes. This solution uses Amazon Kinesis Streams to load streaming data, Amazon Kinesis Analytics to filter and process that data, and Amazon Kinesis Firehose to deliver the data to various data stores for search, storage, or further analytics.

Cost

You are responsible for the cost of the AWS services used while running the Streaming Analytics Pipeline. The total cost of this solution depends on the amount of data you stream through the Streaming Analytics Pipeline. As of the date of publication, the cost of running this solution with the default settings in the US East (N. Virginia) Region is approximately $1.38 per hour. The cost estimate assumes the solution will stream 1,000 records per second with an average size of three kilobytes per record, and that the external destination is an Amazon Simple Storage Service (Amazon S3) bucket. Prices are subject to change. For full details, see the pricing webpage for each AWS service you will use in this solution.

We recommend adjusting your AWS Lambda and Amazon Kinesis Firehose batch configurations as your record count and data size increase to manage costs.

On this page: