Streaming Analytics Pipeline
Streaming Analytics Pipeline


When you build systems on AWS infrastructure, security responsibilities are shared between you and AWS. This shared model can reduce your operational burden as AWS operates, manages, and controls the components from the host operating system and virtualization layer down to the physical security of the facilities in which the services operate. For more information about security on AWS, visit the AWS Security Center.

Security Groups

The Streaming Analytics Pipeline does not create any security groups. However, we recommend that you follow best practices for least-privilege access when creating access rules for associated resources. If you selected an existing Amazon Redshift cluster as your external destination, and your cluster is in an Amazon VPC with a publicly available IP address, you must open the Amazon Redshift security group to the Amazon Kinesis Firehose CIDR block for your AWS Region. For more information, see Prerequisites.

IAM Roles

AWS Identity and Access Management (IAM) roles enable customers to assign granular access policies and permissions to services and users on the AWS Cloud. Depending on your configuration, the Streaming Analytics Pipeline creates between two and five IAM roles. The solution creates the following roles:

  • A role with granular access policies for each Amazon Kinesis Firehose delivery stream that the solution creates. The policies allow the Amazon Kinesis Firehose delivery streams to log their events, get a particular AWS KMS encryption key to encrypt data in a specific Amazon S3 prefix, and send streaming events to a specific Amazon S3 prefix (and to Amazon Elasticsearch Service if the customer has selected this as their external destination).

  • A role for the new Amazon Kinesis Analytics application. This role grants the application least-privilege permissions to get streaming records from the source Kinesis stream, put the analyzed results to a specific Kinesis Firehose delivery stream or Kinesis stream, and log its events.

  • A role for the AWS Lambda custom resource that creates the Amazon Kinesis Analytics application. This role has permission to create, delete, describe, and list applications, log its events, and get details from AWS CloudFormation and Amazon CloudWatch for configuration and to gather metrics.

  • A role that allows AWS Lambda function to get records from the source Kinesis stream, put batch events to Amazon Kinesis Firehose, and log its events. This role is only created if you choose to persist raw streaming data to Amazon S3.

  • A role that allows an Amazon CloudWatch rule to invoke an AWS Lambda function, which collects and sends anonymous metrics. This role is only created if you choose to send anonymous data to AWS.

AWS KMS Encryption

This solution allows you to encrypt your data at rest when it reaches the destination. If you choose to encrypt your data, the solution creates an AWS Key Management Service (AWS KMS) encryption key, and automatically configures the Amazon Kinesis Firehose delivery streams to use the key. By default, no services or users will have permission to use or control the AWS KMS encryption key. To set access policies for the key, set them manually in the AWS KMS console.