Identify and alert when Amazon Data Firehose resources are not encrypted with an AWS KMS key

Created by Ram Kandaswamy (AWS)

Environment: Production

Technologies: Management & governance; Analytics; Big data; Cloud-native; Infrastructure; Security, identity, compliance

AWS services: AWS CloudTrail; Amazon CloudWatch; AWS Identity and Access Management; Amazon Kinesis; AWS Lambda; Amazon SNS

Summary

For compliance, some organizations must have encryption enabled on data delivery resources such as Amazon Data Firehose. This pattern shows a way to monitor, detect, and notify when resources are out of compliance.

To maintain the encryption requirement, this pattern can be used on Amazon Web Services (AWS) to provide automated monitoring and detection of Firehose delivery resources that are not encrypted with AWS Key Management Service (AWS KMS) key. The solution sends alert notifications, and it can be extended to perform automatic remediation. This solution can be applied to an individual account or a multiple-account environment, such as an environment using AWS Landing Zone or AWS Control Tower.

Prerequisites and limitations

Prerequisites 

• Firehose delivery stream

• Sufficient permissions and familiarity with AWS CloudFormation, which is used in this infrastructure automation

Limitations 

The solution is not real time because it uses AWS CloudTrail events for detection, and there is a delay between the time an unencrypted resource is created and the notification is sent.

Architecture

Target technology stack  

The solution uses serverless technology and the following services:

• AWS CloudTrail

• Amazon CloudWatch

• AWS Command Line Interface (AWS CLI)

• AWS Identity and Access Management (IAM)

• Amazon Data Firehose

• AWS Lambda

• Amazon Simple Notification Service (Amazon SNS)

Target architecture 

1. A user creates or modifies Firehose.

2. A CloudTrail event is detected and matched.

3. Lambda is invoked.

4. Noncompliant resources are identified.

5. Email notification is sent.

Automation and scale

Using AWS CloudFormation StackSets, you can apply this solution to multiple AWS Regions or accounts with a single command.

Tools

• AWS CloudTrail – AWS CloudTrail is an AWS service that helps you enable governance, compliance, and operational and risk auditing of your AWS account. Actions taken by a user, role, or an AWS service are recorded as events in CloudTrail. Events include actions taken in the AWS Management Console, AWS Command Line Interface, and AWS SDKs and API operations.

• Amazon CloudWatch Events – Amazon CloudWatch Events delivers a near-real-time stream of system events that describe changes in AWS resources.

• AWS CLI – AWS Command Line Interface (AWS CLI) is an open source tool that enables you to interact with AWS services using commands in your command line shell. 

• IAM – AWS Identity and Access Management (IAM) is a web service that helps you securely control access to AWS resources. You use IAM to control who is authenticated (signed in) and authorized (has permissions) to use resources. 

• Amazon Data Firehose – Amazon Data Firehose is a fully managed service for delivering real-time streaming data. With Firehose, you don't need to write applications or manage resources. You configure your data producers to send data to Firehose, and it automatically delivers the data to the destination that you specified.

• AWS Lambda – AWS Lambda is a compute service that supports running code without provisioning or managing servers. Lambda runs your code only when needed and scales automatically, from a few requests per day to thousands per second. You pay only for the compute time that you consume—there is no charge when your code is not running. 

• Amazon SNS – Amazon Simple Notification Service (Amazon SNS) is a managed service that provides message delivery from publishers to subscribers (also known as producers and consumers).

Epics

Enforce encryption for compliance

Task	Description	Skills required
Deploy AWS CloudFormation StackSets.	In the AWS CLI, use the firehose-encryption-checker.yaml template (attached) to create the stack set by running the following command.  Provide a valid Amazon SNS topic Amazon Resource Name (ARN) for the parameter. The deployment should successfully create CloudWatch Events rules, the Lambda function, and an IAM role with the necessary permissions as described in the template. aws cloudformation create-stack-set    --stack-set-name my-stack-set   --template-body file://firehose-encryption-checker.yaml	Cloud architect, Systems administrator
Create stack instances.	Stacks need to be created in the AWS Regions of your choice as well as in one or more accounts.  To create stack instances, run the following command, replacing the stack name, account numbers, and Regions with your own. aws cloudformation create-stack-instances     --stack-set-name my-stack-set    --accounts 123456789012 223456789012   --regions us-east-1 us-east-2 us-west-1 us-west-2     --operation-preferences FailureToleranceCount=1	Cloud architect, Systems administrator

Related resources

• Working with AWS CloudFormation StackSets

• What is Amazon CloudWatch Events?

Additional information

AWS Config does not support the Firehose delivery stream resource type, so an AWS Config rule cannot be used in the solution.

Attachments

To access additional content that is associated with this document, unzip the following file: attachment.zip