Automate encryption enforcement in AWS Glue using an AWS CloudFormation template - AWS Prescriptive Guidance
SummaryPrerequisites and limitationsArchitectureToolsBest practicesEpicsRelated resources

Automate encryption enforcement in AWS Glue using an AWS CloudFormation template

Created by Diogo Guedes (AWS)

Code repository: AWS Glue Encryption Enforcement

Environment: Production

Technologies: Analytics; Security, identity, compliance

Workload: All other workloads

AWS services: Amazon EventBridge; AWS Glue; AWS KMS; AWS Lambda; AWS CloudFormation

Summary

This pattern shows you how to set up and automate encryption enforcement in AWS Glue by using an AWS CloudFormation template. The template creates all the required configurations and resources for enforcing encryption. These resources include an initial configuration, a preventive control created by an Amazon EventBridge rule, and an AWS Lambda function.

Prerequisites and limitations

Prerequisites 

  • An active AWS account

  • Permissions to deploy the CloudFormation template and its resources

Limitations 

This security control is regional. You must deploy the security control in each AWS Region where you want to set up encryption enforcement in AWS Glue.

Architecture

Target technology stack  

  • Amazon CloudWatch Logs (from AWS Lambda)

  • Amazon EventBridge rule

  • AWS CloudFormation stack

  • AWS CloudTrail

  • AWS Identity and Access Management (IAM) managed role and policy

  • AWS Key Management Service (AWS KMS)

  • AWS KMS alias

  • AWS Lambda function

  • AWS Systems Manager Parameter Store

Target architecture

The following diagram shows how to automate encryption enforcement in AWS Glue.

Diagram shows how to automate encryption enforcement in AWS Glue using a CloudFormation template.

The diagram shows the following workflow:

  1. A CloudFormation template creates all the resources, including the initial configuration and detective control for encryption enforcement in AWS Glue.

  2. An EventBridge rule detects a state change in the encryption configuration.

  3. A Lambda function is invoked for evaluation and logging through CloudWatch Logs. For non-compliant detection, the Parameter Store is recovered with an Amazon Resource Name (ARN) for an AWS KMS key. The service is remediated to compliant status with encryption enabled.

Automation and scale

If you’re using AWS Organizations, you can use AWS CloudFormation StackSets to deploy this template in multiple accounts where you want to enable encryption enforcement in AWS Glue.

Tools

  • Amazon CloudWatch helps you monitor the metrics of your AWS resources and the applications you run on AWS in real time.

  • Amazon EventBridge is a serverless event bus service that helps you connect your applications with real-time data from a variety of sources. For example, Lambda functions, HTTP invocation endpoints using API destinations, or event buses in other AWS accounts.

  • AWS CloudFormation helps you set up AWS resources, provision them quickly and consistently, and manage them throughout their lifecycle across AWS accounts and Regions.

  • AWS CloudTrail helps you enable operational and risk auditing, governance, and compliance of your AWS account.

  • AWS Glue is a fully managed extract, transform, and load (ETL) service. It helps you reliably categorize, clean, enrich, and move data between data stores and data streams.

  • AWS Key Management Service (AWS KMS) helps you create and control cryptographic keys to help protect your data.

  • AWS Lambda is a compute service that helps you run code without needing to provision or manage servers. It runs your code only when needed and scales automatically, so you pay only for the compute time that you use.

  • AWS Systems Manager helps you manage your applications and infrastructure running in the AWS Cloud. It simplifies application and resource management, shortens the time to detect and resolve operational problems, and helps you manage your AWS resources securely at scale.

Code

The code for this pattern is available in the GitHub aws-custom-guardrail-event-driven repository.

Best practices

AWS Glue supports data encryption at rest for authoring jobs in AWS Glue and developing scripts using development endpoints.

Consider the following best practices:

  • Configure ETL jobs and development endpoints to use AWS KMS keys to write encrypted data at rest.

  • Encrypt the metadata stored in the AWS Glue Data Catalog by using keys that you manage through AWS KMS.

  • Use AWS KMS keys to encrypt job bookmarks and the logs generated by crawlers and ETL jobs.

Epics

TaskDescriptionSkills required

Deploy the CloudFormation template.

Download the aws-custom-guardrail-event-driven.yaml template from the GitHub repository, and then deploy the template. The CREATE_COMPLETE status indicates that your template was successfully deployed.

Note: The template requires no input parameters.

Cloud architect
TaskDescriptionSkills required

Check the AWS KMS key configurations.

  1. Sign in to the AWS Management Console and then open the AWS Glue console.

  2. In the navigation pane, under Data Catalog, choose Catalog settings.

  3. Verify that the Metadata encryption and Encrypt connection passwords settings are flagged and configured to use KMSKeyGlue.

Cloud architect
TaskDescriptionSkills required

Identify the encryption setting in CloudFormation.

  1. Sign in to the AWS Management Console and then open the CloudFormation console.

  2. In the navigation pane, choose Stacks, and then choose your stack.

  3. Choose the Resources tab.

  4. In the Resources table, find the encryption setting by Logical ID.

Cloud architect

Switch the provisioned infrastructure to an uncompliant state.

  1. Sign in to the AWS Management Console and then open the AWS Glue console.

  2. In the navigation pane, under Data Catalog, choose Catalog settings.

  3. Clear the Metadata encryption check box.

  4. Clear the Encrypt connection passwords check box.

  5. Choose Save.

  6. Refresh the AWS Glue console.

The guardrail detects the uncompliant state in AWS Glue after you clear the check boxes, and then enforces compliance by automatically remediating the encryption misconfiguration. As a result, the encryption check boxes should again be selected after you refresh the page.

Cloud architect

Related resources