Create detailed cost and usage reports for AWS Glue jobs by using AWS Cost Explorer

Created by Parijat Bhide (AWS) and Aromal Raj Jayarajan (AWS)

Environment: Production

Technologies: Cost management; Analytics

AWS services: AWS Billing and Cost Management; AWS Glue

Summary

This pattern shows how to track the usage costs of AWS Glue data integration jobs by configuring user-defined cost allocation tags. You can use these tags to create detailed cost and usage reports in AWS Cost Explorer for jobs across multiple dimensions. For example, you can track usage costs at the team, project, or cost center level.

Prerequisites and limitations

Prerequisites

• An active AWS account

• One or more AWS Glue jobs that have user-defined tags activated

Architecture

Target technology stack

• AWS Glue

• AWS Cost Explorer

The following diagram shows how you can apply tags to track usage costs for AWS Glue jobs.

The diagram shows the following workflow:

1. A data engineer or AWS administrator creates user-defined cost allocation tags for the AWS Glue jobs.

2. An AWS administrator activates the tags.

3. The tags report metadata to AWS Cost Explorer.

Tools

• AWS Glue is a fully managed extract, transform, and load (ETL) service. It helps you reliably categorize, clean, enrich, and move data between data stores and data streams.

• AWS Cost Explorer helps you view and analyze your AWS costs and usage.

Epics

Create and activate tags for your AWS Glue jobs

Task	Description	Skills required
Create user-defined cost allocation tags for your AWS Glue jobs.	To add tags to an existing AWS Glue job Sign in to the AWS Management console, and then open the AWS Glue console. In the left navigation pane, under ETL, choose Jobs. In the Your jobs section, choose the name of the job that you’re tagging. Choose the Job details tab. Then, expand the Advanced properties section. For Tags, choose Add new tag. For Key, enter a name for your tag. (Optional) For Value, enter a value that you want associated with the key. (Optional) Repeat steps 5-7 for each tag that you want to create for the job. Choose Save. To add tags to a new AWS Glue job Create a new AWS Glue job based on your use case requirements. For instructions, see Working with jobs on the AWS Glue Console in the AWS Glue Developer Guide. When you configure the Job details settings, follow steps 4-9 of the To add tags to an existing AWS Glue job section of this task. Note: For more information, see AWS tags in AWS Glue in the AWS Glue Developer Guide.	Data engineer
Activate the user-defined cost allocation tags.	Follow the instructions in Activating user-defined cost allocation tags in the AWS Billing User Guide.	AWS administrator

Create cost and usage reports for your AWS Glue jobs

Task	Description	Skills required
Create cost and usage reports for your AWS Glue jobs by using tag filters in AWS Cost Explorer.	Sign in to the AWS Management Console and open the AWS Cost Management console. In the left navigation pane, choose Reports. Choose Create new report. For Select a report type, choose Cost and usage (recommended). Then, choose Create Report. For Filters, choose Service. The Service dropdown appears. Select the check boxes next to Glue. Then, choose Apply filters. For Filters, choose Tag. The Tag dropdown appears. Choose Team. Then, select the check boxes next to the teams that you’ve assigned tags to. Exclude any teams that you haven’t assigned tags to. Then, choose Apply filters. At the top of the chart, choose Tag. Then, choose the tags for the AWS Glue jobs that you want to create a report for. At the top of the chart, choose the Last 3 Months dropdown and choose the timeframe that you want the report to cover. Then, choose the Monthly dropdown and choose how you want the line items in the report to be aggregated based on timeframe. Choose Save as. Then, enter a title for your report. Choose Save Report. For more information, see Exploring your data using Cost Explorer in the AWS Cost Management User Guide.	General AWS, AWS administrator