Publish Amazon CloudWatch metrics to a CSV file

Created by Abdullahi Olaoye (AWS)

Environment: PoC or pilot

Technologies: DevOps

AWS services: Amazon CloudWatch

Summary

This pattern uses a Python script to retrieve Amazon CloudWatch metrics and to convert the metrics information into a comma-separated values (CSV) file for improved readability. The script takes the AWS service whose metrics should be retrieved as a required argument. The AWS Region and AWS credential profile can be passed as optional arguments. If those optional arguments are not passed, the script uses the configured default Region and the default AWS credential that are configured for the workstation where the script is run. After the script runs, it generates and stores a CSV file in the same directory.

Prerequisites and limitations

Prerequisites 

• Python 3.x

• AWS Command Line Interface (AWS CLI)

Limitations 

The script currently supports the following AWS services:

• AWS Lambda

• Amazon Elastic Compute Cloud (Amazon EC2)

• Amazon Relational Database Service (Amazon RDS)

  • However, the script doesn't support Amazon Aurora

• Application Load Balancer

• Network Load Balancer

• Amazon API Gateway

Tools

• Amazon CloudWatch – Amazon CloudWatch is a monitoring and observability service built for DevOps engineers, developers, site reliability engineers (SREs), and IT managers. CloudWatch provides data and actionable insights to help you monitor your applications, respond to systemwide performance changes, optimize resource utilization, and get a unified view of operational health. CloudWatch collects monitoring and operational data in the form of logs, metrics, and events, and provides a unified view of AWS resources, applications, and services that run on AWS and on-premises servers.

Epics

Install and configure the prerequisites

Task	Description	Skills required
Install the prerequisites.	Run the following command: $ pip3 install -r requirements.txt	Developer
Configure the AWS CLI.	Run the following command:  $ aws configure	Developer

Configure the Python script

Task	Description	Skills required
Open the script.	To change the default configuration of the script, open metrics.yaml.	Developer
Set the period for the script.	This is the time period to fetch. The default period is 5 minutes (300 seconds). You can change the period, but note the following limitations:  If the hours value that you specify is between 3 hours and 15 days ago, use a multiple of 60 seconds (1 minute) for the period.  If the hours value that you specify is between 15 hours and 63 days ago, use a multiple of 300 seconds (5 minutes) for the period.  If the hours value that you specify is greater than 63 days ago, use a multiple of 3,600 seconds (1 hour) for the period Otherwise, the API operation won't return any data points.	Developer
Set the hours for the script.	This value specifies how many hours of metrics you want to fetch. The default is 1 hour. To retrieve multiple days of metrics, provide the value in hours. For example, for 2 days, specify 48.	Developer
Change statistics values for the script.	(Optional) The global statistics value is Average, which is used when fetching metrics that do not have a specific statistics value assigned. The script supports the statistics values Maximum, SampleCount, and Sum.	Developer

Run the Python script

Task	Description	Skills required
Run the script.	Use the following command:  $ python cwreport.py <service> To see a list of service values and the optional region and profile parameters, run the following command:  $ python cwreport.py -h For more information about the optional parameters, see the Additional information section.	Developer

Related resources

• Configuring the AWS CLI

• Using Amazon CloudWatch metrics

• Amazon CloudWatch documentation

Additional information

Script usage

$ python cwreport.py -h

Example syntax

python cwreport.py <service> <--region=Optional Region> <--profile=Optional credential profile>

Parameters

• service (required) - The service you want to run the script against. The script currently supports these services: AWS Lambda, Amazon EC2, Amazon RDS, Application Load Balancer, Network Load Balancer, and API Gateway.

• region (optional) - The AWS Region to fetch metrics from. The default Region is ap-southeast-1.

• profile (optional) - The AWS CLI named profile to use. If specified, the default configured credential isn't used.

Examples

• Using default Region ap-southeast-1 and default configured credentials to fetch Amazon EC2 metrics: $ python cwreport.py ec2

• Specifying a Region and fetching Amazon API Gateway metrics: $ python cwreport.py apigateway --region us-east-1

• Specifying an AWS profile: $ python cwreport.py ec2 --profile testprofile

• Specifying both Region and profile: $ python cwreport.py ec2 --region us-east-1 --profile testprofile

Attachments

To access additional content that is associated with this document, unzip the following file: attachment.zip