Menu
Cost Optimization Monitor
Cost Optimization Monitor

Automated Deployment

Before you launch the automated deployment, please review the architecture, configuration, and other information discussed in this guide. Follow the step-by-step instructions in this section to configure and deploy the Cost Optimization Monitor into your account.

Time to deploy: Approximately 25 minutes

Prerequisites

Customers who use Consolidated Billing to manage payments for multiple accounts must launch this template in the payer account. For more information on Consolidated Billing, see the AWS Billing and Cost Management User Guide.

What We'll Cover

The procedure for deploying this architecture on AWS consists of the following steps. For detailed instructions, follow the links for each step.

Step 1. Launch the Stack

  • Launch the AWS CloudFormation template into your AWS account.

  • Enter values for required parameters: Stack Name, User Name, Password Access CIDR Block, SSH Key, Domain Name

  • Review the other template parameters, and adjust if necessary.

Step 2. Configure Billing Reports to Save to Amazon S3

  • Configure detailed billing reports to save to the appropriate Amazon S3 bucket.

Step 3. Open the Kibana Dashboard

  • View the first batch of billing data in the preconfigured dashboard.

Step 1. Launch the Stack

This automated AWS CloudFormation template deploys the Cost Optimization Monitor on the AWS Cloud.

Note

You are responsible for the cost of the AWS services used while running this solution. See the Cost section for more details. For full details, see the pricing webpage for each AWS service you will be using in this solution.

  1. Log in to the AWS Management Console and click the button below to launch the cost-optimization-monitor AWS CloudFormation template.

    
                                Cost Optimization Monitor launch button

    You can also download the template as a starting point for your own implementation.

  2. The template is launched in the US East (N. Virginia) by default. To launch the Cost Optimization Monitor in a different AWS Region, use the region selector in the console navigation bar.

    Note

    This solution uses AWS Lambda, which is currently available in specific AWS Regions only. Therefore, you must deploy this solution an AWS Region where Lambda is available. For the most current service availability by region, see AWS service offerings by region.

  3. On the Select Template page, verify that you selected the correct template and choose Next.

  4. On the Specify Details page, assign a name to your Cost Optimization Monitor stack.

  5. Under Parameters, review the parameters for the template, and modify them as necessary. This solution uses the following default values.

    Parameter Default Description
    User Name <Requires Input> User name for access to the Amazon EC2 instance
    Password <Requires Input> Password for access to the Amazon EC2 instance

    Note

    Must be six characters or longer and must contain one uppercase letter, one lower case letter, and a special character (!@#$%^&+)

    Access CIDR Block <Requires Input> This IP address range will have HTTP and SSH access to the Amazon EC2 instance.
    SSH Key <Requires Input> Public and private key pair, which allows you to connect securely to the Amazon EC2 instance. When you created an AWS account, this is the key pair you created in your preferred AWS Region.
    Domain Name <Requires Input> The name of the Amazon ES domain that this template will create.

    Note

    Amazon ES domain names must start with a lowercase letter and must be between 3 and 28 characters. Valid characters are a-z (lowercase only), 0-9, and – (hyphen).

    Cluster Size Small A drop-down box with three options: Small, Medium, Large
    Use Existing Bucket? No Choose Yes to use an existing Amazon S3 bucket. Otherwise, choose No to create one at launch.
    Existing S3 Bucket Name <Requires Input> If you chose Yes above, enter an existing bucket name. Otherwise, leave this field blank.
    VPC CIDR Block 10.255.0.0/16 CIDR block for the solution’s VPC. You can modify the address range to avoid overlapping with existing networks.
    1st Subnet Network 10.250.250.0/24 CIDR block for the VPC subnet created in AZ1
    2nd Subnet Network 10.250.251.0/24 CIDR block for the VPC subnet created in AZ2
    Send Anonymous Usage Data Yes Send anonymous data to AWS to help us understand solution usage and related cost savings across our customer base as a whole. To opt out of this feature, choose No. For more information, see Appendix B.
  6. Choose Next.

  7. On the Options page, you can specify tags (key-value pairs) for resources in your stack and set additional options, and then choose Next.

  8. On the Review page, review and confirm the settings. Be sure to check the box acknowledging that the template will create AWS Identity and Access Management (IAM) resources.

  9. Choose Create to deploy the stack.

    You can view the status of the stack in the AWS CloudFormation console in the Status column. You should see a status of CREATE_COMPLETE in roughly 20 minutes.

  10. To see details for the stack resources, choose the Outputs tab. The following table describes some of these outputs in more detail.

    Key Description
    BucketName

    Amazon S3 bucket for storing detailed billing records

    SingleDashboardURL

    For single AWS accounts: the URL for front-end access to the Kibana 4 dashboard via the proxy server

    Note

    Use this link to open the Kibana dashboard webpage. It will not have billing data to show immediately. You must wait for the first batch of detailed billing reports from AWS Billing and Cost Management, and then for the python scripts to parse and upload that data. This occurs daily as a scheduled cron job.

    ConsolidatedDashboardURL

    For AWS accounts using Consolidated Billing: the URL for front-end access to the Kibana 4 dashboard via the proxy server

Note

This solution deploys an AWS Lambda function, solution-helper, which runs only during initial configuration or when resources are updated or deleted. You will see the solution-helper function in the AWS Lambda console, which is necessary to manage associated resources for as long as the solution is running.

Step 2. Configure Billing Reports to Save to Amazon S3

After the Cost Optimization Monitor stack launch completes, complete the following procedure to configure your account to save detailed billing records to the Amazon S3 bucket that this solution created.

  1. In the AWS CloudFormation console, in the Outputs tab, select and copy the name of the S3 bucket (the BucketName description).

  2. Open the Billing and Cost Management console. (To navigate to the console, select Billing & Cost Management from the drop-down box field under your role in the top menu bar.)

  3. On the left navigation pane, choose Preferences.

  4. Select the Receive Billing Reports check box.

  5. In the Save to S3 Bucket field, paste the bucket name and choose Verify to make sure the bucket policy has been configured with the correct permissions (the solution’s AWS CloudFormation template configures the policy automatically).

  6. Under Report, select the Detailed billing report with resources and tags* check box and choose Save preferences.

It can take up to 24 hours for AWS to start delivering detailed billing reports to your S3 bucket. Once they are delivered, they will be processed with the next scheduled run of the Python scripts. As more billing data is generated, the detailed billing report data is overwritten in the bucket.

Step 3. Open the Kibana Dashboard

A Kibana dashboard displays a group of visualizations that you can modify, save, and share. The sample visualizations for this solution display information such as cost by resource tag, Amazon EC2 instances running each hour, and the percentage of instances using the Reserved Instance or Spot instance pricing model. See Appendix A for detailed information on each cost metric in the default dashboard.

The solution uses a cron expression (cron job) to run the Python scripts and process the detailed billing report data daily at 2330 UTC (current month’s data) and midnight UTC (previous month’s data). As mentioned previously, it can take up to 24 hours to receive your first detailed billing reports. Once the reports are delivered to your S3 bucket, they will be processed with the next scheduled Python run. The solution automatically imports your data to Kibana and configures an initial dashboard. After your first batch of data is processed, you will be able to see sample visualizations in the Kibana dashboard.

  1. Go to the AWS CloudFormation console, and in the Outputs tab, open the applicable link (SingleDashboardURL or ConsolidatedDashboardURL) to go to the Kibana dashboard.

  2. When prompted, log in to the dashboard with the user name and password you specified in Step 1. Launch the Stack.

  3. A default dashboard will load. In the upper-right corner, you can adjust the data time period (clock icon). You can also set an interval for the webpage refresh rate (Auto-refresh).

  4. In the top menu under Settings, choose the Objects tab. You will see the default saved dashboards for this solution.

Explore and experiment with the dashboard settings to create and save additional visualizations. For more information, go to the Kibana User Guide.


                Example Kibana dashboard

Figure 2: Example Kibana dashboard