Example: Extract Fields from an Apache Log - Amazon CloudWatch Logs

Example: Extract Fields from an Apache Log

Sometimes, instead of counting, it is helpful to use values within individual log events for metric values. This example shows how you can create an extraction rule to create a metric that measures the bytes transferred by an Apache webserver.

This extraction rule matches the seven fields of the log event. The metric value is the value of the seventh matched token. You can see the reference to the token as "$7" in the metricValue field of the extraction rule.

To create a metric filter using the CloudWatch console

  1. Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/.

  2. In the navigation pane, choose Log groups.

  3. Choose the name of the log group for the Apache server.

  4. Choose Actions, Create metric filter.

  5. For Filter Pattern, type [ip, id, user, timestamp, request, status_code, size].

  6. To test your filter pattern, choose Test Pattern.

  7. Choose Assign Metric, and then for Filter Name, type size.

  8. Under Metric Details, for Metric Namespace, type MyNameSpace.

  9. For Metric Name, type BytesTransferred

  10. For Metric Value, enter $size.

  11. For Default Value enter 0, and then choose Next.

  12. Choose Create metric filter.

To create a metric filter using the AWS CLI

At a command prompt, run the following command

aws logs put-metric-filter \ --log-group-name MyApp/access.log \ --filter-name BytesTransferred \ --filter-pattern '[ip, id, user, timestamp, request, status_code, size]' \ --metric-transformations \ metricName=BytesTransferred,metricNamespace=MyNamespace,metricValue=$size,defaultValue=0

You can use the following data in put-log-event calls to test this rule. This generates two different metrics if you did not remove monitoring rule in the previous example.

127.0.0.1 - - [24/Sep/2013:11:49:52 -0700] "GET /index.html HTTP/1.1" 404 287 127.0.0.1 - - [24/Sep/2013:11:49:52 -0700] "GET /index.html HTTP/1.1" 404 287 127.0.0.1 - - [24/Sep/2013:11:50:51 -0700] "GET /~test/ HTTP/1.1" 200 3 127.0.0.1 - - [24/Sep/2013:11:50:51 -0700] "GET /favicon.ico HTTP/1.1" 404 308 127.0.0.1 - - [24/Sep/2013:11:50:51 -0700] "GET /favicon.ico HTTP/1.1" 404 308 127.0.0.1 - - [24/Sep/2013:11:51:34 -0700] "GET /~test/index.html HTTP/1.1" 200 3