Example: Extract fields from an Apache log and assign dimensions
Sometimes, instead of counting, it is helpful to use values within individual log events for metric values. This example shows how you can create an extraction rule to create a metric that measures the bytes transferred by an Apache webserver.
This example also shows how to assign dimensions to the metric that you are creating.
To create a metric filter using the CloudWatch console
-
Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/
. -
In the navigation pane, choose Log groups.
-
Choose the name of the log group for the Apache server.
-
Choose
Actions
, Create metric filter. -
For Filter pattern, enter
[ip, id, user, timestamp, request, status_code, size]
. -
(Optional) To test your filter pattern, under Test Pattern, enter one or more log events to use to test the pattern. Each log event must be within one line, because line breaks are used to separate log events in the Log event messages box.
-
Choose Next, and then for Filter name, type
size
. -
Under Metric details, for Metric namespace, enter
MyNameSpace
. Because this is a new namespace, be sure that Create new is selected. -
For Metric name, enter
BytesTransferred
For Metric value, enter
$size
.For Unit, select Bytes.
For Dimension Name, type
IP
.For Dimension Value, type
$ip
and then choose Next.Choose Create metric filter.
To create this metric filter using the AWS CLI
At a command prompt, run the following command
aws logs put-metric-filter \ --log-group-name MyApp/access.log \ --filter-name BytesTransferred \ --filter-pattern '[ip, id, user, timestamp, request, status_code, size]' \ --metric-transformations \ metricName=BytesTransferred,metricNamespace=MyNamespace,metricValue='$size'
aws logs put-metric-filter \ --log-group-name MyApp/access.log \ --filter-name BytesTransferred \ --filter-pattern '[ip, id, user, timestamp, request, status_code, size]' \ --metric-transformations \ metricName=BytesTransferred,metricNamespace=MyNamespace,metricValue='$size',unit=Bytes,dimensions='{{IP=$ip}}'
Note
In this command, use this format to specify multiple dimensions.
aws logs put-metric-filter \ --log-group-name
my-log-group-name
\ --filter-namemy-filter-name
\ --filter-pattern 'my-filter-pattern
' \ --metric-transformations \ metricName=my-metric-name
,metricNamespace=my-metric-namespace
,metricValue=my-token
,unit=unit
,dimensions='{dimension1=$dim,dimension2=$dim2,dim3=$dim3}'
You can use the following data in put-log-event calls to test this rule. This generates two different metrics if you did not remove monitoring rule in the previous example.
127.0.0.1 - - [24/Sep/2013:11:49:52 -0700] "GET /index.html HTTP/1.1" 404 287 127.0.0.1 - - [24/Sep/2013:11:49:52 -0700] "GET /index.html HTTP/1.1" 404 287 127.0.0.1 - - [24/Sep/2013:11:50:51 -0700] "GET /~test/ HTTP/1.1" 200 3 127.0.0.1 - - [24/Sep/2013:11:50:51 -0700] "GET /favicon.ico HTTP/1.1" 404 308 127.0.0.1 - - [24/Sep/2013:11:50:51 -0700] "GET /favicon.ico HTTP/1.1" 404 308 127.0.0.1 - - [24/Sep/2013:11:51:34 -0700] "GET /~test/index.html HTTP/1.1" 200 3