Menu
Amazon CloudWatch
Developer Guide (API Version 2010-08-01)

Filter and Pattern Syntax

You use metric filters to search for and match terms, phrases, or values in your log events. When a metric filter finds one of the terms, phrases, or values in your log events, it counts each occurrence in a CloudWatch metric. For example, you can create a metric filter to search for and count the occurrence of the word ERROR in your log events. Metric filters can also extract values from space-delimited log events, such as the latency of web requests. You can also use conditional operators and wildcards to create exact matches. Before you create a metric filter, you can test your search patterns in the CloudWatch console. The following sections explain the metric filter syntax in more detail.

Matching Terms in Log Events

To search for a term such as ERROR in your log events, you would use ERROR as your metric filter pattern. You can specify multiple terms in a metric filter pattern, but all terms must appear in your log events for there to be a match. For example, the filter pattern ERROR would match the following log event messages:

  • [ERROR] A fatal exception has occurred

  • Exiting with ERRORCODE: -1

The filter pattern ERROR Exception would match the following log event messages:

  • [ERROR] Caught IllegalArgumentException

  • [ERROR] Unhandled Exception

The filter pattern "Failed to process the request" would match the following log event messages:

  • [WARN] Failed to process the request

  • [ERROR] Unable to continue: Failed to process the request

Note

Metric filter terms that include characters other than alphanumeric or underscore must be placed inside double quotes (""). Metric filters are case sensitive.

Matching Terms in JSON Log Events

You can extract values from JSON log events. To extract values from JSON log events, you need to create a string-based metric filter. Strings containing scientific notation are not supported. The items in the JSON log event data must exactly match the metric filter. You might want to create metric filters in JSON log events to indicate whenever:

  • A certain event occurs. For example eventName is "UpdateTrail".

  • The IP is outside a known subnet. For example, sourceIPAddress is not in some known subnet range.

  • A combination of two or more other conditions are true. For example, the eventName is "UpdateTrail" and the recipientAccountId is 123456789012.

Using Metric Filters to Extract Values from Space-Delimited Log Events

You can use metric filters to extract values from space-delimited log events. The characters between a pair of square brackets [] or two double quotes ("") are treated as a single field. For example:

127.0.0.1 - frank [10/Oct/2000:13:25:15 -0700] "GET /apache_pb.gif HTTP/1.0" 200 1534
127.0.0.1 - frank [10/Oct/2000:13:35:22 -0700] "GET /apache_pb.gif HTTP/1.0" 500 5324
127.0.0.1 - frank [10/Oct/2000:13:50:35 -0700] "GET /apache_pb.gif HTTP/1.0" 200 4355

To specify a metric filter pattern that parses space-delimited events, the metric filter pattern has to specify the fields with a name, separated by commas, with the entire pattern enclosed in square brackets. For example: [ip, user, username, timestamp, request, status_code, bytes].

In cases where you don't know the number of fields, you can use shorthand notification using an ellipsis (…). For example:

[..., status_code, bytes]
[ip, user, ..., status_code, bytes]
[ip, user, ...]

You can also add conditions to your fields so that only log events that match all conditions would match the filters. For example:

[ip, user, username, timestamp, request, status_code, bytes > 1000]
[ip, user, username, timestamp, request, status_code = 200, bytes]
[ip, user, username, timestamp, request, status_code = 4*, bytes]
[ip, user, username, timestamp, request = *html*, status_code = 4*, bytes]

CloudWatch Logs supports both string and numeric conditional fields. For string fields, you can use = or != operators with an asterisk (*).

For numeric fields, you can use the >, <, >=, <=, =, and != operators.

Using Metric Filters to Extract Values from JSON Log Events

You can use metric filters to extract values from JSON log events. The metric filter syntax for JSON log events uses the following format:

{ SELECTOR EQUALITY_OPERATOR STRING }

The metric filter must be enclosed in curly braces { }, to indicate this is a JSON expression. The metric filter is contains the following parts:

SELECTOR

Specifies what JSON property to check. Property selectors always start with dollar sign ($), which signifies the root of the JSON. Property selectors are alphanumeric strings that also support '-' and '_' characters. Array elements are denoted with [NUMBER] syntax, and must follow a property. Examples are: $.eventId, $.users[0], $.users[0].id, $.requestParameters.instanceId.

EQUALITY_OPERATOR

Can be either = or !=.

STRING

A string with or without quotes. You can use the asterisk '*' wildcard character to match any text at, before, or after a search term. For example, *Event will match PutEvent and GetEvent. Event* will match EventId and EventName. Ev*ent will only match the actual string Ev*ent. Strings that consist entirely of alphanumeric characters do not need to be quoted. Strings that have unicode and other characters such as ‘@,‘ ‘$,' ‘\,' etc. must be enclosed in double quotes to be valid.

Metric Filter Examples

{
  "eventType": "UpdateTrail",
  "sourceIPAddress": "111.111.111.111",
  "arrayKey": [
        "value",
        "another value"
  ],
  "objectList": [
       {
         "name": "a",
         "id": 1
       },
       {
         "name": "b",
         "id": 2
       }
  ],
  "SomeObject": null,
  "ThisFlag": true
}

The above JSON example would be matched by any of these filters:

{ $.eventType = "UpdateTrail" }

Filter on the event type being UpdateTrail.

{ $.sourceIPAddress != 123.123.* }

Filter on the IP address being outside the subnet 123.123 prefix.

{ $.arrayKey[0] = "value" }

Filter on the first entry in arrayKey being "value". If arrayKey is not an array this will be false.

{ $.objectList[1].id = 2 }

Filter on the second entry in objectList having a property called id = 2. If objectList is not an array this will be false. If the items in objectList are not objects or do not have an id property, this will be false.

{ $.SomeObject IS NULL }

Filter on SomeObject being set to null. This will only be true is the specified object is set to null.

{ $.SomeOtherObject NOT EXISTS }

Filter on SomeOtherObject being non-existent. This will only be true if specified object does not exist in log data.

{ $.ThisFlag IS TRUE }

Filters on ThisFlag being TRUE. This also works for boolean filters which check for FALSE value.

Compound Conditions

You can combine multiple conditions into a compound expression using OR (||) and AND (&&). Parenthesis are allowed and the syntax follows standard order of operations () > && > ||.

{
    "user": {
        "id": 1, 
        "email": "John.Stiles@example.com"
    },
    "users": [
        {
         "id": 2,
         "email": "John.Doe@example.com"
        },
        {
         "id": 3,
         "email": "Jane.Doe@example.com"
        }
    ],
    "actions": [
        "GET",
        "PUT",
        "DELETE"
    ],
    "coordinates": [
        [0, 1, 2],
        [4, 5, 6],
        [7, 8, 9]
    ]
}

Examples

{ ($.user.id = 1) && ($.users[0].email = "John.Doe@example.com") }

Matches the JSON above.

{ ($.user.id = 2 && $.users[0].email = "nonmatch") || $.actions[2] = "GET" }

Doesn't match the JSON above.

{ $.user.email = "John.Stiles@example.com" || $.coordinates[0][1] = nonmatch && $.actions[2] = nomatch }

Matches the JSON above.

{ ($.user.email = "John.Stiles@example.com" || $.coordinates[0][1] = nonmatch) && $.actions[2] = nomatch }

Doesn't match the JSON above.

Special Considerations

The SELECTOR must point to a value node (string or number) in the JSON. If it points to an array or object, the filter will not be applied because the log format doesn't match the filter. For example, both {$.users = 1} and {$.users != 1} will fail to match a log event where users is an array:

{
  "users": [1, 2, 3]
}

Numeric Comparisons

The metric filter syntax supports precise matching on numeric comparisons. The following numeric comparisons are supported: <, >, >=, <=, =, !=

Numeric filters have a syntax of

{ SELECTOR NUMERIC_OPERATOR NUMBER }

The metric filter must be enclosed in curly braces { }, to indicate this is a JSON expression. The metric filter is contains the following parts:

SELECTOR

Specifies what JSON property to check. Property selectors always start with dollar sign ($), which signifies the root of the JSON. Property selectors are alphanumeric strings that also support '-' and '_' characters. Array elements are denoted with [NUMBER] syntax, and must follow a property. Examples are: $.latency, $.numbers[0], $.errorCode, $.processes[4].averageRuntime.

NUMERIC_OPERATOR

Can be one of the following: =, !=, <, >, <=, or >=.

NUMBER

An integer with an optional + or - sign, a decimal with an optional + or - sign, or a number in scientific notation, which is an integer or a decimal with an optional + or - sign, followed by 'e', followed by an integer with an optional + or - sign.

Examples:

{ $.latency >= 500 }
{ $.numbers[0] < 10e3 }
{ $.numbers[0] < 10e-3 }
{ $.processes[4].averageRuntime <= 55.5 }
{ $.errorCode = 400 }
{ $.errorCode != 500 }
{ $.latency > +1000 }

Metric Filter Value Extraction

You can publish numeric values found in JSON events as metric values to CloudWatch. The following procedure shows how to publish a metric with the latency found in the JSON request metricFilter: { $.latency = * } metricValue: $.latency.

To publish a metric with the latency in a JSON request

  1. Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/.

  2. If necessary, change the region. From the navigation bar, select the region that meets your needs. For more information, see Regions and Endpoints in the Amazon Web Services General Reference.

  3. In the navigation pane, click Logs.

  4. In the contents pane, select a log group, and then click Create Metric Filter.

  5. On the Define Logs Metric Filter screen, in the Filter Pattern field enter { $.latency = * }, and then click Assign Metric.

  6. On the Create Metric Filter and Assign a Metric screen, click Show advanced metric settings.

  7. In the Metric Name field, enter myMetric.

  8. In the Metric Value field, enter $.latency, and then click Create Filter.

The following log event would publish a value of 50 to the metric myMetric following filter creation.

{
"latency": 50,
"requestType": "GET"
}