Menu
Amazon CloudWatch Logs
User Guide

Filter and Pattern Syntax

You use metric filters to search for and match terms, phrases, or values in your log events. When a metric filter finds one of the terms, phrases, or values in your log events, it counts each occurrence in a CloudWatch metric. For example, you can create a metric filter to search for and count the occurrence of the word ERROR in your log events. Metric filters can also extract values from space-delimited log events, such as the latency of web requests. You can also use conditional operators and wildcards to create exact matches. Before you create a metric filter, you can test your search patterns in the CloudWatch console. The following sections explain the metric filter syntax in more detail.

Matching Terms in Log Events

To search for a term in your log events, use the term as your metric filter pattern. You can specify multiple terms in a metric filter pattern, but all terms must appear in a log event for there to be a match. Metric filters are case sensitive.

Metric filter terms that include characters other than alphanumeric or underscore must be placed inside double quotes ("").

To exclude a term, use a minus sign (-) before the term.

Example 1: Single term

The filter pattern "ERROR" matches log event messages that contain this term, such as the following:

  • [ERROR] A fatal exception has occurred

  • Exiting with ERRORCODE: -1

Example 2: Include a term and exclude a term

In the previous example, if you change the filter pattern to "ERROR -Exiting", the log event message "Exiting with ERRORCODE: -1" would be excluded.

Example 3: Multiple terms

The filter pattern "ERROR Exception" matches log event messages that contain both terms, such as the following:

  • [ERROR] Caught IllegalArgumentException

  • [ERROR] Unhandled Exception

The filter pattern "Failed to process the request" matches log event messages that contain all terms, such as the following:

  • [WARN] Failed to process the request

  • [ERROR] Unable to continue: Failed to process the request

Matching Terms in JSON Log Events

You can extract values from JSON log events. To extract values from JSON log events, you need to create a string-based metric filter. Strings containing scientific notation are not supported. The items in the JSON log event data must exactly match the metric filter. You might want to create metric filters in JSON log events to indicate the following:

  • A certain event occurs. For example eventName is "UpdateTrail".

  • The IP is outside a known subnet. For example, sourceIPAddress is not in some known subnet range.

  • A combination of two or more other conditions are true. For example, the eventName is "UpdateTrail" and the recipientAccountId is 123456789012.

Using Metric Filters to Extract Values from Space-Delimited Log Events

You can use metric filters to extract values from space-delimited log events. The characters between a pair of square brackets [] or two double quotes ("") are treated as a single field. For example:

127.0.0.1 - frank [10/Oct/2000:13:25:15 -0700] "GET /apache_pb.gif HTTP/1.0" 200 1534
127.0.0.1 - frank [10/Oct/2000:13:35:22 -0700] "GET /apache_pb.gif HTTP/1.0" 500 5324
127.0.0.1 - frank [10/Oct/2000:13:50:35 -0700] "GET /apache_pb.gif HTTP/1.0" 200 4355

To specify a metric filter pattern that parses space-delimited events, the metric filter pattern has to specify the fields with a name, separated by commas, with the entire pattern enclosed in square brackets. For example: [ip, user, username, timestamp, request, status_code, bytes].

In cases where you don't know the number of fields, you can use shorthand notification using an ellipsis (…). For example:

[..., status_code, bytes]
[ip, user, ..., status_code, bytes]
[ip, user, ...]

You can also add conditions to your fields so that only log events that match all conditions would match the filters. For example:

[ip, user, username, timestamp, request, status_code, bytes > 1000]
[ip, user, username, timestamp, request, status_code = 200, bytes]
[ip, user, username, timestamp, request, status_code = 4*, bytes]
[ip, user, username, timestamp, request = *html*, status_code = 4*, bytes]

CloudWatch Logs supports both string and numeric conditional fields. For string fields, you can use = or != operators with an asterisk (*).

For numeric fields, you can use the >, <, >=, <=, =, and != operators.

If you are using a space-delimited filter, extracted fields map to the names of the space-delimited fields (as expressed in the filter) to the value of each of these fields. If you are not using a space-delimited filter, this will be empty.

Example Filter:

[..., request=*.html*, status_code=4*,]

Example log event for the filter:

127.0.0.1 - frank [10/Oct/2000:13:25:15 -0700] \"GET /index.html HTTP/1.0\" 404 1534

Extracted fields for the log event and filter pattern:

{
   "$status_code": "400", 
   "$request": "GET /products/index.html HTTP/1.0", 
   "$7": "1534", 
   "$4": "10/Oct/2000:13:25:15 -0700", 
   "$3": "frank",
   "$2": "-", 
   "$1": "127.0.0.1"
}

Using Metric Filters to Extract Values from JSON Log Events

You can use metric filters to extract values from JSON log events. The metric filter syntax for JSON log events uses the following format:

{ SELECTOR EQUALITY_OPERATOR STRING }

The metric filter must be enclosed in curly braces { }, to indicate this is a JSON expression. The metric filter is contains the following parts:

SELECTOR

Specifies what JSON property to check. Property selectors always start with dollar sign ($), which signifies the root of the JSON. Property selectors are alphanumeric strings that also support '-' and '_' characters. Array elements are denoted with [NUMBER] syntax, and must follow a property. Examples are: $.eventId, $.users[0], $.users[0].id, $.requestParameters.instanceId.

EQUALITY_OPERATOR

Can be either = or !=.

STRING

A string with or without quotes. You can use the asterisk '*' wildcard character to match any text at, before, or after a search term. For example, *Event will match PutEvent and GetEvent. Event* will match EventId and EventName. Ev*ent will only match the actual string Ev*ent. Strings that consist entirely of alphanumeric characters do not need to be quoted. Strings that have unicode and other characters such as ‘@,‘ ‘$,' ‘\,' etc. must be enclosed in double quotes to be valid.

Metric Filter Examples

The following is a JSON example:

{
  "eventType": "UpdateTrail",
  "sourceIPAddress": "111.111.111.111",
  "arrayKey": [
        "value",
        "another value"
  ],
  "objectList": [
       {
         "name": "a",
         "id": 1
       },
       {
         "name": "b",
         "id": 2
       }
  ],
  "SomeObject": null,
  "ThisFlag": true
}

The following filters would match:

{ $.eventType = "UpdateTrail" }

Filter on the event type being UpdateTrail.

{ $.sourceIPAddress != 123.123.* }

Filter on the IP address being outside the subnet 123.123 prefix.

{ $.arrayKey[0] = "value" }

Filter on the first entry in arrayKey being "value". If arrayKey is not an array this will be false.

{ $.objectList[1].id = 2 }

Filter on the second entry in objectList having a property called id = 2. If objectList is not an array this will be false. If the items in objectList are not objects or do not have an id property, this will be false.

{ $.SomeObject IS NULL }

Filter on SomeObject being set to null. This will only be true is the specified object is set to null.

{ $.SomeOtherObject NOT EXISTS }

Filter on SomeOtherObject being non-existent. This will only be true if specified object does not exist in log data.

{ $.ThisFlag IS TRUE }

Filters on ThisFlag being TRUE. This also works for boolean filters which check for FALSE value.

Compound Conditions

You can combine multiple conditions into a compound expression using OR (||) and AND (&&). Parenthesis are allowed and the syntax follows standard order of operations () > && > ||.

{
    "user": {
        "id": 1, 
        "email": "John.Stiles@example.com"
    },
    "users": [
        {
         "id": 2,
         "email": "John.Doe@example.com"
        },
        {
         "id": 3,
         "email": "Jane.Doe@example.com"
        }
    ],
    "actions": [
        "GET",
        "PUT",
        "DELETE"
    ],
    "coordinates": [
        [0, 1, 2],
        [4, 5, 6],
        [7, 8, 9]
    ]
}

Examples

{ ($.user.id = 1) && ($.users[0].email = "John.Doe@example.com") }

Matches the JSON above.

{ ($.user.id = 2 && $.users[0].email = "nonmatch") || $.actions[2] = "GET" }

Doesn't match the JSON above.

{ $.user.email = "John.Stiles@example.com" || $.coordinates[0][1] = nonmatch && $.actions[2] = nomatch }

Matches the JSON above.

{ ($.user.email = "John.Stiles@example.com" || $.coordinates[0][1] = nonmatch) && $.actions[2] = nomatch }

Doesn't match the JSON above.

Special Considerations

The SELECTOR must point to a value node (string or number) in the JSON. If it points to an array or object, the filter will not be applied because the log format doesn't match the filter. For example, both {$.users = 1} and {$.users != 1} will fail to match a log event where users is an array:

{
  "users": [1, 2, 3]
}

Numeric Comparisons

The metric filter syntax supports precise matching on numeric comparisons. The following numeric comparisons are supported: <, >, >=, <=, =, !=

Numeric filters have a syntax of

{ SELECTOR NUMERIC_OPERATOR NUMBER }

The metric filter must be enclosed in curly braces { }, to indicate this is a JSON expression. The metric filter is contains the following parts:

SELECTOR

Specifies what JSON property to check. Property selectors always start with dollar sign ($), which signifies the root of the JSON. Property selectors are alphanumeric strings that also support '-' and '_' characters. Array elements are denoted with [NUMBER] syntax, and must follow a property. Examples are: $.latency, $.numbers[0], $.errorCode, $.processes[4].averageRuntime.

NUMERIC_OPERATOR

Can be one of the following: =, !=, <, >, <=, or >=.

NUMBER

An integer with an optional + or - sign, a decimal with an optional + or - sign, or a number in scientific notation, which is an integer or a decimal with an optional + or - sign, followed by 'e', followed by an integer with an optional + or - sign.

Examples:

{ $.latency >= 500 }
{ $.numbers[0] < 10e3 }
{ $.numbers[0] < 10e-3 }
{ $.processes[4].averageRuntime <= 55.5 }
{ $.errorCode = 400 }
{ $.errorCode != 500 }
{ $.latency > +1000 }

Metric Filter Value Extraction

You can publish numeric values found in JSON events as metric values to CloudWatch. The following procedure shows how to publish a metric with the latency found in the JSON request metricFilter: { $.latency = * } metricValue: $.latency.

To publish a metric with the latency in a JSON request

  1. Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/.

  2. In the navigation pane, choose Logs.

  3. In the contents pane, select a log group, and then choose Create Metric Filter.

  4. On the Define Logs Metric Filter screen, for Filter Pattern, type { $.latency = * }, and then choose Assign Metric.

  5. On the Create Metric Filter and Assign a Metric screen, choose Show advanced metric settings.

  6. For Metric Name, type myMetric.

  7. For Metric Value, enter $.latency, and then choose Create Filter.

The following log event would publish a value of 50 to the metric myMetric following filter creation.

{
"latency": 50,
"requestType": "GET"
}