Show / Hide Table of Contents

Namespace Amazon.CDK.AWS.Logs

Amazon CloudWatch Logs Construct Library

--- cfn-resources: Stable cdk-constructs: Stable

This library supplies constructs for working with CloudWatch Logs.

Log Groups/Streams

The basic unit of CloudWatch is a Log Group. Every log group typically has the same kind of data logged to it, in the same format. If there are multiple applications or services logging into the Log Group, each of them creates a new Log Stream.

Every log operation creates a "log event", which can consist of a simple string or a single-line JSON object. JSON objects have the advantage that they afford more filtering abilities (see below).

The only configurable attribute for log streams is the retention period, which configures after how much time the events in the log stream expire and are deleted.

The default retention period if not supplied is 2 years, but it can be set to one of the values in the RetentionDays enum to configure a different retention period (including infinite retention).

// Configure log group for short retention
LogGroup logGroup = new LogGroup(stack, "LogGroup", new LogGroupProps {
    Retention = RetentionDays.ONE_WEEK
});// Configure log group for infinite retention
LogGroup logGroup = new LogGroup(stack, "LogGroup", new LogGroupProps {
    Retention = Infinity
});

LogRetention

The LogRetention construct is a way to control the retention period of log groups that are created outside of the CDK. The construct is usually used on log groups that are auto created by AWS services, such as AWS lambda.

This is implemented using a CloudFormation custom resource which pre-creates the log group if it doesn't exist, and sets the specified log retention period (never expire, by default).

By default, the log group will be created in the same region as the stack. The logGroupRegion property can be used to configure log groups in other regions. This is typically useful when controlling retention for log groups auto-created by global services that publish their log group to a specific region, such as AWS Chatbot creating a log group in us-east-1.

Resource Policy

CloudWatch Resource Policies allow other AWS services or IAM Principals to put log events into the log groups. A resource policy is automatically created when addToResourcePolicy is called on the LogGroup for the first time:

LogGroup logGroup = new LogGroup(this, "LogGroup");
logGroup.AddToResourcePolicy(new PolicyStatement(new PolicyStatementProps {
    Actions = new [] { "logs:CreateLogStream", "logs:PutLogEvents" },
    Principals = new [] { new ServicePrincipal("es.amazonaws.com") },
    Resources = new [] { logGroup.LogGroupArn }
}));

Or more conveniently, write permissions to the log group can be granted as follows which gives same result as in the above example.

LogGroup logGroup = new LogGroup(this, "LogGroup");
logGroup.GrantWrite(new ServicePrincipal("es.amazonaws.com"));

Be aware that any ARNs or tokenized values passed to the resource policy will be converted into AWS Account IDs. This is because CloudWatch Logs Resource Policies do not accept ARNs as principals, but they do accept Account ID strings. Non-ARN principals, like Service principals or Any princpals, are accepted by CloudWatch.

Encrypting Log Groups

By default, log group data is always encrypted in CloudWatch Logs. You have the option to encrypt log group data using a AWS KMS customer master key (CMK) should you not wish to use the default AWS encryption. Keep in mind that if you decide to encrypt a log group, any service or IAM identity that needs to read the encrypted log streams in the future will require the same CMK to decrypt the data.

Here's a simple example of creating an encrypted Log Group using a KMS CMK.

using Amazon.CDK.AWS.KMS;


new LogGroup(this, "LogGroup", new LogGroupProps {
    EncryptionKey = new Key(this, "Key")
});

See the AWS documentation for more detailed information about encrypting CloudWatch Logs.

Subscriptions and Destinations

Log events matching a particular filter can be sent to either a Lambda function or a Kinesis stream.

If the Kinesis stream lives in a different account, a CrossAccountDestination object needs to be added in the destination account which will act as a proxy for the remote Kinesis stream. This object is automatically created for you if you use the CDK Kinesis library.

Create a SubscriptionFilter, initialize it with an appropriate Pattern (see below) and supply the intended destination:

using Amazon.CDK.AWS.Logs.Destinations;
Function fn;
LogGroup logGroup;


new SubscriptionFilter(this, "Subscription", new SubscriptionFilterProps {
    LogGroup = logGroup,
    Destination = new LambdaDestination(fn),
    FilterPattern = FilterPattern.AllTerms("ERROR", "MainThread")
});

Metric Filters

CloudWatch Logs can extract and emit metrics based on a textual log stream. Depending on your needs, this may be a more convenient way of generating metrics for you application than making calls to CloudWatch Metrics yourself.

A MetricFilter either emits a fixed number every time it sees a log event matching a particular pattern (see below), or extracts a number from the log event and uses that as the metric value.

Example:

new MetricFilter(this, "MetricFilter", new MetricFilterProps {
    LogGroup = logGroup,
    MetricNamespace = "MyApp",
    MetricName = "Latency",
    FilterPattern = FilterPattern.Exists("$.latency"),
    MetricValue = "$.latency"
});

Remember that if you want to use a value from the log event as the metric value, you must mention it in your pattern somewhere.

A very simple MetricFilter can be created by using the logGroup.extractMetric() helper function:

LogGroup logGroup;

logGroup.ExtractMetric("$.jsonField", "Namespace", "MetricName");

Will extract the value of jsonField wherever it occurs in JSON-structed log records in the LogGroup, and emit them to CloudWatch Metrics under the name Namespace/MetricName.

Exposing Metric on a Metric Filter

You can expose a metric on a metric filter by calling the MetricFilter.metric() API. This has a default of statistic = 'avg' if the statistic is not set in the props.

LogGroup logGroup;

MetricFilter mf = new MetricFilter(this, "MetricFilter", new MetricFilterProps {
    LogGroup = logGroup,
    MetricNamespace = "MyApp",
    MetricName = "Latency",
    FilterPattern = FilterPattern.Exists("$.latency"),
    MetricValue = "$.latency"
});

//expose a metric from the metric filter
Metric metric = mf.Metric();

//you can use the metric to create a new alarm
//you can use the metric to create a new alarm
new Alarm(this, "alarm from metric filter", new AlarmProps {
    Metric = metric,
    Threshold = 100,
    EvaluationPeriods = 2
});

Patterns

Patterns describe which log events match a subscription or metric filter. There are three types of patterns:

    All patterns are constructed by using static functions on the FilterPattern class.

    In addition to the patterns above, the following special patterns exist:

      Text Patterns

      Text patterns match if the literal strings appear in the text form of the log line.

        Examples:

        // Search for lines that contain both "ERROR" and "MainThread"
        IFilterPattern pattern1 = FilterPattern.AllTerms("ERROR", "MainThread");
        
        // Search for lines that either contain both "ERROR" and "MainThread", or
        // both "WARN" and "Deadlock".
        IFilterPattern pattern2 = FilterPattern.AnyTermGroup(new [] { "ERROR", "MainThread" }, new [] { "WARN", "Deadlock" });

        JSON Patterns

        JSON patterns apply if the log event is the JSON representation of an object (without any other characters, so it cannot include a prefix such as timestamp or log level). JSON patterns can make comparisons on the values inside the fields.

          Fields in the JSON structure are identified by identifier the complete object as $ and then descending into it, such as $.field or $.list[0].field.

            Example:

            // Search for all events where the component field is equal to
            // "HttpServer" and either error is true or the latency is higher
            // than 1000.
            JsonPattern pattern = FilterPattern.All(FilterPattern.StringValue("$.component", "=", "HttpServer"), FilterPattern.Any(FilterPattern.BooleanValue("$.error", true), FilterPattern.NumberValue("$.latency", ">", 1000)));

            Space-delimited table patterns

            If the log events are rows of a space-delimited table, this pattern can be used to identify the columns in that structure and add conditions on any of them. The canonical example where you would apply this type of pattern is Apache server logs.

            Text that is surrounded by "..." quotes or [...] square brackets will be treated as one column.

              After constructing a SpaceDelimitedTextPattern, you can use the following two members to add restrictions:

                Multiple restrictions can be added on the same column; they must all apply.

                Example:

                // Search for all events where the component is "HttpServer" and the
                // result code is not equal to 200.
                SpaceDelimitedTextPattern pattern = FilterPattern.SpaceDelimited("time", "component", "...", "result_code", "latency").WhereString("component", "=", "HttpServer").WhereNumber("result_code", "!=", 200);

                Logs Insights Query Definition

                Creates a query definition for CloudWatch Logs Insights.

                Example:

                new QueryDefinition(this, "QueryDefinition", new QueryDefinitionProps {
                    QueryDefinitionName = "MyQuery",
                    QueryString = new QueryString(new QueryStringProps {
                        Fields = new [] { "@timestamp", "@message" },
                        Sort = "@timestamp desc",
                        Limit = 20
                    })
                });

                Notes

                Be aware that Log Group ARNs will always have the string :* appended to them, to match the behavior of the CloudFormation AWS::Logs::LogGroup resource.

                Classes

                CfnDestination

                A CloudFormation AWS::Logs::Destination.

                CfnDestinationProps

                Properties for defining a CfnDestination.

                CfnLogGroup

                A CloudFormation AWS::Logs::LogGroup.

                CfnLogGroupProps

                Properties for defining a CfnLogGroup.

                CfnLogStream

                A CloudFormation AWS::Logs::LogStream.

                CfnLogStreamProps

                Properties for defining a CfnLogStream.

                CfnMetricFilter

                A CloudFormation AWS::Logs::MetricFilter.

                CfnMetricFilter.DimensionProperty

                Specifies the CloudWatch metric dimensions to publish with this metric.

                CfnMetricFilter.MetricTransformationProperty

                MetricTransformation is a property of the AWS::Logs::MetricFilter resource that describes how to transform log streams into a CloudWatch metric.

                CfnMetricFilterProps

                Properties for defining a CfnMetricFilter.

                CfnQueryDefinition

                A CloudFormation AWS::Logs::QueryDefinition.

                CfnQueryDefinitionProps

                Properties for defining a CfnQueryDefinition.

                CfnResourcePolicy

                A CloudFormation AWS::Logs::ResourcePolicy.

                CfnResourcePolicyProps

                Properties for defining a CfnResourcePolicy.

                CfnSubscriptionFilter

                A CloudFormation AWS::Logs::SubscriptionFilter.

                CfnSubscriptionFilterProps

                Properties for defining a CfnSubscriptionFilter.

                ColumnRestriction
                CrossAccountDestination

                A new CloudWatch Logs Destination for use in cross-account scenarios.

                CrossAccountDestinationProps

                Properties for a CrossAccountDestination.

                FilterPattern

                A collection of static methods to generate appropriate ILogPatterns.

                JsonPattern

                Base class for patterns that only match JSON log events.

                LogGroup

                Define a CloudWatch Log Group.

                LogGroupProps

                Properties for a LogGroup.

                LogRetention

                Creates a custom resource to control the retention policy of a CloudWatch Logs log group.

                LogRetentionProps

                Construction properties for a LogRetention.

                LogRetentionRetryOptions

                Retry options for all AWS API calls.

                LogStream

                Define a Log Stream in a Log Group.

                LogStreamProps

                Properties for a LogStream.

                LogSubscriptionDestinationConfig

                Properties returned by a Subscription destination.

                MetricFilter

                A filter that extracts information from CloudWatch Logs and emits to CloudWatch Metrics.

                MetricFilterOptions

                Properties for a MetricFilter created from a LogGroup.

                MetricFilterProps

                Properties for a MetricFilter.

                QueryDefinition

                Define a query definition for CloudWatch Logs Insights.

                QueryDefinitionProps

                Properties for a QueryDefinition.

                QueryString

                Define a QueryString.

                QueryStringProps

                Properties for a QueryString.

                ResourcePolicy

                Resource Policy for CloudWatch Log Groups.

                ResourcePolicyProps

                Properties to define Cloudwatch log group resource policy.

                RetentionDays

                How long, in days, the log contents will be retained.

                SpaceDelimitedTextPattern

                Space delimited text pattern.

                StreamOptions

                Properties for a new LogStream created from a LogGroup.

                SubscriptionFilter

                A new Subscription on a CloudWatch log group.

                SubscriptionFilterOptions

                Properties for a new SubscriptionFilter created from a LogGroup.

                SubscriptionFilterProps

                Properties for a SubscriptionFilter.

                Interfaces

                CfnMetricFilter.IDimensionProperty

                Specifies the CloudWatch metric dimensions to publish with this metric.

                CfnMetricFilter.IMetricTransformationProperty

                MetricTransformation is a property of the AWS::Logs::MetricFilter resource that describes how to transform log streams into a CloudWatch metric.

                ICfnDestinationProps

                Properties for defining a CfnDestination.

                ICfnLogGroupProps

                Properties for defining a CfnLogGroup.

                ICfnLogStreamProps

                Properties for defining a CfnLogStream.

                ICfnMetricFilterProps

                Properties for defining a CfnMetricFilter.

                ICfnQueryDefinitionProps

                Properties for defining a CfnQueryDefinition.

                ICfnResourcePolicyProps

                Properties for defining a CfnResourcePolicy.

                ICfnSubscriptionFilterProps

                Properties for defining a CfnSubscriptionFilter.

                IColumnRestriction
                ICrossAccountDestinationProps

                Properties for a CrossAccountDestination.

                IFilterPattern

                Interface for objects that can render themselves to log patterns.

                ILogGroup
                ILogGroupProps

                Properties for a LogGroup.

                ILogRetentionProps

                Construction properties for a LogRetention.

                ILogRetentionRetryOptions

                Retry options for all AWS API calls.

                ILogStream
                ILogStreamProps

                Properties for a LogStream.

                ILogSubscriptionDestination

                Interface for classes that can be the destination of a log Subscription.

                ILogSubscriptionDestinationConfig

                Properties returned by a Subscription destination.

                IMetricFilterOptions

                Properties for a MetricFilter created from a LogGroup.

                IMetricFilterProps

                Properties for a MetricFilter.

                IQueryDefinitionProps

                Properties for a QueryDefinition.

                IQueryStringProps

                Properties for a QueryString.

                IResourcePolicyProps

                Properties to define Cloudwatch log group resource policy.

                IStreamOptions

                Properties for a new LogStream created from a LogGroup.

                ISubscriptionFilterOptions

                Properties for a new SubscriptionFilter created from a LogGroup.

                ISubscriptionFilterProps

                Properties for a SubscriptionFilter.

                Back to top Generated by DocFX