Namespace Amazon.CDK.AWS.Logs
Amazon CloudWatch Logs Construct Library
This library supplies constructs for working with CloudWatch Logs.
Log Groups/Streams
The basic unit of CloudWatch is a Log Group. Every log group typically has the same kind of data logged to it, in the same format. If there are multiple applications or services logging into the Log Group, each of them creates a new Log Stream.
Every log operation creates a "log event", which can consist of a simple string or a single-line JSON object. JSON objects have the advantage that they afford more filtering abilities (see below).
The only configurable attribute for log streams is the retention period, which configures after how much time the events in the log stream expire and are deleted.
The default retention period if not supplied is 2 years, but it can be set to
one of the values in the RetentionDays
enum to configure a different
retention period (including infinite retention).
// Configure log group for short retention
var logGroup = new LogGroup(stack, "LogGroup", new LogGroupProps {
Retention = RetentionDays.ONE_WEEK
});// Configure log group for infinite retention
var logGroup = new LogGroup(stack, "LogGroup", new LogGroupProps {
Retention = Infinity
});
LogRetention
The LogRetention
construct is a way to control the retention period of log groups that are created outside of the CDK. The construct is usually
used on log groups that are auto created by AWS services, such as AWS
lambda.
This is implemented using a CloudFormation custom resource which pre-creates the log group if it doesn't exist, and sets the specified log retention period (never expire, by default).
By default, the log group will be created in the same region as the stack. The logGroupRegion
property can be used to configure
log groups in other regions. This is typically useful when controlling retention for log groups auto-created by global services that
publish their log group to a specific region, such as AWS Chatbot creating a log group in us-east-1
.
By default, the log group created by LogRetention will be retained after the stack is deleted. If the RemovalPolicy is set to DESTROY, then the log group will be deleted when the stack is deleted.
Log Group Class
CloudWatch Logs offers two classes of log groups:
For more details please check: log group class documentation
Resource Policy
CloudWatch Resource Policies allow other AWS services or IAM Principals to put log events into the log groups.
A resource policy is automatically created when addToResourcePolicy
is called on the LogGroup for the first time:
var logGroup = new LogGroup(this, "LogGroup");
logGroup.AddToResourcePolicy(new PolicyStatement(new PolicyStatementProps {
Actions = new [] { "logs:CreateLogStream", "logs:PutLogEvents" },
Principals = new [] { new ServicePrincipal("es.amazonaws.com") },
Resources = new [] { logGroup.LogGroupArn }
}));
Or more conveniently, write permissions to the log group can be granted as follows which gives same result as in the above example.
var logGroup = new LogGroup(this, "LogGroup");
logGroup.GrantWrite(new ServicePrincipal("es.amazonaws.com"));
Similarly, read permissions can be granted to the log group as follows.
var logGroup = new LogGroup(this, "LogGroup");
logGroup.GrantRead(new ServicePrincipal("es.amazonaws.com"));
Be aware that any ARNs or tokenized values passed to the resource policy will be converted into AWS Account IDs. This is because CloudWatch Logs Resource Policies do not accept ARNs as principals, but they do accept Account ID strings. Non-ARN principals, like Service principals or Any principals, are accepted by CloudWatch.
Encrypting Log Groups
By default, log group data is always encrypted in CloudWatch Logs. You have the option to encrypt log group data using a AWS KMS customer master key (CMK) should you not wish to use the default AWS encryption. Keep in mind that if you decide to encrypt a log group, any service or IAM identity that needs to read the encrypted log streams in the future will require the same CMK to decrypt the data.
Here's a simple example of creating an encrypted Log Group using a KMS CMK.
using Amazon.CDK.AWS.KMS;
new LogGroup(this, "LogGroup", new LogGroupProps {
EncryptionKey = new Key(this, "Key")
});
See the AWS documentation for more detailed information about encrypting CloudWatch Logs.
Subscriptions and Destinations
Log events matching a particular filter can be sent to either a Lambda function or a Kinesis stream.
If the Kinesis stream lives in a different account, a CrossAccountDestination
object needs to be added in the destination account which will act as a proxy
for the remote Kinesis stream. This object is automatically created for you
if you use the CDK Kinesis library.
Create a SubscriptionFilter
, initialize it with an appropriate Pattern
(see
below) and supply the intended destination:
using Amazon.CDK.AWS.Logs.Destinations;
Function fn;
LogGroup logGroup;
new SubscriptionFilter(this, "Subscription", new SubscriptionFilterProps {
LogGroup = logGroup,
Destination = new LambdaDestination(fn),
FilterPattern = FilterPattern.AllTerms("ERROR", "MainThread"),
FilterName = "ErrorInMainThread"
});
When you use KinesisDestination
, you can choose the method used to
distribute log data to the destination by setting the distribution
property.
using Amazon.CDK.AWS.Logs.Destinations;
using Amazon.CDK.AWS.Kinesis;
Stream stream;
LogGroup logGroup;
new SubscriptionFilter(this, "Subscription", new SubscriptionFilterProps {
LogGroup = logGroup,
Destination = new KinesisDestination(stream),
FilterPattern = FilterPattern.AllTerms("ERROR", "MainThread"),
FilterName = "ErrorInMainThread",
Distribution = Distribution.RANDOM
});
Metric Filters
CloudWatch Logs can extract and emit metrics based on a textual log stream. Depending on your needs, this may be a more convenient way of generating metrics for you application than making calls to CloudWatch Metrics yourself.
A MetricFilter
either emits a fixed number every time it sees a log event
matching a particular pattern (see below), or extracts a number from the log
event and uses that as the metric value.
Example:
new MetricFilter(this, "MetricFilter", new MetricFilterProps {
LogGroup = logGroup,
MetricNamespace = "MyApp",
MetricName = "Latency",
FilterPattern = FilterPattern.All(FilterPattern.Exists("$.latency"), FilterPattern.RegexValue("$.message", "=", "bind: address already in use")),
MetricValue = "$.latency"
});
Remember that if you want to use a value from the log event as the metric value, you must mention it in your pattern somewhere.
A very simple MetricFilter can be created by using the logGroup.extractMetric()
helper function:
LogGroup logGroup;
logGroup.ExtractMetric("$.jsonField", "Namespace", "MetricName");
Will extract the value of jsonField
wherever it occurs in JSON-structured
log records in the LogGroup, and emit them to CloudWatch Metrics under
the name Namespace/MetricName
.
Exposing Metric on a Metric Filter
You can expose a metric on a metric filter by calling the MetricFilter.metric()
API.
This has a default of statistic = 'avg'
if the statistic is not set in the props
.
LogGroup logGroup;
var mf = new MetricFilter(this, "MetricFilter", new MetricFilterProps {
LogGroup = logGroup,
MetricNamespace = "MyApp",
MetricName = "Latency",
FilterPattern = FilterPattern.Exists("$.latency"),
MetricValue = "$.latency",
Dimensions = new Dictionary<string, string> {
{ "ErrorCode", "$.errorCode" }
},
Unit = Unit.MILLISECONDS
});
//expose a metric from the metric filter
var metric = mf.Metric();
//you can use the metric to create a new alarm
//you can use the metric to create a new alarm
new Alarm(this, "alarm from metric filter", new AlarmProps {
Metric = metric,
Threshold = 100,
EvaluationPeriods = 2
});
Metrics for IncomingLogs and IncomingBytes
Metric methods have been defined for IncomingLogs and IncomingBytes within LogGroups. These metrics allow for the creation of alarms on log ingestion, ensuring that the log ingestion process is functioning correctly.
To define an alarm based on these metrics, you can use the following template:
var logGroup = new LogGroup(this, "MyLogGroup");
var incomingEventsMetric = logGroup.MetricIncomingLogEvents();
new Alarm(this, "HighLogVolumeAlarm", new AlarmProps {
Metric = incomingEventsMetric,
Threshold = 1000,
EvaluationPeriods = 1
});
var logGroup = new LogGroup(this, "MyLogGroup");
var incomingBytesMetric = logGroup.MetricIncomingBytes();
new Alarm(this, "HighDataVolumeAlarm", new AlarmProps {
Metric = incomingBytesMetric,
Threshold = 5000000, // 5 MB
EvaluationPeriods = 1
});
Patterns
Patterns describe which log events match a subscription or metric filter. There are three types of patterns:
All patterns are constructed by using static functions on the FilterPattern
class.
In addition to the patterns above, the following special patterns exist:
Text Patterns
Text patterns match if the literal strings appear in the text form of the log line.
Examples:
// Search for lines that contain both "ERROR" and "MainThread"
var pattern1 = FilterPattern.AllTerms("ERROR", "MainThread");
// Search for lines that either contain both "ERROR" and "MainThread", or
// both "WARN" and "Deadlock".
var pattern2 = FilterPattern.AnyTermGroup(new [] { "ERROR", "MainThread" }, new [] { "WARN", "Deadlock" });
JSON Patterns
JSON patterns apply if the log event is the JSON representation of an object (without any other characters, so it cannot include a prefix such as timestamp or log level). JSON patterns can make comparisons on the values inside the fields.
Fields in the JSON structure are identified by identifier the complete object as $
and then descending into it, such as $.field
or $.list[0].field
.
Example:
// Search for all events where the component field is equal to
// "HttpServer" and either error is true or the latency is higher
// than 1000.
var pattern = FilterPattern.All(FilterPattern.StringValue("$.component", "=", "HttpServer"), FilterPattern.Any(FilterPattern.BooleanValue("$.error", true), FilterPattern.NumberValue("$.latency", ">", 1000)), FilterPattern.RegexValue("$.message", "=", "bind address already in use"));
Space-delimited table patterns
If the log events are rows of a space-delimited table, this pattern can be used to identify the columns in that structure and add conditions on any of them. The canonical example where you would apply this type of pattern is Apache server logs.
Text that is surrounded by "..."
quotes or [...]
square brackets will
be treated as one column.
After constructing a SpaceDelimitedTextPattern
, you can use the following
two members to add restrictions:
Multiple restrictions can be added on the same column; they must all apply.
Example:
// Search for all events where the component is "HttpServer" and the
// result code is not equal to 200.
var pattern = FilterPattern.SpaceDelimited("time", "component", "...", "result_code", "latency").WhereString("component", "=", "HttpServer").WhereNumber("result_code", "!=", 200);
Logs Insights Query Definition
Creates a query definition for CloudWatch Logs Insights.
Example:
new QueryDefinition(this, "QueryDefinition", new QueryDefinitionProps {
QueryDefinitionName = "MyQuery",
QueryString = new QueryString(new QueryStringProps {
Fields = new [] { "@timestamp", "@message" },
ParseStatements = new [] { "@message \"[*] *\" as loggingType, loggingMessage", "@message \"<*>: *\" as differentLoggingType, differentLoggingMessage" },
FilterStatements = new [] { "loggingType = \"ERROR\"", "loggingMessage = \"A very strange error occurred!\"" },
Sort = "@timestamp desc",
Limit = 20
})
});
Data Protection Policy
Creates a data protection policy and assigns it to the log group. A data protection policy can help safeguard sensitive data that's ingested by the log group by auditing and masking the sensitive log data. When a user who does not have permission to view masked data views a log event that includes masked data, the sensitive data is replaced by asterisks.
For more information, see Protect sensitive log data with masking.
For a list of types of managed identifiers that can be audited and masked, see Types of data that you can protect.
If a new identifier is supported but not yet in the DataIdentifiers
enum, the name of the identifier can be supplied as name
in the constructor instead.
To add a custom data identifier, supply a custom name
and regex
to the CustomDataIdentifiers
constructor.
For more information on custom data identifiers, see Custom data identifiers.
Each policy may consist of a log group, S3 bucket, and/or Firehose delivery stream audit destination.
Example:
using Amazon.CDK.AWS.KinesisFirehose.Alpha;
using Amazon.CDK.AWS.KinesisFirehose.Destinations.Alpha;
var logGroupDestination = new LogGroup(this, "LogGroupLambdaAudit", new LogGroupProps {
LogGroupName = "auditDestinationForCDK"
});
var bucket = new Bucket(this, "audit-bucket");
var s3Destination = new S3Bucket(bucket);
var deliveryStream = new DeliveryStream(this, "Delivery Stream", new DeliveryStreamProps {
Destination = s3Destination
});
var dataProtectionPolicy = new DataProtectionPolicy(new DataProtectionPolicyProps {
Name = "data protection policy",
Description = "policy description",
Identifiers = new [] { DataIdentifier.DRIVERSLICENSE_US, // managed data identifier
new DataIdentifier("EmailAddress"), // forward compatibility for new managed data identifiers
new CustomDataIdentifier("EmployeeId", "EmployeeId-\\d{9}") }, // custom data identifier
LogGroupAuditDestination = logGroupDestination,
S3BucketAuditDestination = bucket,
DeliveryStreamNameAuditDestination = deliveryStream.DeliveryStreamName
});
new LogGroup(this, "LogGroupLambda", new LogGroupProps {
LogGroupName = "cdkIntegLogGroup",
DataProtectionPolicy = dataProtectionPolicy
});
Notes
Be aware that Log Group ARNs will always have the string :*
appended to
them, to match the behavior of the CloudFormation AWS::Logs::LogGroup
resource.
Classes
Cfn |
Creates or updates an account-level data protection policy or subscription filter policy that applies to all log groups or a subset of log groups in the account. |
Cfn |
Properties for defining a |
Cfn |
This structure contains information about one delivery in your account. |
Cfn |
This structure contains information about one delivery destination in your account. |
Cfn |
Properties for defining a |
Cfn |
Properties for defining a |
Cfn |
Creates or updates one delivery source in your account. |
Cfn |
Properties for defining a |
Cfn |
The AWS::Logs::Destination resource specifies a CloudWatch Logs destination. |
Cfn |
Properties for defining a |
Cfn |
Creates an integration between CloudWatch Logs and another service in this account. |
Cfn |
This structure contains configuration details about an integration between CloudWatch Logs and OpenSearch Service. |
Cfn |
This structure contains configuration details about an integration between CloudWatch Logs and another entity. |
Cfn |
Properties for defining a |
Cfn |
Creates or updates an anomaly detector that regularly scans one or more log groups and look for patterns and anomalies in the logs. |
Cfn |
Properties for defining a |
Cfn |
The |
Cfn |
Properties for defining a |
Cfn |
The |
Cfn |
Properties for defining a |
Cfn |
The |
Cfn |
Specifies the CloudWatch metric dimensions to publish with this metric. |
Cfn |
|
Cfn |
Properties for defining a |
Cfn |
Creates a query definition for CloudWatch Logs Insights. |
Cfn |
Properties for defining a |
Cfn |
Creates or updates a resource policy that allows other AWS services to put log events to this account. |
Cfn |
Properties for defining a |
Cfn |
The |
Cfn |
Properties for defining a |
Cfn |
Creates or updates a log transformer for a single log group. |
Cfn |
This object defines one key that will be added with the addKeys processor. |
Cfn |
This processor adds new key-value pairs to the log event. |
Cfn |
This object defines one value to be copied with the copyValue processor. |
Cfn |
This processor copies values within a log event. |
Cfn |
The |
Cfn |
This processor converts a datetime string into a format that you specify. |
Cfn |
This processor deletes entries from a log event. These entries are key-value pairs. |
Cfn |
This processor uses pattern matching to parse and structure unstructured data. |
Cfn |
This processor takes a list of objects that contain key fields, and converts them into a map of target keys. |
Cfn |
This processor converts a string to lowercase. |
Cfn |
This object defines one key that will be moved with the moveKey processor. |
Cfn |
This processor moves a key from one field to another. The original key is deleted. |
Cfn |
This processor parses CloudFront vended logs, extract fields, and convert them into JSON format. |
Cfn |
This processor parses log events that are in JSON format. |
Cfn |
This processor parses a specified field in the original log event into key-value pairs. |
Cfn |
Use this processor to parse RDS for PostgreSQL vended logs, extract fields, and and convert them into a JSON format. |
Cfn |
Use this processor to parse Route 53 vended logs, extract fields, and and convert them into a JSON format. |
Cfn |
Use this processor to parse Amazon VPC vended logs, extract fields, and and convert them into a JSON format. |
Cfn |
Use this processor to parse AWS WAF vended logs, extract fields, and and convert them into a JSON format. |
Cfn |
This structure contains the information about one processor in a log transformer. |
Cfn |
This object defines one key that will be renamed with the renameKey processor. |
Cfn |
Use this processor to rename keys in a log event. |
Cfn |
This object defines one log field that will be split with the splitString processor. |
Cfn |
Use this processor to split a field into an array of strings using a delimiting character. |
Cfn |
This object defines one log field key that will be replaced using the substituteString processor. |
Cfn |
This processor matches a key’s value against a regular expression and replaces all matches with a replacement string. |
Cfn |
Use this processor to remove leading and trailing whitespace. |
Cfn |
This object defines one value type that will be converted using the typeConverter processor. |
Cfn |
Use this processor to convert a value type associated with the specified key to the specified type. |
Cfn |
This processor converts a string field to uppercase. |
Cfn |
Properties for defining a |
Column |
|
Cross |
A new CloudWatch Logs Destination for use in cross-account scenarios. |
Cross |
Properties for a CrossAccountDestination. |
Custom |
A custom data identifier. |
Data |
A data protection identifier. |
Data |
Creates a data protection policy for CloudWatch Logs log groups. |
Data |
Properties for creating a data protection policy. |
Distribution | The method used to distribute log data to the destination. |
Filter |
A collection of static methods to generate appropriate ILogPatterns. |
Json |
Base class for patterns that only match JSON log events. |
Log |
Define a CloudWatch Log Group. |
Log |
Class of Log Group. |
Log |
Properties for a LogGroup. |
Log |
Creates a custom resource to control the retention policy of a CloudWatch Logs log group. |
Log |
Construction properties for a LogRetention. |
Log |
Retry options for all AWS API calls. |
Log |
Define a Log Stream in a Log Group. |
Log |
Properties for a LogStream. |
Log |
Properties returned by a Subscription destination. |
Metric |
A filter that extracts information from CloudWatch Logs and emits to CloudWatch Metrics. |
Metric |
Properties for a MetricFilter created from a LogGroup. |
Metric |
Properties for a MetricFilter. |
Query |
Define a query definition for CloudWatch Logs Insights. |
Query |
Properties for a QueryDefinition. |
Query |
Define a QueryString. |
Query |
Properties for a QueryString. |
Resource |
Resource Policy for CloudWatch Log Groups. |
Resource |
Properties to define Cloudwatch log group resource policy. |
Retention |
How long, in days, the log contents will be retained. |
Space |
Space delimited text pattern. |
Stream |
Properties for a new LogStream created from a LogGroup. |
Subscription |
A new Subscription on a CloudWatch log group. |
Subscription |
Properties for a new SubscriptionFilter created from a LogGroup. |
Subscription |
Properties for a SubscriptionFilter. |
Interfaces
Cfn |
This structure contains configuration details about an integration between CloudWatch Logs and OpenSearch Service. |
Cfn |
This structure contains configuration details about an integration between CloudWatch Logs and another entity. |
Cfn |
Specifies the CloudWatch metric dimensions to publish with this metric. |
Cfn |
|
Cfn |
This object defines one key that will be added with the addKeys processor. |
Cfn |
This processor adds new key-value pairs to the log event. |
Cfn |
This object defines one value to be copied with the copyValue processor. |
Cfn |
This processor copies values within a log event. |
Cfn |
The |
Cfn |
This processor converts a datetime string into a format that you specify. |
Cfn |
This processor deletes entries from a log event. These entries are key-value pairs. |
Cfn |
This processor uses pattern matching to parse and structure unstructured data. |
Cfn |
This processor takes a list of objects that contain key fields, and converts them into a map of target keys. |
Cfn |
This processor converts a string to lowercase. |
Cfn |
This object defines one key that will be moved with the moveKey processor. |
Cfn |
This processor moves a key from one field to another. The original key is deleted. |
Cfn |
This processor parses CloudFront vended logs, extract fields, and convert them into JSON format. |
Cfn |
This processor parses log events that are in JSON format. |
Cfn |
This processor parses a specified field in the original log event into key-value pairs. |
Cfn |
Use this processor to parse RDS for PostgreSQL vended logs, extract fields, and and convert them into a JSON format. |
Cfn |
Use this processor to parse Route 53 vended logs, extract fields, and and convert them into a JSON format. |
Cfn |
Use this processor to parse Amazon VPC vended logs, extract fields, and and convert them into a JSON format. |
Cfn |
Use this processor to parse AWS WAF vended logs, extract fields, and and convert them into a JSON format. |
Cfn |
This structure contains the information about one processor in a log transformer. |
Cfn |
This object defines one key that will be renamed with the renameKey processor. |
Cfn |
Use this processor to rename keys in a log event. |
Cfn |
This object defines one log field that will be split with the splitString processor. |
Cfn |
Use this processor to split a field into an array of strings using a delimiting character. |
Cfn |
This object defines one log field key that will be replaced using the substituteString processor. |
Cfn |
This processor matches a key’s value against a regular expression and replaces all matches with a replacement string. |
Cfn |
Use this processor to remove leading and trailing whitespace. |
Cfn |
This object defines one value type that will be converted using the typeConverter processor. |
Cfn |
Use this processor to convert a value type associated with the specified key to the specified type. |
Cfn |
This processor converts a string field to uppercase. |
ICfn |
Properties for defining a |
ICfn |
Properties for defining a |
ICfn |
Properties for defining a |
ICfn |
Properties for defining a |
ICfn |
Properties for defining a |
ICfn |
Properties for defining a |
ICfn |
Properties for defining a |
ICfn |
Properties for defining a |
ICfn |
Properties for defining a |
ICfn |
Properties for defining a |
ICfn |
Properties for defining a |
ICfn |
Properties for defining a |
ICfn |
Properties for defining a |
ICfn |
Properties for defining a |
IColumn |
|
ICross |
Properties for a CrossAccountDestination. |
IData |
Properties for creating a data protection policy. |
IFilter |
Interface for objects that can render themselves to log patterns. |
ILog |
|
ILog |
Properties for a LogGroup. |
ILog |
Construction properties for a LogRetention. |
ILog |
Retry options for all AWS API calls. |
ILog |
|
ILog |
Properties for a LogStream. |
ILog |
Interface for classes that can be the destination of a log Subscription. |
ILog |
Properties returned by a Subscription destination. |
IMetric |
Properties for a MetricFilter created from a LogGroup. |
IMetric |
Properties for a MetricFilter. |
IQuery |
Properties for a QueryDefinition. |
IQuery |
Properties for a QueryString. |
IResource |
Properties to define Cloudwatch log group resource policy. |
IStream |
Properties for a new LogStream created from a LogGroup. |
ISubscription |
Properties for a new SubscriptionFilter created from a LogGroup. |
ISubscription |
Properties for a SubscriptionFilter. |