Amazon Kinesis Agent for Microsoft Windows
User Guide

Sink Declarations

Sink declarations specify where and in what form logs, events, and metrics should be sent to various AWS services. The following sections describe configurations for the built-in sink types that are available in Amazon Kinesis Agent for Microsoft Windows. Because Kinesis Agent for Windows is extensible, you can add custom sink types. Each sink type typically requires unique key-value pairs in the configuration declarations that are relevant for that sink type.

All sink declarations can contain the following key-value pairs:

Id

A unique string that identifies a particular sink within the configuration file (required).

SinkType

The name of the sink type for this sink (required). The sink type specifies the destination of the log, event, or metric data that is being streamed by this sink.

AccessKey

Specifies the AWS access key to use when authorizing access to the AWS service that is associated with the sink type. This key-value pair is optional. For more information, see Sink Security Configuration.

SecretKey

Specifies the AWS secret key to use when authorizing access to the AWS service that is associated with the sink type. This key-value pair is optional. For more information, see Sink Security Configuration.

Region

Specifies which AWS Region contains the destination resources for streaming. This key-value pair is optional.

ProfileName

Specifies which AWS profile to use for authentication. This key-value pair is optional, but if specified, it overrides any specified access key and secret key. For more information, see Sink Security Configuration.

RoleARN

Specifies the IAM role to use when accessing the AWS service that is associated with the sink type. This option is useful when Kinesis Agent for Windows is running on an Amazon EC2 instance, but a different role would be more appropriate than the role referenced by the instance profile. For example, a cross-account role can be used to target resources that are not in the same AWS account as the EC2 instance. This key-value pair is optional.

Format

Specifies the kind of serialization that is applied to logs and event data before streaming. Valid values are json and xml. This option is helpful when downstream analytics in the data pipeline require or prefer data in a particular form. This key-value pair is optional, and if not specified, ordinary text from the source is streamed from the sink to the AWS service that is associated with the sink type.

TextDecoration

When no Format is specified, TextDecoration specifies what additional text should be included when streaming log or event records. For more information, see Configuring Sink Decorations. This key-value pair is optional.

ObjectDecoration

When Format is specified, ObjectDecoration specifies what additional data is included in the log or event record before serialization and streaming. For more information, see Configuring Sink Decorations. This key-value pair is optional.

BufferInterval

To minimize API calls to the AWS service that is associated with the sink type, Kinesis Agent for Windows buffers up multiple log, event, or metric records before streaming. This can save money for services that charge per API call. BufferInterval specifies the maximum length of time (in seconds) that records should be buffered before streaming to the AWS service. This key-value pair is optional, and if specified, use a string to represent the value.

BufferSize

To minimize API calls to the AWS service that is associated with the sink type, Kinesis Agent for Windows buffers up multiple log, event, or metric records before streaming. This can save money for services that charge per API call. BufferSize specifies the maximum number of records to buffer before streaming to the AWS service. This key-value pair is optional, and if it is specified, use a string to represent the value.

MaxAttempts

Specifies the maximum number of times Kinesis Agent for Windows tries to stream a set of log, event, and metric records to an AWS service if the streaming consistently fails. This key-value pair is optional. If it is specified, use a string to represent the value. The default value is "3".

For examples of complete configuration files that use various kinds of sinks, see Streaming from the Windows Application Event Log to Various Sinks.

KinesisStream Sink Configuration

The KinesisStream sink type streams log records and events to the Kinesis Data Streams service. Typically, data that is streamed to Kinesis Data Streams is processed by one or more custom applications that execute using various AWS services. Data is streamed to a named stream that is configured using Kinesis Data Streams. For more information, see the Amazon Kinesis Data Streams Developer Guide.

The following is an example Kinesis Data Streams sink declaration:

{ "Id": "TestKinesisStreamSink", "SinkType": "KinesisStream", "StreamName": "MyTestStream", "Region": "us-west-2" }

All KinesisStream sink declarations can provide the following additional key-value pairs:

SinkType

Must be specified, and the value must be the literal string KinesisStream.

StreamName

Specifies the name of the Kinesis data stream that receives the data streamed from the KinesisStream sink type (required). Before streaming the data, configure the stream in the AWS Management Console, the AWS CLI, or through an application using the Kinesis Data Streams API.

RecordsPerSecond

Specifies the maximum number of records streamed to Kinesis Data Streams per second. This key-value pair is optional. If it is specified, use an integer to represent the value. The default value is 1000 records.

BytesPerSecond

Specifies the maximum number of bytes streamed to Kinesis Data Streams per second. This key-value pair is optional. If it is specified, use an integer to represent the value. The default value is 1 MB.

The default BufferInterval for this sink type is 1 second, and the default BufferSize is 500 records.

KinesisFirehose Sink Configuration

The KinesisFirehose sink type streams log records and events to the Kinesis Data Firehose service. Kinesis Data Firehose delivers the streamed data to other services for storage. Typically the stored data is then analyzed in subsequent stages of the data pipeline. Data is streamed to a named delivery stream that is configured using Kinesis Data Firehose. For more information, see the Amazon Kinesis Data Firehose Developer Guide.

The following is an example Kinesis Data Firehose sink declaration:

{ "Id": "TestKinesisFirehoseSink", "SinkType": "KinesisFirehose", "StreamName": "MyTestFirehoseDeliveryStream", "Region": "us-east-1" }

All KinesisFirehose sink declarations can provide the following additional key-value pairs:

SinkType

Must be specified, and the value must be the literal string KinesisFirehose.

StreamName

Specifies the name of the Kinesis Data Firehose delivery stream that receives the data streamed from the KinesisStream sink type (required). Before streaming the data, configure the delivery stream using the AWS Management Console, the AWS CLI, or through an application using the Kinesis Data Firehose API.

RecordsPerSecond

Specifies the maximum number of records that are streamed to Kinesis Data Streams per second. This key-value pair is optional. If it is specified, use an integer to represent the value. The default value is 5000 records.

BytesPerSecond

Specifies the maximum number of bytes that are streamed to Kinesis Data Streams per second. This key-value pair is optional. If it is specified, use an integer to represent the value. The default value is 5 MB.

The default BufferInterval for this sink type is 1 second, and the default BufferSize is 500 records.

CloudWatch Sink Configuration

The CloudWatch sink type streams metrics to the CloudWatch service. You can view the metrics in the AWS Management Console. For more information, see the Amazon CloudWatch User Guide.

The following is an example CloudWatch sink declaration:

{ "Id": "CloudWatchSink", "SinkType": "CloudWatch" }

All CloudWatch sink declarations can provide the following additional key-value pairs:

SinkType

Must be specified, and the value must be the literal string CloudWatch.

Interval

Specifies how frequently (in seconds) Kinesis Agent for Windows reports metrics to the CloudWatch service. This key-value pair is optional. If it is specified, use an integer to represent the value. The default value is 60 seconds. Specify 1 second if you want high-resolution CloudWatch metrics.

Namespace

Specifies the CloudWatch namespace where the metric data is reported. CloudWatch namespaces group a set of metrics together. This key-value pair is optional. The default value is KinesisTap.

Dimensions

Specifies the CloudWatch dimensions that are used to isolate metric sets within a namespace. This can be useful to provide separate sets of metric data for each desktop or server, for example. This key-value pair is optional, and if specified, the value must comply with the following format: "key1=value1;key2=value2...". The default value is "ComputerName={computername};InstanceId={instance_id}". This value supports sink variable substitution. For more information, see Configuring Sink Variable Substitutions.

MetricsFilter

Specifies which metrics are streamed to CloudWatch from the built-in Kinesis Agent for Windows metrics source. For more information about the built-in Kinesis Agent for Windows metrics source, including the details of the syntax of the value of this key-value pair, see Kinesis Agent for Windows Built-In Metrics Source.

CloudWatchLogs Sink Configuration

The CloudWatchLogs sink type streams log records and events to Amazon CloudWatch Logs. You can view logs in the AWS Management Console, or process them via additional stages of a data pipeline. Data is streamed to a named log stream that is configured in CloudWatch Logs. Log streams are organized into named log groups. For more information, see the Amazon CloudWatch Logs User Guide.

The following is an example CloudWatch Logs sink declaration:

{ "Id": "MyCloudWatchLogsSink", "SinkType": "CloudWatchLogs", "BufferInterval": "60", "BufferSize": "100", "Region": "us-west-2", "LogGroup": "MyTestLogGroup", "LogStream": "MyTestStream" }

All CloudWatchLogs sink declarations must provide the following additional key-value pairs:

SinkType

Must be the literal string CloudWatchLogs.

LogGroup

Specifies the name of the CloudWatch Logs log group that contains the log stream that receives the log and event records streamed by the CloudWatchLogs sink type. If the specified log group does not exist, Kinesis Agent for Windows attempts to create it.

LogStream

Specifies the name of the CloudWatch Logs log stream that receives the log and event records stream by the CloudWatchLogs sink type. This value supports sink variable substitution. For more information, see Configuring Sink Variable Substitutions. If the specified log stream does not exist, Kinesis Agent for Windows attempts to create it.

The default BufferInterval for this sink type is 1 second, and the default BufferSize is 500 records. The maximum buffer size is 10,000 records.

Sink Security Configuration

Configuring Authentication

For Kinesis Agent for Windows to stream logs, events, and metrics to AWS services, access must be authenticated. There are several ways to provide authentication for Kinesis Agent for Windows, depending on the situation where Kinesis Agent for Windows is executing and the specific security requirements for a particular organization.

  • If Kinesis Agent for Windows is executing on an Amazon EC2 host, the most secure and simplest way to provide authentication is to create an IAM role with sufficient access to the required operations for the required AWS services, and an EC2 instance profile that references that role. For information about creating instance profiles, see Using Instance Profiles. For information about what policies to attach to the IAM role, see Configuring Authorization.

    After creating the instance profile, you can associate it with any EC2 instances that use Kinesis Agent for Windows. If instances already have an associated instance profile, you can attach the appropriate policies to the role that is associated with that instance profile.

  • If Kinesis Agent for Windows executes on an EC2 host in one account, but the resources that are the target of the sink reside in a different account, you can create an IAM role for cross-account access. For more information, see Tutorial: Delegate Access Across AWS Accounts Using IAM Roles. After creating the cross-account role, specify the Amazon Resource Name (ARN) for the cross-account role as the value of the RoleARN key-value pair in the sink declaration. Kinesis Agent for Windows then attempts to assume the specified cross-account role when accessing AWS resources that are associated with the sink type for that sink.

  • If Kinesis Agent for Windows is executing outside of Amazon EC2 (for example, on-premises), several options exist:

    • If it is acceptable to register the on-premises server or desktop machine as an Amazon EC2 Systems Manager managed-instance, use the following process to configure authentication:

      1. Use the process described in Setting Up AWS Systems Manager in Hybrid Environments to create a service role, create an activation for a managed instance, and install the SSM agent.

      2. Attach the appropriate policies to the service role to enable Kinesis Agent for Windows to access the resources necessary for streaming data from the configured sinks. For information about what policies to attach to the IAM role, see Configuring Authorization.

      This is the recommended approach for non-EC2 instances because credentials are securely managed by SSM and AWS.

    • If it is acceptable to run the AWSKinesisTap service for Kinesis Agent for Windows under a specific user instead of the default system account, use the following process to configure authentication:

      1. Create an IAM user in the AWS account where the AWS services will be used. Capture the access key and secret key of this user during the creation process. You need this information for later steps in this process.

      2. Attach policies to the IAM user that authorize access to the required operations for the required services. For information about what policies to attach to the IAM user, see Configuring Authorization.

      3. Change the AWSKinesisTap service on each desktop or server so that it runs under a specific user rather than the default system account.

      4. Create a profile in the SDK store using the access key and secret key recorded earlier. For more information, see Configuring AWS Credentials.

      5. Update the AWSKinesisTap.exe.config file in the %PROGRAMFILES%\Amazon\AWSKinesisTap directory to specify the name of the profile created in the previous step. For more information, see Configuring AWS Credentials.

      This is the recommended approach for non-EC2 hosts that cannot be managed instances because the credentials are encrypted for the specific host and the specific user.

    • If it is required to run the AWSKinesisTap service for Kinesis Agent for Windows under the default system account, you must use a shared credential file. This is because the system account has no Windows user profile for enabling the SDK store. Shared credential files are not encrypted, so we do not recommend this approach. For information about how to use shared configuration files, see Configuring AWS Credentials in the AWS SDK for .NET. If you use this approach, we recommend that you use NTFS encryption and restricted file access to the shared configuration file. Keys should be rotated by a management platform, and the shared configuration file must be updated when key rotation occurs.

Although it is possible to directly provide access keys and secret keys in the sink declarations, this approach is discouraged because the declarations are not encrypted.

Configuring Authorization

Attach the appropriate policies that follow to the IAM user or role that Kinesis Agent for Windows will use to stream data to AWS services:

Kinesis Data Streams

{ "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "kinesis:PutRecord", "kinesis:PutRecords" ], "Resource": "arn:aws:kinesis:*:*:stream/*" } ] }

To limit authorization to a specific Region, account, or stream name, replace the appropriate asterisks in the ARN with specific values. For more information, see "Amazon Resource Names (ARNs) for Kinesis Data Streams" in Controlling Access to Amazon Kinesis Data Streams Resources Using IAM.

Kinesis Data Firehose

{ "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor1", "Effect": "Allow", "Action": [ "firehose:PutRecord", "firehose:PutRecordBatch" ], "Resource": "arn:aws:firehose:*:*:deliverystream/*" } ] }

To limit authorization to a specific Region, account, or delivery stream name, replace the appropriate asterisks in the ARN with specific values. For more information, see Controlling Access with Amazon Kinesis Data Firehose in the Amazon Kinesis Data Firehose Developer Guide.

CloudWatch

{ "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor2", "Effect": "Allow", "Action": "cloudwatch:PutMetricData", "Resource": "*" } ] }

For more information, see Overview of Managing Access Permissions to Your CloudWatch Resources in the Amazon CloudWatch Logs User Guide.

CloudWatch Logs with an Existing Log Group and Log Stream

{ "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor3", "Effect": "Allow", "Action": [ "logs:DescribeLogGroups", "logs:DescribeLogStreams", "logs:PutLogEvents" ], "Resource": "arn:aws:logs:*:*:log-group:*" }, { "Sid": "VisualEditor4", "Effect": "Allow", "Action": "logs:PutLogEvents", "Resource": "arn:aws:logs:*:*:log-group:*:*:*" } ] }

To restrict access to a specific Region, account, log group, or log stream, replace the appropriate asterisks in the ARNs with appropriate values. For more information, see Overview of Managing Access Permissions to Your CloudWatch Logs Resources in the Amazon CloudWatch Logs User Guide.

CloudWatch Logs with Extra Permissions for Kinesis Agent for Windows to Create Log Groups and Log Streams

{ "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor5", "Effect": "Allow", "Action": [ "logs:CreateLogStream", "logs:DescribeLogGroups", "logs:DescribeLogStreams", "logs:PutLogEvents" ], "Resource": "arn:aws:logs:*:*:log-group:*" }, { "Sid": "VisualEditor6", "Effect": "Allow", "Action": "logs:PutLogEvents", "Resource": "arn:aws:logs:*:*:log-group:*:*:*" }, { "Sid": "VisualEditor7", "Effect": "Allow", "Action": "logs:CreateLogGroup", "Resource": "*" } ] }

To restrict access to a specific Region, account, log group, or log stream, replace the appropriate asterisks in the ARNs with appropriate values. For more information, see Overview of Managing Access Permissions to Your CloudWatch Logs Resources in the Amazon CloudWatch Logs User Guide.

Permissions Required for EC2 Tag Variable Expansion

Using variable expansion with the ec2tag variable prefix requires the ec2:Describe* permission.

{ "Version": "2012-10-17", "Statement": [{ "Sid": "VisualEditor8", "Effect": "Allow", "Action": "ec2:Describe*", "Resource": "*" } ] }

Note

You can combine multiple statements into a single policy as long as the Sid for each statement is unique within that policy. For information about creating policies, see Creating IAM Policies in the IAM User Guide.

Configuring Sink Decorations

Sink declarations can optionally include key-value pairs that specify additional data to stream to various AWS services to enhance the records gathered from the source.

TextDecoration

Use this key-value pair when no Format is specified in the sink declaration. The value is a special format string where variable substitution occurs. For example, suppose that a TextDecoration of "{ComputerName}:::{timestamp:yyyy-MM-dd HH:mm:ss}:::{_record}" is provided for a sink. When a source emits a log record that contains the text The system has resumed from sleep., and that source is connected to the sink via a pipe, then the text MyComputer1:::2017-10-26 06:14:22:::The system has resumed from sleep. is streamed to the AWS service associated with the sink type. The {_record} variable references the original text record delivered by the source.

ObjectDecoration

Use this key-value pair when Format is specified in the sink declaration to add additional data before record serialization. For example, suppose that an ObjectDecoration of "ComputerName={ComputerName};DT={timestamp:yyyy-MM-dd HH:mm:ss}" is provided for a sink that specifies JSON Format. The resulting JSON streamed to the AWS service associated with the sink type includes the following key-value pairs in addition to the original data from the source:

{ ComputerName: "MyComputer2", DT: "2017-10-17 21:09:04" }

For an example of using ObjectDecoration, see Tutorial: Stream JSON Log Files to Amazon S3 Using Amazon Kinesis Agent for Microsoft Windows.

If the source type of the source connected to the sink is DirectorySource, then the sink can use three additional variables:

_FilePath

The full path to the log file.

_FileName

The file name and file name extension of the file.

_Position

An integer that represents where the record is located in the log file.

These variables are useful when you use a source that gathers log records from multiple files connected to a sink that streams all the records to a single stream. Injecting the values of these variables into the streaming records enables downstream analytics in the data pipeline to order the records by file and by location within each file.

Configuring Sink Variable Substitutions

The KinesisStream, KinesisFirehose, and CloudWatchLogs sink declarations require either a LogStream or StreamName key-value pair. The value of these key-values can contain variable references that are automatically resolved by Kinesis Agent for Windows. For CloudWatchLogs, the LogGroup key-value pair is also required and can contain variables references that are automatically resolved by Kinesis Agent for Windows. The variables are specified using the template {prefix:variablename} where prefix: is optional. The supported prefixes are as follows:

  • env — The variable reference is resolved to the value of the environment variable of the same name.

  • ec2 — The variable reference is resolved to the EC2 instance metadata of the same name.

  • ec2tag — The variable reference is resolved to the value of the EC2 instance tag of the same name. The ec2:Describe* permission is required to access instance tags. For more information, see Permissions Required for EC2 Tag Variable Expansion.

If the prefix isn't specified, if there is an environment variable with the same name as variablename, the variable reference is resolved to the value of the environment variable. Otherwise, if variablename is instance_id or hostname, the variable reference is resolved to the value of the EC2 metadata of the same name. Otherwise, the variable reference is not resolved.

The following are examples of valid key-value pairs using variable references:

"LogStream": "LogStream_{instance_id}" "LogStream": "LogStream_{hostname}" "LogStream": "LogStream_{ec2:local-hostname}" "LogStream": "LogStream_{computername}" "LogStream": "LogStream_{env:computername}"

The CloudWatchLogs sink declarations support a special format timestamp variable that allows the timestamp of the original log or event record from the source to alter the name of the log stream. The format is {timestamp:timeformat}. See the following example:

"LogStream": "LogStream_{timestamp:yyyyMMdd}"

If the log or event record was generated on June 5, 2017, the value of the LogStream key-value pair in the previous example would resolve to "LogStream_20170605".

If authorized, the CloudWatchLogs sink type can automatically create new log streams when required based on the generated names. You cannot do this for other sink types because they require additional configuration beyond the name of the stream.

There are special variable substitutions that occur in text and object decoration. For more information, see Configuring Sink Decorations.

Configuring Sink Queuing

The KinesisStream, KinesisFirehose, and CloudWatchLogs sink declarations can optionally enable queuing of records that have failed to stream to the AWS service associated with those sink types due to transient connectivity issues. To enable queuing and automatic streaming retries when connectivity is restored, use the following key-value pairs in the sink declarations:

QueueType

Specifies the kind of queuing mechanism to use. The only supported value is file, which indicates that records should be queued up in a file. This key-value pair is required in order to enable the queuing feature of Kinesis Agent for Windows. If it is not specified, the default behavior is to queue in memory only, and fail to stream when in memory queueing limits are reached.

QueuePath

Specifies the path to the folder that contains the files of queued records. This key-value pair is optional. The default value is %PROGRAMDATA%\KinesisTap\Queue\SinkId where SinkId is the identifier you assigned as the value of the Id for the sink declaration.

QueueMaxBatches

Limits the total amount of space that Kinesis Agent for Windows can consume when queuing records for streaming. The amount of space is limited to the value of this key-value pair multiplied by the maximum number of bytes per batch. The maximum bytes per batch for the KinesisStream, KinesisFirehose, and CloudWatchLogs sink types are 5 MB, 4 MB, and 1 MB respectively. When this limit is reached, any streaming failures are not queued and are reported as non-recoverable failures. This key-value pair is optional. The default value is 10,000 batches.

Configuring a Proxy for Sinks

To configure a proxy for all the Kinesis Agent for Windows sink types that access AWS services, edit the Kinesis Agent for Windows configuration file located at %Program Files%\Amazon\KinesisTap\AWSKinesisTap.exe.config. For instructions, see the proxy section in Configuration Files Reference for AWS SDK for .NET in the AWS SDK for .NET Developer Guide.