Pipe Declarations

Use pipe declarations to connect a source (see Source Declarations) to a sink (see Sink Declarations) in Amazon Kinesis Agent for Microsoft Windows. A pipe declaration is expressed as a JSON object. After Kinesis Agent for Windows starts, the logs, events, or metrics are gathered from the source for a given pipe. They are then streamed to various AWS services using the sink that is associated with that pipe.

The following is an example pipe declaration:

{
   "Id": "MyAppLogToCloudWatchLogs", 
   "SourceRef": "MyAppLog", 
   "SinkRef": "MyCloudWatchLogsSink" 
}

Configuring Pipes

All pipe declarations can contain the following key-value pairs:

Id

Specifies the name of the pipe (required). It must be unique within the configuration file.

Type

Specifies the type of transformation (if any) that is applied by the pipe as log data is transferred from the source to the sink. The only supported value is RegexFilterPipe. This value enables regular expression filtering of the underlying textual representation of the log record. Using filtering can reduce transmission and storage costs by sending only relevant log records downstream to the data pipeline. This key-value pair is optional. The default value is to provide no transformation.

FilterPattern

Specifies the regular expression for RegexFilterPipe pipelines that are used to filter log records gathered by the source before being transferred to the sink. Log records are transferred by RegexFilterPipe type pipes when the regular expression matches the underlying textual representation of the record. Structured log records that are generated, for example, when using the ExtractionPattern key-value pair in a DirectorySource declaration, can still be filtered using the RegexFilterPipe mechanism. This is because this mechanism operates against the original textual representation before parsing. This key-value pair is optional, but it must be provided if the pipe specifies the RegexFilterPipe type.

The following is an example RegexFilterPipe pipe declaration:

{
	"Id": "MyAppLog2ToFirehose",
	"Type": "RegexFilterPipe",
	"SourceRef": "MyAppLog2",
	"SinkRef": "MyFirehose",
	"FilterPattern": "^(10|11),.*",
	"IgnoreCase": false,
	"Negate": false
}

SourceRef

Specifies the name (the value of the Id key-value pair) of the source declaration that defines the source that is collecting log, event, and metric data for the pipe (required).

SinkRef

Specifies the name (the value of the Id key-value pair) of the sink declaration that defines the sink that is receiving the log, event, and metric data for the pipe (required).

IgnoreCase

Optional. Accepts values of true or false. When set to true, the Regex will match records in a case-insensitive manner.

Negate

Optional. Accepts values of true or false. When set to true, the pipe will forward the records that do not match the regular expression.

For an example of a complete configuration file that uses the RegexFilterPipe pipe type, see Using Pipes.

Configuring Kinesis Agent for Windows Metric Pipes

There is a built-in metric source named _KinesisTapMetricsSource that produces metrics about Kinesis Agent for Windows. If there is a CloudWatch sink declaration with an Id of MyCloudWatchSink, the following example pipeline declaration transfers Kinesis Agent for Windows-generated metrics to that sink:

{
   "Id": "KinesisAgentMetricsToCloudWatch",
   "SourceRef": "_KinesisTapMetricsSource",
   "SinkRef": "MyCloudWatchSink"
}

For more information about the Kinesis Agent for Windows built-in metrics source, see Kinesis Agent for Windows Built-In Metrics Source.

If the configuration file also streams Windows performance counter metrics, we recommend that you use a separate pipe and sink rather than using the same sink for both Kinesis Agent for Windows metrics and Windows performance counter metrics.