Pipe Declarations
Use pipe declarations to connect a source (see Source Declarations) to a sink (see Sink Declarations) in Amazon Kinesis Agent for Microsoft Windows. A pipe declaration is expressed as a JSON object. After Kinesis Agent for Windows starts, the logs, events, or metrics are gathered from the source for a given pipe. They are then streamed to various AWS services using the sink that is associated with that pipe.
The following is an example pipe declaration:
{ "Id": "MyAppLogToCloudWatchLogs", "SourceRef": "MyAppLog", "SinkRef": "MyCloudWatchLogsSink" }
Configuring Pipes
All pipe declarations can contain the following key-value pairs:
Id
-
Specifies the name of the pipe (required). It must be unique within the configuration file.
Type
-
Specifies the type of transformation (if any) that is applied by the pipe as log data is transferred from the source to the sink. The only supported value is
RegexFilterPipe
. This value enables regular expression filtering of the underlying textual representation of the log record. Using filtering can reduce transmission and storage costs by sending only relevant log records downstream to the data pipeline. This key-value pair is optional. The default value is to provide no transformation. FilterPattern
-
Specifies the regular expression for
RegexFilterPipe
pipelines that are used to filter log records gathered by the source before being transferred to the sink. Log records are transferred byRegexFilterPipe
type pipes when the regular expression matches the underlying textual representation of the record. Structured log records that are generated, for example, when using theExtractionPattern
key-value pair in aDirectorySource
declaration, can still be filtered using theRegexFilterPipe
mechanism. This is because this mechanism operates against the original textual representation before parsing. This key-value pair is optional, but it must be provided if the pipe specifies theRegexFilterPipe
type.The following is an example
RegexFilterPipe
pipe declaration:{ "Id": "MyAppLog2ToFirehose", "Type": "RegexFilterPipe", "SourceRef": "MyAppLog2", "SinkRef": "MyFirehose", "FilterPattern": "^(10|11),.*", "IgnoreCase": false, "Negate": false }
SourceRef
-
Specifies the name (the value of the
Id
key-value pair) of the source declaration that defines the source that is collecting log, event, and metric data for the pipe (required). SinkRef
-
Specifies the name (the value of the
Id
key-value pair) of the sink declaration that defines the sink that is receiving the log, event, and metric data for the pipe (required). IgnoreCase
Optional. Accepts values of
true
orfalse
. When set totrue
, the Regex will match records in a case-insensitive manner.Negate
Optional. Accepts values of
true
orfalse
. When set totrue
, the pipe will forward the records that do not match the regular expression.
For an example of a complete configuration file that uses the RegexFilterPipe
pipe type, see Using Pipes.
Configuring Kinesis Agent for Windows Metric Pipes
There is a built-in metric source named _KinesisTapMetricsSource
that produces
metrics about Kinesis Agent for Windows. If there is a CloudWatch
sink declaration with an
Id
of MyCloudWatchSink
, the following example pipeline declaration transfers
Kinesis Agent for Windows-generated metrics to that sink:
{ "Id": "KinesisAgentMetricsToCloudWatch", "SourceRef": "_KinesisTapMetricsSource", "SinkRef": "MyCloudWatchSink" }
For more information about the Kinesis Agent for Windows built-in metrics source, see Kinesis Agent for Windows Built-In Metrics Source.
If the configuration file also streams Windows performance counter metrics, we recommend that you use a separate pipe and sink rather than using the same sink for both Kinesis Agent for Windows metrics and Windows performance counter metrics.