Kinesis Agent for Windows Configuration Examples - Amazon Kinesis Agent for Microsoft Windows
Streaming from Various Sources to Kinesis Data StreamsStreaming from the Windows Application Event Log to SinksUsing PipesUsing Multiple Sources and Pipes

Kinesis Agent for Windows Configuration Examples

The appsettings.json configuration file is a JSON document that controls how Amazon Kinesis Agent for Microsoft Windows collects logs, events, and metrics. It also controls how Kinesis Agent for Windows transforms that data and streams it to various AWS services. For details about the source, sink, and pipe declarations in the configuration file, see Source Declarations, Sink Declarations, and Pipe Declarations.

The following sections contain examples of configuration files for several different kinds of scenarios.

Streaming from Various Sources to Kinesis Data Streams

The following example appsettings.json configuration files demonstrate streaming logs and events from various sources to Kinesis Data Streams and from Windows performance counters to Amazon CloudWatch metrics.

DirectorySource, SysLog Record Parser

The following file streams syslog format log records from all files with a .log file extension in the C:\LogSource\ directory to the SyslogKinesisDataStream Kinesis Data Streams stream in the us-east-1 Region. A bookmark is established to ensure that all data from the log files is sent even if the agent is shut down and restarted later. A custom application can read and process the records from the SyslogKinesisDataStream stream.

{
  "Sources": [
    {
      "Id": "SyslogDirectorySource",
      "SourceType": "DirectorySource",
      "Directory": "C:\\LogSource\\",
      "FileNameFilter": "*.log",
      "RecordParser": "SysLog",
      "TimeZoneKind": "UTC",
      "InitialPosition": "Bookmark"
    }
  ],
  "Sinks": [
    {
      "Id": "KinesisStreamSink",
      "SinkType": "KinesisStream",
      "StreamName": "SyslogKinesisDataStream",
      "Region": "us-east-1"
    }
  ],
  "Pipes": [
    {
      "Id": "SyslogDS2KSSink",
      "SourceRef": "SyslogDirectorySource",
      "SinkRef": "KinesisStreamSink"
    }
  ]
}

DirectorySource, SingleLineJson Record Parser

The following file streams JSON-formatted log records from all files with a .log file extension in the C:\LogSource\ directory to the JsonKinesisDataStream Kinesis Data Streams stream in the us-east-1 Region. Before streaming, key-value pairs for the ComputerName and DT keys are added to each JSON object, with values for the computer name and the date and time the record is processed. A custom application can read and process the records from the JsonKinesisDataStream stream. 

{
  "Sources": [
    {
      "Id": "JsonLogSource",
      "SourceType": "DirectorySource",
      "RecordParser": "SingleLineJson",
      "Directory": "C:\\LogSource\\",
      "FileNameFilter": "*.log",
      "InitialPosition": 0
    }
  ],
  "Sinks": [
    {
      "Id": "KinesisStreamSink",
      "SinkType": "KinesisStream",
      "StreamName": "JsonKinesisDataStream",
      "Region": "us-east-1",
      "Format": "json",
      "ObjectDecoration": "ComputerName={ComputerName};DT={timestamp:yyyy-MM-dd HH:mm:ss}"
    }
  ],
  "Pipes": [
    {
      "Id": "JsonLogSourceToKinesisStreamSink",
      "SourceRef": "JsonLogSource",
      "SinkRef": "KinesisStreamSink"
    }
  ]
}

ExchangeLogSource

The following file streams log records generated by Microsoft Exchange and stored in files with the .log extension in the C:\temp\ExchangeLog\ directory to the ExchangeKinesisDataStream Kinesis data stream in the us-east-1 Region in JSON format. Although the Exchange logs are not in JSON format, Kinesis Agent for Windows can parse the logs and transform them to JSON. Before streaming, key-value pairs for the ComputerName and DT keys are added to each JSON object containing values for the computer name and the date and time the record is processed. A custom application can read and process the records from the ExchangeKinesisDataStream stream. 

{
  "Sources": [
    {
       "Id": "ExchangeSource",
       "SourceType": "ExchangeLogSource",
       "Directory": "C:\\temp\\ExchangeLog\",
       "FileNameFilter": "*.log"
    }
  ],
  "Sinks": [
    {
      "Id": "KinesisStreamSink",
      "SinkType": "KinesisStream",
      "StreamName": "ExchangeKinesisDataStream",
      "Region": "us-east-1",
      "Format": "json",
      "ObjectDecoration": "ComputerName={ComputerName};DT={timestamp:yyyy-MM-dd HH:mm:ss}"
    }
  ],
  "Pipes": [
    {
      "Id": "ExchangeSourceToKinesisStreamSink",
      "SourceRef": "ExchangeSource",
      "SinkRef": "KinesisStreamSink"
    }
  ]
}

W3SVCLogSource

The following file streams Internet Information Services (IIS) for Windows log records stored in the standard location for those files to the IISKinesisDataStream Kinesis Data Streams stream in the us-east-1 Region. A custom application can read and process the records from the IISKinesisDataStream stream. IIS is a web server for Windows. 

{
  "Sources": [
    {
       "Id": "IISLogSource",
       "SourceType": "W3SVCLogSource",
       "Directory": "C:\\inetpub\\logs\\LogFiles\\W3SVC1",
       "FileNameFilter": "*.log"
    }
  ],
  "Sinks": [
    {
      "Id": "KinesisStreamSink",
      "SinkType": "KinesisStream",
      "StreamName": "IISKinesisDataStream",
      "Region": "us-east-1"
    }
  ],
  "Pipes": [
    {
      "Id": "IISLogSourceToKinesisStreamSink",
      "SourceRef": "IISLogSource",
      "SinkRef": "KinesisStreamSink"
    }
  ]
}

WindowsEventLogSource with Query

The following file streams log events from the Windows system event log that have a level of Critical or Error (less than or equal to 2) to the SystemKinesisDataStream Kinesis data stream in the us-east-1 Region in JSON format. A custom application can read and process the records from the SystemKinesisDataStream stream. 

{
  "Sources": [
    {
         "Id": "SystemLogSource",
         "SourceType": "WindowsEventLogSource",
         "LogName": "System",
         "Query": "*[System/Level<=2]"
    }
  ],
  "Sinks": [
    {
      "Id": "KinesisStreamSink",
      "SinkType": "KinesisStream",
      "StreamName": "SystemKinesisDataStream",
      "Region": "us-east-1",
      "Format": "json"
    }
  ],
  "Pipes": [
    {
      "Id": "SLSourceToKSSink",
      "SourceRef": "SystemLogSource",
      "SinkRef": "KinesisStreamSink"
    }
  ]
}

WindowsETWEventSource

The following file streams Microsoft Common Language Runtime (CLR) exception and security events to the ClrKinesisDataStream Kinesis data stream in the us-east-1 Region in JSON format. A custom application can read and process the records from the ClrKinesisDataStream stream. 

{
  "Sources": [
    {
       "Id": "ClrETWEventSource",
       "SourceType": "WindowsETWEventSource",
       "ProviderName": "Microsoft-Windows-DotNETRuntime",
       "TraceLevel": "Verbose",
       "MatchAnyKeyword": "0x00008000, 0x00000400"
    }
  ],
  "Sinks": [
    {
      "Id": "KinesisStreamSink",
      "SinkType": "KinesisStream",
      "StreamName": "ClrKinesisDataStream",
      "Region": "us-east-1",
      "Format": "json"
    }
  ],
  "Pipes": [
    {
      "Id": "ETWSourceToKSSink",
      "SourceRef": "ClrETWEventSource",
      "SinkRef": "KinesisStreamSink"
    }
  ]
}

WindowsPerformanceCounterSource

The following file streams performance counters for total files open, total login attempts since reboot, number of disk reads per second, and percentage of free disk space to CloudWatch metrics in the us-east-1 Region. You can graph these metrics in CloudWatch, build dashboards from the graphs, and set alarms that send notifications when thresholds are exceeded. 

{
  "Sources": [
    {
      "Id": "PerformanceCounter",
      "SourceType": "WindowsPerformanceCounterSource",
      "Categories": [
        {
          "Category": "Server",
          "Counters": [
            "Files Open",
            "Logon Total"
          ]
        },
        {
          "Category": "LogicalDisk",
          "Instances": "*",
          "Counters": [
            "% Free Space",
            {
              "Counter": "Disk Reads/sec",
              "Unit": "Count/Second"
            }
          ]
        }
      ],
    }
  ],
  "Sinks": [
    {
      "Namespace": "MyServiceMetrics",
      "Region": "us-east-1",
      "Id": "CloudWatchSink",
      "SinkType": "CloudWatch"
    }
  ],
  "Pipes": [
    {
      "Id": "PerformanceCounterToCloudWatch",
      "SourceRef": "PerformanceCounter",
      "SinkRef": "CloudWatchSink"
    }
  ]
}

Streaming from the Windows Application Event Log to Sinks

The following example appsettings.json configuration files demonstrate streaming Windows application event logs to various sinks in Amazon Kinesis Agent for Microsoft Windows. For examples of using the KinesisStream and CloudWatch sink types, see Streaming from Various Sources to Kinesis Data Streams.

KinesisFirehose

The following file streams Critical or Error Windows application log events to the WindowsLogFirehoseDeliveryStream Firehose delivery stream in the us-east-1 Region. If connectivity to Firehose is interrupted, events are first queued in memory. Then if necessary, they are queued to a file on disk until connectivity is restored. Then events are unqueued and sent followed by any new events.

You can configure Firehose to store the streamed data to several different kinds of storage and analysis services based on data pipeline requirements. 

{
  "Sources": [
    {
         "Id": "ApplicationLogSource",
         "SourceType": "WindowsEventLogSource",
         "LogName": "Application",
         "Query": "*[System/Level<=2]"
    }
  ],
  "Sinks": [
    {
       "Id": "WindowsLogKinesisFirehoseSink",
       "SinkType": "KinesisFirehose",
       "StreamName": "WindowsLogFirehoseDeliveryStream",
       "Region": "us-east-1",
       "QueueType": "file"
    }  
    ],
  "Pipes": [
    {
      "Id": "ALSource2ALKFSink",
      "SourceRef": "ApplicationLogSource",
      "SinkRef": "WindowsLogKinesisFirehoseSink"
    }
  ]
}

CloudWatchLogs

The following file streams Critical or Error Windows application log events to CloudWatch Logs log streams in the MyServiceApplicationLog-Group log group. The name of each stream begins with Stream-. It ends with the four-digit year, two-digit month, and two-digit day that the stream was created, all concatenated (for example, Stream-20180501 is the stream created on May 1, 2018). 

{
  "Sources": [
    {
         "Id": "ApplicationLogSource",
         "SourceType": "WindowsEventLogSource",
         "LogName": "Application",
         "Query": "*[System/Level<=2]"
    }
  ],
  "Sinks": [
    {
      "Id": "CloudWatchLogsSink",
      "SinkType": "CloudWatchLogs",
      "LogGroup": "MyServiceApplicationLog-Group",
      "LogStream": "Stream-{timestamp:yyyyMMdd}",
      "Region": "us-east-1",
      "Format": "json"
    }
  ],
  "Pipes": [
    {
      "Id": "ALSource2CWLSink",
      "SourceRef": "ApplicationLogSource",
      "SinkRef": "CloudWatchLogsSink"
    }
  ]
}

Using Pipes

The following example appsettings.json configuration file demonstrates using pipe-related features.

This example streams log entries from the c:\LogSource\ to the ApplicationLogFirehoseDeliveryStream Firehose delivery stream. It includes only lines that match the regular expression specified by the FilterPattern key-value pair. Specifically, only lines in the log file that start with 10 or 11 are streamed to Firehose. 

{
  "Sources": [
    {
      "Id": "ApplicationLogSource",
      "SourceType": "DirectorySource",
      "Directory": "C:\\LogSource\\",
      "FileNameFilter": "*.log",
      "RecordParser": "SingleLine"
    }
  ],
  "Sinks": [
    {
       "Id": "ApplicationLogKinesisFirehoseSink",
       "SinkType": "KinesisFirehose",
       "StreamName": "ApplicationLogFirehoseDeliveryStream",
       "Region": "us-east-1"
    }  
    ],
  "Pipes": [
    {
      "Id": "ALSourceToALKFSink",
      "Type": "RegexFilterPipe",
      "SourceRef": "ApplicationLogSource",
      "SinkRef": "ApplicationLogKinesisFirehoseSink",
      "FilterPattern": "^(10|11),.*"
    }
  ]
}

Using Multiple Sources and Pipes

The following example appsettings.json configuration file demonstrates using multiple sources and pipes.

This example streams the application, security, and system Windows Event Logs to the EventLogStream Firehose delivery stream using three sources, three pipes, and a single sink.

{
    "Sources": [
		{
		  "Id": "ApplicationLog",
		  "SourceType": "WindowsEventLogSource",
		  "LogName": "Application"
		},
		{
		  "Id": "SecurityLog",
		  "SourceType": "WindowsEventLogSource",
		  "LogName": "Security"
		},
		{
		  "Id": "SystemLog",
		  "SourceType": "WindowsEventLogSource",
		  "LogName": "System"
		}
    ],
    "Sinks": [
		{
		  "Id": "EventLogSink",
		  "SinkType": "KinesisFirehose",
		  "StreamName": "EventLogStream",
		  "Format": "json"
		},
    ],
    "Pipes": [
		{
		  "Id": "ApplicationLogToFirehose",
		  "SourceRef": "ApplicationLog",
		  "SinkRef": "EventLogSink"
		},
		{
		  "Id": "SecurityLogToFirehose",
		  "SourceRef": "SecurityLog",
		  "SinkRef": "EventLogSink"
		},
		{
		  "Id": "SystemLogToFirehose",
		  "SourceRef": "SystemLog",
		  "SinkRef": "EventLogSink"
		}
    ]
}