Tutorial: Stream JSON Log Files to Amazon S3 Using Kinesis Agent for Windows

This tutorial presents detailed steps for setting up a data pipeline using Amazon Kinesis Agent for Microsoft Windows (Kinesis Agent for Windows).

The tutorial includes the following steps:

• Using Kinesis Agent for Windows to stream JSON-formatted log files to Amazon Simple Storage Service (Amazon S3) via Amazon Data Firehose. For information about Kinesis Agent for Windows, see What Is Amazon Kinesis Agent for Microsoft Windows?.

• Enhancing the log data before streaming using object decoration. For more information, see Configuring Sink Decorations.

• Using Amazon Athena to search for particular kinds of log records.

Prerequisites

If you don't already have an AWS account, follow the instructions in Setting Up an AWS Account to get one.

Topics

• Step 1: Configure AWS Services
• Step 2: Install, Configure, and Run Kinesis Agent for Windows
• Step 3: Query the Log Data in Amazon S3
• Next Steps

Next Steps

Use the AWS Management Console to clean up the resources created during the tutorial:

1. Terminate the EC2 instance (see step 3 in Getting Started with Amazon EC2 Windows Instances).

   Important

   If you launched an instance that was not within the AWS Free Tier, you are charged for the instance until you terminate it.

2. Delete the Firehose delivery stream.

   1. Open the Firehose console at https://console.aws.amazon.com/firehose/.

   2. Choose the delivery stream that you created.

   3. Choose Delete.

   4. Choose Delete delivery stream.

3. Delete the S3 bucket. For instructions, see How Do I Delete an S3 Bucket? in the Amazon Simple Storage Service User Guide.

For more information, see the following topics:

• Configuring Amazon Kinesis Agent for Microsoft Windows

• What Is Amazon Kinesis Data Firehose?

• What Is Amazon S3?

• What is Amazon Athena?