Streaming CloudWatch Logs Data into the Amazon Elasticsearch Service
You can configure a CloudWatch Logs log group to stream data it receives to your Amazon Elasticsearch Service (Amazon ES) cluster in near real-time through a CloudWatch Logs subscription. For more information about subscriptions, see Real-time Processing of Log Data with Subscriptions. The CloudWatch console offers a simple wizard to assist you in setting this up. For more information about Amazon ES, see What Is Amazon Elasticsearch Service? in the Amazon Elasticsearch Service Developer Guide.
Streaming large amounts of CloudWatch Logs data into Amazon ES might result in high usage charges on your AWS bill. We recommend that you monitor your AWS bill to help avoid higher-than-expected charges. For more information, see Monitor Your Estimated Charges Using CloudWatch.
Step 1: Create an Amazon ES domain
In this step, you'll create a new Amazon Elasticsearch Service (Amazon ES) domain with default settings.
Step 2: Subscribe a Log Group to Amazon ES
In this step, you will use the CloudWatch console to subscribe a log group to Amazon Elasticsearch Service (Amazon ES).
To subscribe a log group to Amazon ES
Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/.
If necessary, change the region. From the navigation bar, select the region that meets your needs. For more information, see Regions and Endpoints in the Amazon Web Services General Reference.
In the navigation pane, click Logs.
In the CloudWatch console, select the log group that you want to stream to Amazon ES, click Actions, and then select Stream to Amazon Elasticsearch Service.
On the Start Streaming to Amazon Elasticsearch Service screen, in the Amazon ES cluster drop-down list, select the cluster you created in Step 1: Create an Amazon ES domain, and then choose Next.
Under Lambda Function, in the Lambda IAM Execution Role drop-down list box, select the IAM role that Lambda should use when executing calls to Amazon ES, and then choose Next.
On the Configure Log Format and Filters screen, in the Log Format drop-down list box, select the log format you want to use.
Under Subscription Filter Pattern, enter the terms or pattern you want to find in your log events. This allows you to limit the data sent to your Amazon ES cluster to only the data you are interested in. For more information on using filter patterns, see Searching and Filtering Log Data.
(Optional): Under Select Log Data to Test, select a log stream and then click Test Pattern to verify that your search filter is returning the results you expect.
Click Next, and then on the Review & Start Streaming to Amazon Elasticsearch Service screen, edit any of the information as necessary. If you're satisfied with the settings, click Start Streaming.