Streaming CloudWatch Logs Data to Amazon Elasticsearch Service
You can configure a CloudWatch Logs log group to stream data it receives to your Amazon Elasticsearch Service (Amazon ES) cluster in near real-time through a CloudWatch Logs subscription. For more information, see Real-time Processing of Log Data with Subscriptions.
Streaming large amounts of CloudWatch Logs data to Amazon ES might result in high usage charges. We recommend that you monitor your AWS bill to help avoid higher-than-expected charges. For more information, see Monitor Your Estimated Charges Using CloudWatch.
Before you begin, create an Amazon ES domain with default settings. You might want to review your Amazon ES domain settings later, and modify you cluster configuration based on the amount of data your cluster will be processing.
For more information about Amazon ES, see the Amazon Elasticsearch Service Developer Guide.
To create an Amazon ES domain
At a command prompt, use the following create-elasticsearch-domain command:
aws es create-elasticsearch-domain –-domain-name
Subscribe a Log Group to Amazon ES
You can use the CloudWatch console to subscribe a log group to Amazon ES.
To subscribe a log group to Amazon ES
Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/.
In the navigation pane, choose Logs.
Select the log group to subscribe.
Choose Actions, Stream to Amazon Elasticsearch Service.
On the Start Streaming to Amazon Elasticsearch Service screen, for Amazon ES cluster, choose the cluster you created in the previous step, and then choose Next.
Under Lambda Function, for Lambda IAM Execution Role, choose the IAM role that Lambda should use when executing calls to Amazon ES, and then choose Next.
On the Configure Log Format and Filters screen, for Log Format, choose a log format.
Under Subscription Filter Pattern, type the terms or pattern to find in your log events. This ensures that you send only the data you are interested in to your Amazon ES cluster. For more information, see Searching and Filtering Log Data.
(Optional) Under Select Log Data to Test, select a log stream and then click Test Pattern to verify that your search filter is returning the results you expect.
Choose Next, and then on the Review & Start Streaming to Amazon Elasticsearch Service screen, choose Start Streaming.