AWS Service Logs

Centralized Logging with OpenSearch supports ingesting AWS service logs into Amazon OpenSearch Service through log analytics pipelines, which you can build using the Centralized Logging with OpenSearch web console or via a standalone CloudFormation template.

Centralized Logging with OpenSearch reads the data source, parse, cleanup/enrich and ingest logs into Amazon OpenSearch Service domains for analysis. Moreover, the solution provides templated dashboards to facilitate log visualization.

Important

AWS managed services must be in the same region as Centralized Logging with OpenSearch. To ingest logs from different AWS regions, we recommend using S3 cross-region replication. The solution will rotate the index on a daily basis, and cannot be adjusted.

Supported AWS Services

Most of AWS managed services output logs to Amazon CloudWatch Logs, Amazon S3, Amazon Kinesis Data Streams or Amazon Kinesis Firehose.

The following table lists the supported AWS services and the corresponding features.

AWS Service	Log Type	Log Location	Automatic Ingestion	Built-in Dashboard
Amazon CloudTrail	N/A	S3	Yes	Yes
Amazon S3	Access logs	S3	Yes	Yes
Amazon RDS/Aurora	MySQL Logs	CloudWatch Logs	Yes	Yes
Amazon CloudFront	Standard access logs	S3	Yes	Yes
Application Load Balancer	Access logs	S3	Yes	Yes
AWS WAF	Web ACL logs	S3	Yes	Yes
AWS Lambda	N/A	CloudWatch Logs	Yes	Yes
Amazon VPC	Flow logs	S3	Yes	Yes
AWS Config	N/A	S3	Yes	Yes

• Automatic Ingestion: The solution detects the log location of the resource automatically and then reads the logs.

• Built-in Dashboard: An out-of-box dashboard for the specified AWS service. The solution will automatically ingest a dashboard into the Amazon OpenSearch Service.

Most of supported AWS services in Centralized Logging with OpenSearch offers built-in dashboard when creating the log analytics pipelines. You go to the OpenSearch Dashboards to view the dashboards after the pipeline being provisioned.

In this chapter, you will learn how to create log ingestion and dashboards for the following AWS services:

• Amazon CloudTrail

• Amazon S3

• Amazon RDS/Aurora

• Amazon CloudFront

• AWS Lambda

• Elastic Load Balancing

• AWS WAF

• Amazon VPC

• AWS Config

Cross-Region Logging

When you deploy Centralized Logging with OpenSearch in one Region, the solution allows you to process service logs from another Region.

Note

For Amazon RDS/Aurora and AWS Lambda service logs, this feature is not supported.

The Region where the service resides is referred to as the Source Region. The Region where the Centralized Logging with OpenSearch console is deployed is referred to as the Logging Region.

For Amazon CloudTrail, you can create a new trail which send logs into a S3 bucket in the Logging Region. To learn how to create a new trail, please refer to Creating a trail.

For other services with logs located in S3 buckets, you can manually transfer logs (for example, using S3 Cross-Region Replication feature) to the Logging Region S3 bucket.

Follow the steps below to implement Cross-Region Logging:

1. Set the service log location in another Region to be the Logging Region (such as AWS WAF), or automatically copy logs from the Source Region to the Logging Region using CRR.

2. In the solution console, choose AWS Service Log in the left navigation pane. Then choose Create a pipeline.

3. In the Select an AWS Service area, choose a service in the list. Choose Next.

4. In Creation Method, choose Manual, then enter the resource name and S3 log location parameter, and choose Next.

5. Set OpenSearch domain and Log Lifecycle as needed, and choose Next.

6. Add tags if you need, and choose Next to create the pipeline.

Then you can use the OpenSearch dashboard to discover logs and view dashboards.