Amazon S3

For Amazon S3, Centralized Logging with OpenSearch ingests logs in a specified Amazon S3 location continuously or performs one-time ingestion. You can also filter logs based on Amazon S3 prefix or parse logs with custom Log Config.

The following guides you to create a log pipeline that ingests logs from an Amazon S3 bucket.

Create a log analytics pipeline (OpenSearch Engine)

Prerequisites

Make sure you have imported an Amazon OpenSearch Service domain. For more information, see Domain operations.

Follow these steps:

1. Sign in to the Centralized Logging with OpenSearch Console.

2. In the left sidebar, under Log Analytics Pipelines, choose Application Log.

3. Choose Create a pipeline.

4. Choose Amazon S3 as Log Source, choose Amazon OpenSearch Service, and choose Next.

5. Choose the Amazon S3 bucket where your logs are stored. If needed, enter Prefix filter, which is optional.

6. Choose Ingestion mode based on your need. If you want to ingest logs continuously, select On-going; if you only must ingest logs once, select One-time.

7. Specify Compression format if your log files are compressed, and choose Next.

You have created a log source for the log analytics pipeline. Now you are ready to make further configurations for the log analytics pipeline with Amazon S3 as log source.

1. Select a log config. If you do not find the desired log config from the dropdown list, choose Create New. Refer to Log Config for more information.

2. Choose Next.

3. Specify Index name in lowercase.

4. In the Specify OpenSearch domain section, select an imported domain for Amazon OpenSearch Service domain.

5. In the Log Lifecycle section, enter the number of days to manage the Amazon OpenSearch Service index lifecycle. The Centralized Logging with OpenSearch creates the associated Index State Management (ISM) policy automatically for this pipeline.

6. Choose Next.

7. Enable Alarms if needed and select an existing SNS topic. If you choose Create a new SNS topic, provide a name and an email address for the new SNS topic.

8. Add tags if needed.

9. Choose Create.

10. Wait for the application pipeline to turn to an "Active" state.

Create a log analytics pipeline (Light Engine)

Follow these steps:

1. Sign in to the Centralized Logging with OpenSearch Console.

2. In the left sidebar, under Log Analytics Pipelines, choose Application Log.

3. Choose Create a pipeline.

4. Choose Amazon S3 as Log Source, choose Light Engine, and choose Next.

5. Choose the Amazon S3 bucket where your logs are stored. If needed, enter Prefix filter, which is optional.

6. Choose Ingestion mode based on your need. If you want to ingest the log continuously, select On-going.

You have created a log source for the log analytics pipeline. Now you are ready to make further configurations for the log analytics pipeline with Amazon S3 as log source.

1. Select a log config. If you do not find the desired log config from the dropdown list, choose Create New. Refer to Log Config for more information.

2. Choose Next.

3. In the Specify Light Engine Configuration section, if you want to ingest associated templated Grafana dashboards, select Yes for the sample dashboard.

4. You can choose an existing Grafana, or if you must import a new one, you can go to Grafana for configuration.

5. Select an S3 bucket to store partitioned logs and define a name for the log table. We have provided a predefined table name, but you can modify it according to your business needs.

6. The log processing frequency is set to 5 minutes by default, with a minimum processing frequency of 1 minute.

7. In the Log Lifecycle section, enter the log merge time and log archive time. We have provided default values, but you can adjust them based on your business requirements.

8. Select Next.

9. Enable Alarms if needed and select an existing SNS topic. If you choose Create a new SNS topic, provide a name and an email address for the new SNS topic.

10. If desired, add tags.

11. Select Create.

12. Wait for the application pipeline to turn to "Active" state.