Tutorial: Collect host-level logs with Fluent Bit
In this tutorial, you set up automated log collection on AWS Batch managed compute environment instances using Fluent Bit. After completing this tutorial, host-level diagnostic logs upload automatically to Amazon S3 for troubleshooting and sharing with AWS Support.
Host-level logs are diagnostic files generated by the underlying Amazon EC2 instance. These
logs include Amazon ECS agent logs, Docker and containerd runtime logs, kernel messages, system
journal entries, and networking state. They differ from job container logs, which capture
the stdout and stderr output of your application. For more
information about container logging, see Use the awslogs log driver.
You can use this approach with AWS Batch managed compute environments that use Amazon EC2 On-Demand and Amazon EC2 Spot. To turn logging on or off, the compute environment must support infrastructure updates. For more information, see Perform infrastructure updates.
The instructions in this topic assume that you are using the Amazon ECS-optimized AMI for Amazon Linux 2023. If you use a custom AMI, you might need additional configuration.
Fluent Bit runs on each instance as a systemd service, installed through a launch template user-data script. The agent collects diagnostic log files, applies a Lua-based redaction filter to remove sensitive data, and uploads compressed log files to Amazon S3 organized by instance ID.
Important
The sample Lua redaction scripts provide best-effort redaction of sensitive data. Redacted data includes secret environment variables, Amazon ECS configuration values, and Docker inspect environment output. Add custom patterns for workload-specific secrets. Review log output before sharing files with external parties.
This tutorial takes approximately 30 minutes to complete.
Important
The resources you create in this tutorial might result in charges to your AWS
account. For more information, see the pricing pages for
Amazon EC2
Prerequisites
An AWS account with permission to create IAM policies, Amazon EC2 launch templates, Amazon S3 buckets, and AWS Batch compute environments.
AWS Command Line Interface installed and configured (if you prefer CLI steps).
To set up host-level log collection, complete the following steps:
Set up the Amazon S3 bucket and Fluent Bit configuration.
Configure IAM permissions on your instance role.
Create a launch template and attach it to a compute environment.
Verify and troubleshoot.
Clean up resources.
Topics
