Use CloudWatch Logs to monitor AWS Batch on Amazon EKS jobs

You can use Amazon CloudWatch Logs to monitor, store, and view all your log files in one location. Using CloudWatch Logs, you can search, filter, and analyze log data from multiple sources.

You can download an AWS for Fluent Bit image that includes a plugin to monitor AWS Batch on Amazon EKS jobs in CloudWatch Logs. Fluent Bit is an open-source log processor and forwarder that's both Docker and Kubernetes compatible. We recommend that you use Fluent Bit as your log router because it's less resource intensive than Fluentd. For more information, see Using the AWS for Fluent Bit image.

Prerequisites

Attach the CloudWatchAgentServerPolicy policy to the AWS Identity and Access Management policy of your worker nodes. For more information, see Verify prerequisites.

Install AWS for Fluent Bit

For instructions on how to install AWS for Fluent Bit and create the CloudWatch groups, see Setting up Fluent Bit or Quick Start with the CloudWatch agent and Fluent Bit.

Tip

Remember that Fluent Bit uses .5 CPU and 100 MB of memory on AWS Batch nodes. This reduces the total available capacity for AWS Batch jobs. Consider this when you size your jobs.

Turn on Fluent Bit for AWS Batch nodes

To ensure the Fluent Bit logging DaemonSet runs on AWS Batch managed nodes, modify the Fluent Bit DaemonSet tolerations:

tolerations:
- key: "batch.amazonaws.com/batch-node"
  operator: "Exists"