Create a custom log parser for Amazon ECS using a Firelens log router

Created by Varun Sharma (AWS)

Summary

Firelens is a log router for Amazon Elastic Container Service (Amazon ECS) and AWS Fargate. You can use Firelens to route container logs from Amazon ECS to Amazon CloudWatch and other destinations (for example, Splunk or Sumo Logic). Firelens works with Fluentd or Fluent Bit as the logging agent, which means that you can use Amazon ECS task definition parameters to route logs.

By choosing to parse logs at the source level, you can analyze your logging data and perform queries to more efficiently and effectively respond to operational issues. Because different applications have different logging patterns, you need to use a custom parser that structures the logs and makes searching easier at your end destination.

This pattern uses a Firelens log router with a custom parser to push logs to CloudWatch from a sample Spring Boot application running on Amazon ECS. You can then use Amazon CloudWatch Logs Insights to filter the logs based on custom fields that are generated by the custom parser.

Prerequisites and limitations

Prerequisites 

• An active Amazon Web Services (AWS) account.

• AWS Command Line Interface (AWS CLI), installed and configured on your local machine.

• Docker, installed and configured on your local machine.

• An existing Spring Boot-based containerized application on Amazon Elastic Container Registry (Amazon ECR). 

Architecture

Technology stack  

• CloudWatch

• Amazon ECR

• Amazon ECS

• Fargate

• Docker

• Fluent Bit

Tools

• Amazon ECR – Amazon Elastic Container Registry (Amazon ECR) is an AWS managed container image registry service that is secure, scalable, and reliable.

• Amazon ECS – Amazon Elastic Container Service (Amazon ECS) is a highly scalable, fast container management service that makes it easy to run, stop, and manage containers on a cluster.

• AWS Identity and Access Management (IAM) – IAM is a web service for securely controlling access to AWS services.

• AWS CLI – AWS Command Line Interface (AWS CLI) is an open-source tool that enables you to interact with AWS services using commands in your command-line shell.

• Docker – Docker is an open platform for developing, shipping, and running applications.

Code

The following files are attached to this pattern:

• customFluentBit.zip – Contains the files to add the custom parsing and configurations.

• firelens_policy.json – Contains the policy document to create an IAM policy.

• Task.json – Contains a sample task definition for Amazon ECS.

Epics

Create a custom Fluent Bit image

Task	Description	Skills required
Create an Amazon ECR repository.	Sign in to the AWS Management Console, open the Amazon ECR console, and create a repository called fluentbit_custom. For more information about this, see Creating a repository in the Amazon ECR documentation.	Systems administrator, Developer
Unzip the customFluentBit.zip package.	Download the customFluentBit.zip package (attached) to your local machine.  Unzip to the customFluentBit directory by running the following command: unzip -d customFluentBit.zip The directory contains the following files that are required for adding the custom parsing and configurations: parsers/springboot_parser.conf – Contains the parser directive and defines the regular expression (regex) pattern for the custom parser. You can add the regex pattern for your specific parser.  conf/parse_springboot.conf – Contains the filter and the service directive. The Dockerfile
Create the custom Docker image.	Change the directory to customFluentBit. Open the Amazon ECR console, choose the fluentbit_custom repository, and then choose View push commands.  Upload your project.  After the upload is complete, copy the build’s URL. This URL is required when you create a container in Amazon ECS For more information about this, see Pushing a Docker image in the Amazon ECR documentation.	Systems administrator, Developer

Set up the Amazon ECS cluster

Task	Description	Skills required
Create an Amazon ECS cluster.	Create an Amazon ECS cluster by following the instructions from the Networking only template section of Creating a cluster in the Amazon ECS documentation. Note Make sure that you choose Create VPC to create a new virtual private cloud (VPC) for your Amazon ECS cluster.	Systems administrator, Developer

Set up the Amazon ECS task

Task	Description	Skills required
Set up the Amazon ECS task execution IAM role.	Create an Amazon ECS task execution IAM role by using the AmazonECSTaskExecutionRolePolicy managed policy. For more information about this, see Amazon ECS task execution IAM role in the Amazon ECS documentation. Note Make sure that you record the IAM role’s Amazon Resource Name (ARN).	Systems administrator, Developer
Attach the IAM policy to the Amazon ECS task execution IAM role.	Create an IAM policy by using the firelens_policy.json (attached) policy document. For more information about this, see Creating policies on the JSON tab in the IAM documentation. Attach this policy to the Amazon ECS task execution IAM role that you created earlier. For more information about this, see Adding IAM policies (AWS CLI) in the IAM documentation.	Systems administrator, Developer
Set up the Amazon ECS task definition.	Update the following sections in the Task.jsonsample task definition (attached): Update the executionRoleArn and taskRoleArn with the ARN of the task execution IAM role Update the image in containerDefinitions with the custom Fluent Bit Docker image that you created earlier Update the image in containerDefinitions with your application image's name Open the Amazon ECS console, choose Task Definitions, choose Create new task definition, and then choose Fargate on the Select compatibilities page.     Choose Configure via Json, paste the updated Task.json file into the text area, and then choose Save. Create the task definition. For more information about this, see Creating a task definition in the Amazon ECS documentation.	Systems administrator, Developer

Run the Amazon ECS task

Task	Description	Skills required
Run the Amazon ECS task.	On the Amazon ECS console, choose Clusters, choose the cluster that you created earlier, and then run the standalone task. For more information about this, see Run a standalone task in the Amazon ECS documentation.	Systems administrator, Developer

Verify the CloudWatch logs

Task	Description	Skills required
Verify the logs.	Open the CloudWatch console, choose Log groups, and then choose /aws/ecs/containerinsights/{{cluster_ARN}}/firelens/application. Verify the logs, particularly the custom fields added by the custom parser. Use CloudWatch to filter logs based on the custom fields.	Systems administrator, Developer

Related resources

• Docker basics for Amazon ECS 

• Amazon ECS on AWS Fargate 

• Configuring basic service parameters 

Attachments

To access additional content that is associated with this document, unzip the following file: attachment.zip