Required IAM permissions Determine when to use the multiline log setting Parse and concatenate options

Concatenating multiline or stack-trace Amazon ECS log messages

Beginning with AWS for Fluent Bit version 2.22.0, a multiline filter is included. The multiline filter helps concatenate log messages that originally belong to one context but were split across multiple records or log lines. For more information about the multiline filter, see the Fluent Bit documentation.

Common examples of split log messages are:

Stack traces.
Applications that print logs on multiple lines.
Log messages that were split because they were longer than the specified runtime max buffer size. You can concatenate log messages split by the container runtime by following the example on GitHub: FireLens Example: Concatenate Partial/Split Container Logs.

Required IAM permissions

You have the necessary IAM permissions for the container agent to pull the container images from Amazon ECR and for the container to route logs to CloudWatch Logs.

For these permissions, you must have the following roles:

A task IAM role.
A task execution IAM role.

To use the JSON policy editor to create a policy

Sign in to the AWS Management Console and open the IAM console at https://console.aws.amazon.com/iam/.
In the navigation pane on the left, choose Policies.

If this is your first time choosing Policies, the Welcome to Managed Policies page appears. Choose Get Started.
At the top of the page, choose Create policy.
In the Policy editor section, choose the JSON option.

Enter the following JSON policy document:


{
	"Version": "2012-10-17",
	"Statement": [{
		"Effect": "Allow",
		"Action": [
			"logs:CreateLogStream",
			"logs:CreateLogGroup",
			"logs:PutLogEvents"
		],
		"Resource": "*"
	}]
}

Choose Next.

Note
You can switch between the Visual and JSON editor options anytime. However, if you make changes or choose Next in the Visual editor, IAM might restructure your policy to optimize it for the visual editor. For more information, see Policy restructuring in the IAM User Guide.
On the Review and create page, enter a Policy name and a Description (optional) for the policy that you are creating. Review Permissions defined in this policy to see the permissions that are granted by your policy.
Choose Create policy to save your new policy.

Determine when to use the multiline log setting

The following are example log snippets that you see in the CloudWatch Logs console with the default log setting. You can look at the line that starts with log to determine if you need the multiline filter. When the context is the same, you can use the multiline log setting, In this example, the context is "com.myproject.model.MyProject".


2022-09-20T15:47:56:595-05-00                           {"container_id": "82ba37cada1d44d389b03e78caf74faa-EXAMPLE", "container_name": "example-app", "source=": "stdout", "log": ": "     at com.myproject.modele.(MyProject.badMethod.java:22)",
    {
      "container_id":  "82ba37cada1d44d389b03e78caf74faa-EXAMPLE",
      "container_name: ": "example-app",
      "source": "stdout",
      "log": ": "     at com.myproject.model.MyProject.badMethod(MyProject.java:22)",
      "ecs_cluster": "default",
      "ecs_task_arn": "arn:aws:region:123456789012:task/default/b23c940d29ed4714971cba72cEXAMPLE",
      "ecs_task_definition": "firelense-example-multiline:3"
     }


2022-09-20T15:47:56:595-05-00                           {"container_id": "82ba37cada1d44d389b03e78caf74faa-EXAMPLE", "container_name": "example-app", "stdout", "log": ": "     at com.myproject.modele.(MyProject.oneMoreMethod.java:18)",
    {
      "container_id":  "82ba37cada1d44d389b03e78caf74faa-EXAMPLE",
      "container_name: ": "example-app",
      "source": "stdout",
      "log": ": "     at com.myproject.model.MyProject.oneMoreMethod(MyProject.java:18)",
      "ecs_cluster": "default",
      "ecs_task_arn": "arn:aws:region:123456789012:task/default/b23c940d29ed4714971cba72cEXAMPLE,
      "ecs_task_definition": "firelense-example-multiline:3"
     }

After you use the multiline log setting, the output will look similar to the example below.


2022-09-20T15:47:56:595-05-00                           {"container_id": "82ba37cada1d44d389b03e78caf74faa-EXAMPLE", "container_name": "example-app", "stdout",...
    {
      "container_id":  "82ba37cada1d44d389b03e78caf74faa-EXAMPLE",
      "container_name: ": "example-app",
      "source": "stdout",
      "log:    "September 20, 2022 06:41:48 Exception in thread \"main\" java.lang.RuntimeException: Something has gone wrong, aborting!\n    
    at com.myproject.module.MyProject.badMethod(MyProject.java:22)\n    at   
    at com.myproject.model.MyProject.oneMoreMethod(MyProject.java:18) com.myproject.module.MyProject.main(MyProject.java:6)",
      "ecs_cluster": "default",
      "ecs_task_arn": "arn:aws:region:123456789012:task/default/b23c940d29ed4714971cba72cEXAMPLE",
      "ecs_task_definition": "firelense-example-multiline:2"
     }

Parse and concatenate options

To parse logs and concatenate lines that were split because of newlines, you can use either of these two options.

Use your own parser file that contains the rules to parse and concatenate lines that belong to the same message.
Use a Fluent Bit built-in parser. For a list of languages supported by the Fluent Bit built-in parsers, see Fluent Bit documentation.

The following tutorial walks you through the steps for each use case. The steps show you how to concatenate multilines and send the logs to Amazon CloudWatch. You can specify a different destination for your logs.

Example: Use a parser that you create

In this example, you will complete the following steps:

Build and upload the image for a Fluent Bit container.
Build and upload the image for a demo multiline application that runs, fails, and generates a multiline stack trace.
Create the task definition and run the task.
View the logs to verify that messages that span multiple lines appear concatenated.

Build and upload the image for a Fluent Bit container

This image will include the parser file where you specify the regular expression and a configuration file that references the parser file.

Create a folder with the name FluentBitDockerImage.

Within the folder, create a parser file that contains the rules to parse the log and concatenate lines that belong in the same message.

Paste the following contents in the parser file:


[MULTILINE_PARSER]
    name          multiline-regex-test
    type          regex
    flush_timeout 1000
    #
    # Regex rules for multiline parsing
    # ---------------------------------
    #
    # configuration hints:
    #
    #  - first state always has the name: start_state
    #  - every field in the rule must be inside double quotes
    #
    # rules |   state name  | regex pattern                  | next state
    # ------|---------------|--------------------------------------------
    rule      "start_state"   "/(Dec \d+ \d+\:\d+\:\d+)(.*)/"  "cont"
    rule      "cont"          "/^\s+at.*/"                     "cont"

As you customize your regex pattern, we recommend you use a regular expression editor to test the expression.

Save the file as parsers_multiline.conf.

Within the FluentBitDockerImage folder, create a custom configuration file that references the parser file that you created in the previous step.

For more information about the custom configuration file, see Specifying a custom configuration file in the Amazon Elastic Container Service Developer Guide
1. Paste the following contents in the file:
```
[SERVICE]
    flush                 1
    log_level             info
    parsers_file          /parsers_multiline.conf
    
[FILTER]
    name                  multiline
    match                 *
    multiline.key_content log
    multiline.parser      multiline-regex-test
```
  Note
  You must use the absolute path of the parser.
2. Save the file as extra.conf.
Within the FluentBitDockerImage folder, create the Dockerfile with the Fluent Bit image and the parser and configuration files that you created.
1. Paste the following contents in the file:
```
FROM public.ecr.aws/aws-observability/aws-for-fluent-bit:latest

ADD parsers_multiline.conf /parsers_multiline.conf
ADD extra.conf /extra.conf
```
2. Save the file as Dockerfile.
Using the Dockerfile, build a custom Fluent Bit image with the parser and custom configuration files included.

Note
You can place the parser file and configuration file anywhere in the Docker image except /fluent-bit/etc/fluent-bit.conf as this file path is used by FireLens.
1. Build the image: docker build -t fluent-bit-multiline-image .
  
  Where: fluent-bit-multiline-image is the name for the image in this example.
2. Verify that the image was created correctly: docker images —filter reference=fluent-bit-multiline-image
  
  If successful, the output shows the image and the latest tag.
Upload the custom Fluent Bit image to Amazon Elastic Container Registry.
1. Create an Amazon ECR repository to store the image: aws ecr create-repository --repository-name fluent-bit-multiline-repo --region us-east-1
  
  Where: fluent-bit-multiline-repo is the name for the repository and us-east-1 is the region in this example.
  
  The output gives you the details of the new repository.
2. Tag your image with the repositoryUri value from the previous output: docker tag fluent-bit-multiline-image repositoryUri
  
  Example: docker tag fluent-bit-multiline-image xxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/fluent-bit-multiline-repo
3. Run the docker image to verify it ran correctly: docker images —filter reference=repositoryUri
  
  In the output, the repository name changes from fluent-bit-multiline-repo to the repositoryUri.
4. Authenticate to Amazon ECR by running the aws ecr get-login-password command and specifying the registry ID you want to authenticate to: aws ecr get-login-password | docker login --username AWS --password-stdin registry ID.dkr.ecr.region.amazonaws.com
  
  Example: ecr get-login-password | docker login --username AWS --password-stdin xxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com
  
  A successful login message appears.
5. Push the image to Amazon ECR: docker push registry ID.dkr.ecr.region.amazonaws.com/repository name
  
  Example: docker push xxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/fluent-bit-multiline-repo

Build and upload the image for a demo multiline application

This image will include a Python script file that runs the application and a sample log file.

When you run the task, the application simulates runs, then fails and creates a stack trace.

Create a folder named multiline-app: mkdir multiline-app

Create a Python script file.

Within the multiline-app folder, create a file and name it main.py.

Paste the following contents in the file:


import os
import time
file1 = open('/test.log', 'r')
Lines = file1.readlines()
 
count = 0

for i in range(10):
    print("app running normally...")
    time.sleep(1)

# Strips the newline character
for line in Lines:
    count += 1
    print(line.rstrip())
print(count)
print("app terminated.")

Save the main.py file.

Create a sample log file.

Within the multiline-app folder, create a file and name it test.log.

Paste the following contents in the file:


single line...
Dec 14 06:41:08 Exception in thread "main" java.lang.RuntimeException: Something has gone wrong, aborting!
    at com.myproject.module.MyProject.badMethod(MyProject.java:22)
    at com.myproject.module.MyProject.oneMoreMethod(MyProject.java:18)
    at com.myproject.module.MyProject.anotherMethod(MyProject.java:14)
    at com.myproject.module.MyProject.someMethod(MyProject.java:10)
    at com.myproject.module.MyProject.main(MyProject.java:6)
another line...

Save the test.log file.

Within the multiline-app folder, create the Dockerfile.

Paste the following contents in the file:


FROM public.ecr.aws/amazonlinux/amazonlinux:latest
ADD test.log /test.log

RUN yum upgrade -y && yum install -y python3

WORKDIR /usr/local/bin

COPY main.py .

CMD ["python3", "main.py"]

Save the Dockerfile file.

Using the Dockerfile, build an image.
1. Build the image: docker build -t multiline-app-image .
  
  Where: multiline-app-image is the name for the image in this example.
2. Verify that the image was created correctly: docker images —filter reference=multiline-app-image
  
  If successful, the output shows the image and the latest tag.
Upload the image to Amazon Elastic Container Registry.
1. Create an Amazon ECR repository to store the image: aws ecr create-repository --repository-name multiline-app-repo --region us-east-1
  
  Where: multiline-app-repo is the name for the repository and us-east-1 is the region in this example.
  
  The output gives you the details of the new repository. Note the repositoryUri value as you will need it in the next steps.
2. Tag your image with the repositoryUri value from the previous output: docker tag multiline-app-image repositoryUri
  
  Example: docker tag multiline-app-image xxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/multiline-app-repo
3. Run the docker image to verify it ran correctly: docker images —filter reference=repositoryUri
  
  In the output, the repository name changes from multiline-app-repo to the repositoryUri value.
4. Push the image to Amazon ECR: docker push aws_account_id.dkr.ecr.region.amazonaws.com/repository name
  
  Example: docker push xxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/multiline-app-repo

Create the task definition and run the task

Create a task definition file with the file name multiline-task-definition.json.

Paste the following contents in the multiline-task-definition.json file:


{
    "family": "firelens-example-multiline",
    "taskRoleArn": "task role ARN,
    "executionRoleArn": "execution role ARN",
    "containerDefinitions": [
        {
            "essential": true,
            "image": "aws_account_id.dkr.ecr.us-east-1.amazonaws.com/fluent-bit-multiline-image:latest",
            "name": "log_router",
            "firelensConfiguration": {
                "type": "fluentbit",
                "options": {
                    "config-file-type": "file",
                    "config-file-value": "/extra.conf"
                }
            },
            "memoryReservation": 50
        },
        {
            "essential": true,
            "image": "aws_account_id.dkr.ecr.us-east-1.amazonaws.com/multiline-app-image:latest",
            "name": "app",
            "logConfiguration": {
                "logDriver": "awsfirelens",
                "options": {
                    "Name": "cloudwatch_logs",
                    "region": "us-east-1",
                    "log_group_name": "multiline-test/application",
                    "auto_create_group": "true",
                    "log_stream_prefix": "multiline-"
                }
            },
            "memoryReservation": 100
        }
    ],
    "requiresCompatibilities": ["FARGATE"],
    "networkMode": "awsvpc",
    "cpu": "256",
    "memory": "512"
}

Replace the following in the multiline-task-definition.json task definition:

task role ARN

To find the task role ARN, go to the IAM console. Choose Roles and find the ecs-task-role-for-firelens task role that you created. Choose the role and copy the ARN that appears in the Summary section.
execution role ARN

To find the execution role ARN, go to the IAM console. Choose Roles and find the ecsTaskExecutionRole role. Choose the role and copy the ARN that appears in the Summary section.
aws_account_id

To find your aws_account_id, log into the AWS Management Console. Choose your user name on the top right and copy your Account ID.
us-east-1

Replace the region if necessary.

Register the task definition file: aws ecs register-task-definition --cli-input-json file://multiline-task-definition.json --region region
Open the console at https://console.aws.amazon.com/ecs/v2.
In the navigation pane, choose Task Definitions and then choose the firelens-example-multiline family because we registered the task definition to this family in the first line of the task definition above.
Choose the latest version.
Choose the Deploy, Run task.
On the Run Task page, For Cluster, choose the cluster, and then under Networking, for Subnets, choose the available subnets for your task.
Choose Create.

Verify that multiline log messages in Amazon CloudWatch appear concatenated

Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/.
From the navigation pane, expand Logs and choose Log groups.
Choose the multiline-test/applicatio log group.

Choose the log. View messages. Lines that matched the rules in the parser file are concatenated and appear as a single message.

The following log snippet shows lines concatenated in a single Java stack trace event:


{
    "container_id": "xxxxxx",
    "container_name": "app",
    "source": "stdout",
    "log": "Dec 14 06:41:08 Exception in thread \"main\" java.lang.RuntimeException: Something has gone wrong, aborting!\n    at com.myproject.module.MyProject.badMethod(MyProject.java:22)\n    at com.myproject.module.MyProject.oneMoreMethod(MyProject.java:18)\n    at com.myproject.module.MyProject.anotherMethod(MyProject.java:14)\n    at com.myproject.module.MyProject.someMethod(MyProject.java:10)\n    at com.myproject.module.MyProject.main(MyProject.java:6)",
    "ecs_cluster": "default",
    "ecs_task_arn": "arn:aws:ecs:us-east-1:xxxxxxxxxxxx:task/default/xxxxxx",
    "ecs_task_definition": "firelens-example-multiline:2"
}

The following log snippet shows how the same message appears with just a single line if you run an Amazon ECS container that is not configured to concatenate multiline log messages.


{
    "log": "Dec 14 06:41:08 Exception in thread \"main\" java.lang.RuntimeException: Something has gone wrong, aborting!",
    "container_id": "xxxxxx-xxxxxx",
    "container_name": "app",
    "source": "stdout",
    "ecs_cluster": "default",
    "ecs_task_arn": "arn:aws:ecs:us-east-1:xxxxxxxxxxxx:task/default/xxxxxx",
    "ecs_task_definition": "firelens-example-multiline:3"
}

Example: Use a Fluent Bit built-in parser

In this example, you will complete the following steps:

Build and upload the image for a Fluent Bit container.
Build and upload the image for a demo multiline application that runs, fails, and generates a multiline stack trace.
Create the task definition and run the task.
View the logs to verify that messages that span multiple lines appear concatenated.

Build and upload the image for a Fluent Bit container

This image will include a configuration file that references the Fluent Bit parser.

Create a folder with the name FluentBitDockerImage.
Within the FluentBitDockerImage folder, create a custom configuration file that references the Fluent Bit built-in parser file.

For more information about the custom configuration file, see Specifying a custom configuration file in the Amazon Elastic Container Service Developer Guide
1. Paste the following contents in the file:
```
[FILTER]
    name                  multiline
    match                 *
    multiline.key_content log
    multiline.parser      go
```
2. Save the file as extra.conf.
Within the FluentBitDockerImage folder, create the Dockerfile with the Fluent Bit image and the parser and configuration files that you created.
1. Paste the following contents in the file:
```
FROM public.ecr.aws/aws-observability/aws-for-fluent-bit:latest
ADD extra.conf /extra.conf
```
2. Save the file as Dockerfile.
Using the Dockerfile, build a custom Fluent Bit image with the custom configuration file included.

Note
You can place the configuration file anywhere in the Docker image except /fluent-bit/etc/fluent-bit.conf as this file path is used by FireLens.
1. Build the image: docker build -t fluent-bit-multiline-image .
  
  Where: fluent-bit-multiline-image is the name for the image in this example.
2. Verify that the image was created correctly: docker images —filter reference=fluent-bit-multiline-image
  
  If successful, the output shows the image and the latest tag.
Upload the custom Fluent Bit image to Amazon Elastic Container Registry.
1. Create an Amazon ECR repository to store the image: aws ecr create-repository --repository-name fluent-bit-multiline-repo --region us-east-1
  
  Where: fluent-bit-multiline-repo is the name for the repository and us-east-1 is the region in this example.
  
  The output gives you the details of the new repository.
2. Tag your image with the repositoryUri value from the previous output: docker tag fluent-bit-multiline-image repositoryUri
  
  Example: docker tag fluent-bit-multiline-image xxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/fluent-bit-multiline-repo
3. Run the docker image to verify it ran correctly: docker images —filter reference=repositoryUri
  
  In the output, the repository name changes from fluent-bit-multiline-repo to the repositoryUri.
4. Authenticate to Amazon ECR by running the aws ecr get-login-password command and specifying the registry ID you want to authenticate to: aws ecr get-login-password | docker login --username AWS --password-stdin registry ID.dkr.ecr.region.amazonaws.com
  
  Example: ecr get-login-password | docker login --username AWS --password-stdin xxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com
  
  A successful login message appears.
5. Push the image to Amazon ECR: docker push registry ID.dkr.ecr.region.amazonaws.com/repository name
  
  Example: docker push xxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/fluent-bit-multiline-repo

Build and upload the image for a demo multiline application

This image will include a Python script file that runs the application and a sample log file.

Create a folder named multiline-app: mkdir multiline-app

Create a Python script file.

Within the multiline-app folder, create a file and name it main.py.

Paste the following contents in the file:


import os
import time
file1 = open('/test.log', 'r')
Lines = file1.readlines()
 
count = 0

for i in range(10):
    print("app running normally...")
    time.sleep(1)

# Strips the newline character
for line in Lines:
    count += 1
    print(line.rstrip())
print(count)
print("app terminated.")

Save the main.py file.

Create a sample log file.

Within the multiline-app folder, create a file and name it test.log.

Paste the following contents in the file:


panic: my panic

goroutine 4 [running]:
panic(0x45cb40, 0x47ad70)
  /usr/local/go/src/runtime/panic.go:542 +0x46c fp=0xc42003f7b8 sp=0xc42003f710 pc=0x422f7c
main.main.func1(0xc420024120)
  foo.go:6 +0x39 fp=0xc42003f7d8 sp=0xc42003f7b8 pc=0x451339
runtime.goexit()
  /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc42003f7e0 sp=0xc42003f7d8 pc=0x44b4d1
created by main.main
  foo.go:5 +0x58

goroutine 1 [chan receive]:
runtime.gopark(0x4739b8, 0xc420024178, 0x46fcd7, 0xc, 0xc420028e17, 0x3)
  /usr/local/go/src/runtime/proc.go:280 +0x12c fp=0xc420053e30 sp=0xc420053e00 pc=0x42503c
runtime.goparkunlock(0xc420024178, 0x46fcd7, 0xc, 0x1000f010040c217, 0x3)
  /usr/local/go/src/runtime/proc.go:286 +0x5e fp=0xc420053e70 sp=0xc420053e30 pc=0x42512e
runtime.chanrecv(0xc420024120, 0x0, 0xc420053f01, 0x4512d8)
  /usr/local/go/src/runtime/chan.go:506 +0x304 fp=0xc420053f20 sp=0xc420053e70 pc=0x4046b4
runtime.chanrecv1(0xc420024120, 0x0)
  /usr/local/go/src/runtime/chan.go:388 +0x2b fp=0xc420053f50 sp=0xc420053f20 pc=0x40439b
main.main()
  foo.go:9 +0x6f fp=0xc420053f80 sp=0xc420053f50 pc=0x4512ef
runtime.main()
  /usr/local/go/src/runtime/proc.go:185 +0x20d fp=0xc420053fe0 sp=0xc420053f80 pc=0x424bad
runtime.goexit()
  /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc420053fe8 sp=0xc420053fe0 pc=0x44b4d1

goroutine 2 [force gc (idle)]:
runtime.gopark(0x4739b8, 0x4ad720, 0x47001e, 0xf, 0x14, 0x1)
  /usr/local/go/src/runtime/proc.go:280 +0x12c fp=0xc42003e768 sp=0xc42003e738 pc=0x42503c
runtime.goparkunlock(0x4ad720, 0x47001e, 0xf, 0xc420000114, 0x1)
  /usr/local/go/src/runtime/proc.go:286 +0x5e fp=0xc42003e7a8 sp=0xc42003e768 pc=0x42512e
runtime.forcegchelper()
  /usr/local/go/src/runtime/proc.go:238 +0xcc fp=0xc42003e7e0 sp=0xc42003e7a8 pc=0x424e5c
runtime.goexit()
  /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc42003e7e8 sp=0xc42003e7e0 pc=0x44b4d1
created by runtime.init.4
  /usr/local/go/src/runtime/proc.go:227 +0x35

goroutine 3 [GC sweep wait]:
runtime.gopark(0x4739b8, 0x4ad7e0, 0x46fdd2, 0xd, 0x419914, 0x1)
  /usr/local/go/src/runtime/proc.go:280 +0x12c fp=0xc42003ef60 sp=0xc42003ef30 pc=0x42503c
runtime.goparkunlock(0x4ad7e0, 0x46fdd2, 0xd, 0x14, 0x1)
  /usr/local/go/src/runtime/proc.go:286 +0x5e fp=0xc42003efa0 sp=0xc42003ef60 pc=0x42512e
runtime.bgsweep(0xc42001e150)
  /usr/local/go/src/runtime/mgcsweep.go:52 +0xa3 fp=0xc42003efd8 sp=0xc42003efa0 pc=0x419973
runtime.goexit()
  /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc42003efe0 sp=0xc42003efd8 pc=0x44b4d1
created by runtime.gcenable
  /usr/local/go/src/runtime/mgc.go:216 +0x58
one more line, no multiline

Save the test.log file.

Within the multiline-app folder, create the Dockerfile.

Paste the following contents in the file:


FROM public.ecr.aws/amazonlinux/amazonlinux:latest
ADD test.log /test.log

RUN yum upgrade -y && yum install -y python3

WORKDIR /usr/local/bin

COPY main.py .

CMD ["python3", "main.py"]

Save the Dockerfile file.

Using the Dockerfile, build an image.
1. Build the image: docker build -t multiline-app-image .
  
  Where: multiline-app-image is the name for the image in this example.
2. Verify that the image was created correctly: docker images —filter reference=multiline-app-image
  
  If successful, the output shows the image and the latest tag.
Upload the image to Amazon Elastic Container Registry.
1. Create an Amazon ECR repository to store the image: aws ecr create-repository --repository-name multiline-app-repo --region us-east-1
  
  Where: multiline-app-repo is the name for the repository and us-east-1 is the region in this example.
  
  The output gives you the details of the new repository. Note the repositoryUri value as you will need it in the next steps.
2. Tag your image with the repositoryUri value from the previous output: docker tag multiline-app-image repositoryUri
  
  Example: docker tag multiline-app-image xxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/multiline-app-repo
3. Run the docker image to verify it ran correctly: docker images —filter reference=repositoryUri
  
  In the output, the repository name changes from multiline-app-repo to the repositoryUri value.
4. Push the image to Amazon ECR: docker push aws_account_id.dkr.ecr.region.amazonaws.com/repository name
  
  Example: docker push xxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/multiline-app-repo

Create the task definition and run the task

Create a task definition file with the file name multiline-task-definition.json.

Paste the following contents in the multiline-task-definition.json file:


{
    "family": "firelens-example-multiline",
    "taskRoleArn": "task role ARN,
    "executionRoleArn": "execution role ARN",
    "containerDefinitions": [
        {
            "essential": true,
            "image": "aws_account_id.dkr.ecr.us-east-1.amazonaws.com/fluent-bit-multiline-image:latest",
            "name": "log_router",
            "firelensConfiguration": {
                "type": "fluentbit",
                "options": {
                    "config-file-type": "file",
                    "config-file-value": "/extra.conf"
                }
            },
            "memoryReservation": 50
        },
        {
            "essential": true,
            "image": "aws_account_id.dkr.ecr.us-east-1.amazonaws.com/multiline-app-image:latest",
            "name": "app",
            "logConfiguration": {
                "logDriver": "awsfirelens",
                "options": {
                    "Name": "cloudwatch_logs",
                    "region": "us-east-1",
                    "log_group_name": "multiline-test/application",
                    "auto_create_group": "true",
                    "log_stream_prefix": "multiline-"
                }
            },
            "memoryReservation": 100
        }
    ],
    "requiresCompatibilities": ["FARGATE"],
    "networkMode": "awsvpc",
    "cpu": "256",
    "memory": "512"
}

Replace the following in the multiline-task-definition.json task definition:

task role ARN

To find the task role ARN, go to the IAM console. Choose Roles and find the ecs-task-role-for-firelens task role that you created. Choose the role and copy the ARN that appears in the Summary section.
execution role ARN

To find the execution role ARN, go to the IAM console. Choose Roles and find the ecsTaskExecutionRole role. Choose the role and copy the ARN that appears in the Summary section.
aws_account_id

To find your aws_account_id, log into the AWS Management Console. Choose your user name on the top right and copy your Account ID.
us-east-1

Replace the region if necessary.

Register the task definition file: aws ecs register-task-definition --cli-input-json file://multiline-task-definition.json --region us-east-1
Open the console at https://console.aws.amazon.com/ecs/v2.
In the navigation pane, choose Task Definitions and then choose the firelens-example-multiline family because we registered the task definition to this family in the first line of the task definition above.
Choose the latest version.
Choose the Deploy, Run task.
On the Run Task page, For Cluster, choose the cluster, and then under Networking, for Subnets, choose the available subnets for your task.
Choose Create.

Verify that multiline log messages in Amazon CloudWatch appear concatenated

Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/.
From the navigation pane, expand Logs and choose Log groups.
Choose the multiline-test/applicatio log group.

Choose the log and view the messages. Lines that matched the rules in the parser file are concatenated and appear as a single message.

The following log snippet shows a Go stack trace that is concatenated into a single event:


{
    "log": "panic: my panic\n\ngoroutine 4 [running]:\npanic(0x45cb40, 0x47ad70)\n  /usr/local/go/src/runtime/panic.go:542 +0x46c fp=0xc42003f7b8 sp=0xc42003f710 pc=0x422f7c\nmain.main.func1(0xc420024120)\n  foo.go:6 +0x39 fp=0xc42003f7d8 sp=0xc42003f7b8 pc=0x451339\nruntime.goexit()\n  /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc42003f7e0 sp=0xc42003f7d8 pc=0x44b4d1\ncreated by main.main\n  foo.go:5 +0x58\n\ngoroutine 1 [chan receive]:\nruntime.gopark(0x4739b8, 0xc420024178, 0x46fcd7, 0xc, 0xc420028e17, 0x3)\n  /usr/local/go/src/runtime/proc.go:280 +0x12c fp=0xc420053e30 sp=0xc420053e00 pc=0x42503c\nruntime.goparkunlock(0xc420024178, 0x46fcd7, 0xc, 0x1000f010040c217, 0x3)\n  /usr/local/go/src/runtime/proc.go:286 +0x5e fp=0xc420053e70 sp=0xc420053e30 pc=0x42512e\nruntime.chanrecv(0xc420024120, 0x0, 0xc420053f01, 0x4512d8)\n  /usr/local/go/src/runtime/chan.go:506 +0x304 fp=0xc420053f20 sp=0xc420053e70 pc=0x4046b4\nruntime.chanrecv1(0xc420024120, 0x0)\n  /usr/local/go/src/runtime/chan.go:388 +0x2b fp=0xc420053f50 sp=0xc420053f20 pc=0x40439b\nmain.main()\n  foo.go:9 +0x6f fp=0xc420053f80 sp=0xc420053f50 pc=0x4512ef\nruntime.main()\n  /usr/local/go/src/runtime/proc.go:185 +0x20d fp=0xc420053fe0 sp=0xc420053f80 pc=0x424bad\nruntime.goexit()\n  /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc420053fe8 sp=0xc420053fe0 pc=0x44b4d1\n\ngoroutine 2 [force gc (idle)]:\nruntime.gopark(0x4739b8, 0x4ad720, 0x47001e, 0xf, 0x14, 0x1)\n  /usr/local/go/src/runtime/proc.go:280 +0x12c fp=0xc42003e768 sp=0xc42003e738 pc=0x42503c\nruntime.goparkunlock(0x4ad720, 0x47001e, 0xf, 0xc420000114, 0x1)\n  /usr/local/go/src/runtime/proc.go:286 +0x5e fp=0xc42003e7a8 sp=0xc42003e768 pc=0x42512e\nruntime.forcegchelper()\n  /usr/local/go/src/runtime/proc.go:238 +0xcc fp=0xc42003e7e0 sp=0xc42003e7a8 pc=0x424e5c\nruntime.goexit()\n  /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc42003e7e8 sp=0xc42003e7e0 pc=0x44b4d1\ncreated by runtime.init.4\n  /usr/local/go/src/runtime/proc.go:227 +0x35\n\ngoroutine 3 [GC sweep wait]:\nruntime.gopark(0x4739b8, 0x4ad7e0, 0x46fdd2, 0xd, 0x419914, 0x1)\n  /usr/local/go/src/runtime/proc.go:280 +0x12c fp=0xc42003ef60 sp=0xc42003ef30 pc=0x42503c\nruntime.goparkunlock(0x4ad7e0, 0x46fdd2, 0xd, 0x14, 0x1)\n  /usr/local/go/src/runtime/proc.go:286 +0x5e fp=0xc42003efa0 sp=0xc42003ef60 pc=0x42512e\nruntime.bgsweep(0xc42001e150)\n  /usr/local/go/src/runtime/mgcsweep.go:52 +0xa3 fp=0xc42003efd8 sp=0xc42003efa0 pc=0x419973\nruntime.goexit()\n  /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc42003efe0 sp=0xc42003efd8 pc=0x44b4d1\ncreated by runtime.gcenable\n  /usr/local/go/src/runtime/mgc.go:216 +0x58",
    "container_id": "xxxxxx-xxxxxx",
    "container_name": "app",
    "source": "stdout",
    "ecs_cluster": "default",
    "ecs_task_arn": "arn:aws:ecs:us-east-1:xxxxxxxxxxxx:task/default/xxxxxx",
    "ecs_task_definition": "firelens-example-multiline:2"
}

The following log snippet shows how the same event appears if you run an ECS container that is not configured to concatenate multiline log messages. The log field contains a single line.


{
    "log": "panic: my panic",
    "container_id": "xxxxxx-xxxxxx",
    "container_name": "app",
    "source": "stdout",
    "ecs_cluster": "default",
    "ecs_task_arn": "arn:aws:ecs:us-east-1:xxxxxxxxxxxx:task/default/xxxxxx",
    "ecs_task_definition": "firelens-example-multiline:3"

Note

If your logs go to log files instead of the standard output, we recommend specifying the multiline.parser and multiline.key_content configuration parameters in the Tail input plugin instead of the Filter.

Warning Javascript is disabled or is unavailable in your browser.

To use the Amazon Web Services Documentation, Javascript must be enabled. Please refer to your browser's Help pages for instructions.

Document Conventions

Sending Amazon Simple Notification Service alerts for task stopped events

Deploying Fluent Bit on Windows containers