Concatenating multiline or stack-trace Amazon ECS log messages
Beginning with AWS for Fluent Bit version 2.22.0, a multiline filter is included.
The multiline filter helps concatenate log messages that originally belong to one
context but were split across multiple records or log lines. For more information about
the multiline filter, see the Fluent
Bit documentation
Common examples of split log messages are:
-
Stack traces.
-
Applications that print logs on multiple lines.
-
Log messages that were split because they were longer than the specified runtime max buffer size. You can concatenate log messages split by the container runtime by following the example on GitHub: FireLens Example: Concatenate Partial/Split Container Logs
.
Required IAM permissions
You have the necessary IAM permissions for the container agent to pull the container images from Amazon ECR and for the container to route logs to CloudWatch Logs.
For these permissions, you must have the following roles:
-
A task IAM role.
-
A task execution IAM role.
To use the JSON policy editor to create a policy
Sign in to the AWS Management Console and open the IAM console at https://console.aws.amazon.com/iam/
. -
In the navigation pane on the left, choose Policies.
If this is your first time choosing Policies, the Welcome to Managed Policies page appears. Choose Get Started.
-
At the top of the page, choose Create policy.
-
In the Policy editor section, choose the JSON option.
-
Enter the following JSON policy document:
{ "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Action": [ "logs:CreateLogStream", "logs:CreateLogGroup", "logs:PutLogEvents" ], "Resource": "*" }] }
-
Choose Next.
Note
You can switch between the Visual and JSON editor options anytime. However, if you make changes or choose Next in the Visual editor, IAM might restructure your policy to optimize it for the visual editor. For more information, see Policy restructuring in the IAM User Guide.
-
On the Review and create page, enter a Policy name and a Description (optional) for the policy that you are creating. Review Permissions defined in this policy to see the permissions that are granted by your policy.
-
Choose Create policy to save your new policy.
Determine when to use the multiline log setting
The following are example log snippets that you see in the CloudWatch Logs console with the
default log setting. You can look at the line that starts with log
to
determine if you need the multiline filter. When the context is the same, you can
use the multiline log setting, In this example, the context is "com.myproject.model.MyProject".
2022-09-20T15:47:56:595-05-00 {"container_id": "82ba37cada1d44d389b03e78caf74faa-EXAMPLE", "container_name": "example-app", "source=": "stdout", "log": ": " at com.myproject.modele.(MyProject.badMethod.java:22)",
{
"container_id": "82ba37cada1d44d389b03e78caf74faa-EXAMPLE",
"container_name: ": "example-app",
"source": "stdout",
"log": ": " at com.myproject.model.MyProject.badMethod(MyProject.java:22)",
"ecs_cluster": "default",
"ecs_task_arn": "arn:aws:region:123456789012:task/default/b23c940d29ed4714971cba72cEXAMPLE",
"ecs_task_definition": "firelense-example-multiline:3"
}
2022-09-20T15:47:56:595-05-00 {"container_id": "82ba37cada1d44d389b03e78caf74faa-EXAMPLE", "container_name": "example-app", "stdout", "log": ": " at com.myproject.modele.(MyProject.oneMoreMethod.java:18)",
{
"container_id": "82ba37cada1d44d389b03e78caf74faa-EXAMPLE",
"container_name: ": "example-app",
"source": "stdout",
"log": ": " at com.myproject.model.MyProject.oneMoreMethod(MyProject.java:18)",
"ecs_cluster": "default",
"ecs_task_arn": "arn:aws:region:123456789012:task/default/b23c940d29ed4714971cba72cEXAMPLE,
"ecs_task_definition": "firelense-example-multiline:3"
}
After you use the multiline log setting, the output will look similar to the example below.
2022-09-20T15:47:56:595-05-00 {"container_id": "82ba37cada1d44d389b03e78caf74faa-EXAMPLE", "container_name": "example-app", "stdout",...
{
"container_id": "82ba37cada1d44d389b03e78caf74faa-EXAMPLE",
"container_name: ": "example-app",
"source": "stdout",
"log: "September 20, 2022 06:41:48 Exception in thread \"main\" java.lang.RuntimeException: Something has gone wrong, aborting!\n
at com.myproject.module.MyProject.badMethod(MyProject.java:22)\n at
at com.myproject.model.MyProject.oneMoreMethod(MyProject.java:18) com.myproject.module.MyProject.main(MyProject.java:6)",
"ecs_cluster": "default",
"ecs_task_arn": "arn:aws:region:123456789012:task/default/b23c940d29ed4714971cba72cEXAMPLE",
"ecs_task_definition": "firelense-example-multiline:2"
}
Parse and concatenate options
To parse logs and concatenate lines that were split because of newlines, you can use either of these two options.
-
Use your own parser file that contains the rules to parse and concatenate lines that belong to the same message.
-
Use a Fluent Bit built-in parser. For a list of languages supported by the Fluent Bit built-in parsers, see Fluent Bit documentation
.
The following tutorial walks you through the steps for each use case. The steps show you how to concatenate multilines and send the logs to Amazon CloudWatch. You can specify a different destination for your logs.
Example: Use a parser that you create
In this example, you will complete the following steps:
-
Build and upload the image for a Fluent Bit container.
-
Build and upload the image for a demo multiline application that runs, fails, and generates a multiline stack trace.
-
Create the task definition and run the task.
-
View the logs to verify that messages that span multiple lines appear concatenated.
Build and upload the image for a Fluent Bit container
This image will include the parser file where you specify the regular expression and a configuration file that references the parser file.
-
Create a folder with the name
FluentBitDockerImage
. -
Within the folder, create a parser file that contains the rules to parse the log and concatenate lines that belong in the same message.
-
Paste the following contents in the parser file:
[MULTILINE_PARSER] name multiline-regex-test type regex flush_timeout 1000 # # Regex rules for multiline parsing # --------------------------------- # # configuration hints: # # - first state always has the name: start_state # - every field in the rule must be inside double quotes # # rules | state name | regex pattern | next state # ------|---------------|-------------------------------------------- rule "start_state" "/(Dec \d+ \d+\:\d+\:\d+)(.*)/" "cont" rule "cont" "/^\s+at.*/" "cont"
As you customize your regex pattern, we recommend you use a regular expression editor to test the expression.
-
Save the file as
parsers_multiline.conf
.
-
-
Within the
FluentBitDockerImage
folder, create a custom configuration file that references the parser file that you created in the previous step.For more information about the custom configuration file, see Specifying a custom configuration file in the Amazon Elastic Container Service Developer Guide
-
Paste the following contents in the file:
[SERVICE] flush 1 log_level info parsers_file /parsers_multiline.conf [FILTER] name multiline match * multiline.key_content log multiline.parser multiline-regex-test
Note
You must use the absolute path of the parser.
-
Save the file as
extra.conf
.
-
-
Within the
FluentBitDockerImage
folder, create the Dockerfile with the Fluent Bit image and the parser and configuration files that you created.-
Paste the following contents in the file:
FROM public.ecr.aws/aws-observability/aws-for-fluent-bit:latest ADD parsers_multiline.conf /parsers_multiline.conf ADD extra.conf /extra.conf
-
Save the file as
Dockerfile
.
-
-
Using the Dockerfile, build a custom Fluent Bit image with the parser and custom configuration files included.
Note
You can place the parser file and configuration file anywhere in the Docker image except
/fluent-bit/etc/fluent-bit.conf
as this file path is used by FireLens.-
Build the image:
docker build -t fluent-bit-multiline-image .
Where:
fluent-bit-multiline-image
is the name for the image in this example. -
Verify that the image was created correctly:
docker images —filter reference=fluent-bit-multiline-image
If successful, the output shows the image and the
latest
tag.
-
-
Upload the custom Fluent Bit image to Amazon Elastic Container Registry.
-
Create an Amazon ECR repository to store the image:
aws ecr create-repository --repository-name fluent-bit-multiline-repo --region us-east-1
Where:
fluent-bit-multiline-repo
is the name for the repository andus-east-1
is the region in this example.The output gives you the details of the new repository.
-
Tag your image with the
repositoryUri
value from the previous output:docker tag
fluent-bit-multiline-image
repositoryUri
Example:
docker tag fluent-bit-multiline-image xxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/fluent-bit-multiline-repo
-
Run the docker image to verify it ran correctly:
docker images —filter reference=
repositoryUri
In the output, the repository name changes from fluent-bit-multiline-repo to the
repositoryUri
. -
Authenticate to Amazon ECR by running the
aws ecr get-login-password
command and specifying the registry ID you want to authenticate to:aws ecr get-login-password | docker login --username AWS --password-stdin
registry ID
.dkr.ecr.region
.amazonaws.comExample:
ecr get-login-password | docker login --username AWS --password-stdin xxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com
A successful login message appears.
-
Push the image to Amazon ECR:
docker push
registry ID
.dkr.ecr.region
.amazonaws.com/repository name
Example:
docker push xxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/fluent-bit-multiline-repo
-
Build and upload the image for a demo multiline application
This image will include a Python script file that runs the application and a sample log file.
When you run the task, the application simulates runs, then fails and creates a stack trace.
-
Create a folder named
multiline-app
:mkdir multiline-app
-
Create a Python script file.
-
Within the
multiline-app
folder, create a file and name itmain.py
. -
Paste the following contents in the file:
import os import time file1 = open('/test.log', 'r') Lines = file1.readlines() count = 0 for i in range(10): print("app running normally...") time.sleep(1) # Strips the newline character for line in Lines: count += 1 print(line.rstrip()) print(count) print("app terminated.")
-
Save the
main.py
file.
-
-
Create a sample log file.
-
Within the
multiline-app
folder, create a file and name ittest.log
. -
Paste the following contents in the file:
single line... Dec 14 06:41:08 Exception in thread "main" java.lang.RuntimeException: Something has gone wrong, aborting! at com.myproject.module.MyProject.badMethod(MyProject.java:22) at com.myproject.module.MyProject.oneMoreMethod(MyProject.java:18) at com.myproject.module.MyProject.anotherMethod(MyProject.java:14) at com.myproject.module.MyProject.someMethod(MyProject.java:10) at com.myproject.module.MyProject.main(MyProject.java:6) another line...
-
Save the
test.log
file.
-
-
Within the
multiline-app
folder, create the Dockerfile.-
Paste the following contents in the file:
FROM public.ecr.aws/amazonlinux/amazonlinux:latest ADD test.log /test.log RUN yum upgrade -y && yum install -y python3 WORKDIR /usr/local/bin COPY main.py . CMD ["python3", "main.py"]
-
Save the
Dockerfile
file.
-
-
Using the Dockerfile, build an image.
-
Build the image:
docker build -t multiline-app-image .
Where:
multiline-app-image
is the name for the image in this example. -
Verify that the image was created correctly:
docker images —filter reference=multiline-app-image
If successful, the output shows the image and the
latest
tag.
-
-
Upload the image to Amazon Elastic Container Registry.
-
Create an Amazon ECR repository to store the image:
aws ecr create-repository --repository-name multiline-app-repo --region us-east-1
Where:
multiline-app-repo
is the name for the repository andus-east-1
is the region in this example.The output gives you the details of the new repository. Note the
repositoryUri
value as you will need it in the next steps. -
Tag your image with the
repositoryUri
value from the previous output:docker tag
multiline-app-image
repositoryUri
Example:
docker tag multiline-app-image xxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/multiline-app-repo
-
Run the docker image to verify it ran correctly:
docker images —filter reference=
repositoryUri
In the output, the repository name changes from
multiline-app-repo
to therepositoryUri
value. -
Push the image to Amazon ECR:
docker push
aws_account_id
.dkr.ecr.region
.amazonaws.com/repository name
Example:
docker push
xxxxxxxxxxxx
.dkr.ecr.us-east-1
.amazonaws.com/multiline-app-repo
-
Create the task definition and run the task
-
Create a task definition file with the file name
multiline-task-definition.json
. -
Paste the following contents in the
multiline-task-definition.json
file:{ "family": "firelens-example-multiline", "taskRoleArn": "
task role ARN
, "executionRoleArn": "execution role ARN
", "containerDefinitions": [ { "essential": true, "image": "aws_account_id
.dkr.ecr.us-east-1
.amazonaws.com/fluent-bit-multiline-image:latest", "name": "log_router", "firelensConfiguration": { "type": "fluentbit", "options": { "config-file-type": "file", "config-file-value": "/extra.conf" } }, "memoryReservation": 50 }, { "essential": true, "image": "aws_account_id
.dkr.ecr.us-east-1
.amazonaws.com/multiline-app-image:latest", "name": "app", "logConfiguration": { "logDriver": "awsfirelens", "options": { "Name": "cloudwatch_logs", "region": "us-east-1
", "log_group_name": "multiline-test/application", "auto_create_group": "true", "log_stream_prefix": "multiline-" } }, "memoryReservation": 100 } ], "requiresCompatibilities": ["FARGATE"], "networkMode": "awsvpc", "cpu": "256", "memory": "512" }Replace the following in the
multiline-task-definition.json
task definition:-
task role ARN
To find the task role ARN, go to the IAM console. Choose Roles and find the
ecs-task-role-for-firelens
task role that you created. Choose the role and copy the ARN that appears in the Summary section. -
execution role ARN
To find the execution role ARN, go to the IAM console. Choose Roles and find the
ecsTaskExecutionRole
role. Choose the role and copy the ARN that appears in the Summary section. -
aws_account_id
To find your
aws_account_id
, log into the AWS Management Console. Choose your user name on the top right and copy your Account ID. -
us-east-1
Replace the region if necessary.
-
-
Register the task definition file:
aws ecs register-task-definition --cli-input-json file://multiline-task-definition.json --region
region
Open the console at https://console.aws.amazon.com/ecs/v2
. -
In the navigation pane, choose Task Definitions and then choose the
firelens-example-multiline
family because we registered the task definition to this family in the first line of the task definition above. -
Choose the latest version.
-
Choose the Deploy, Run task.
-
On the Run Task page, For Cluster, choose the cluster, and then under Networking, for Subnets, choose the available subnets for your task.
-
Choose Create.
Verify that multiline log messages in Amazon CloudWatch appear concatenated
Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/
. -
From the navigation pane, expand Logs and choose Log groups.
-
Choose the
multiline-test/applicatio
log group. -
Choose the log. View messages. Lines that matched the rules in the parser file are concatenated and appear as a single message.
The following log snippet shows lines concatenated in a single Java stack trace event:
{ "container_id": "xxxxxx", "container_name": "app", "source": "stdout", "log": "Dec 14 06:41:08 Exception in thread \"main\" java.lang.RuntimeException: Something has gone wrong, aborting!\n at com.myproject.module.MyProject.badMethod(MyProject.java:22)\n at com.myproject.module.MyProject.oneMoreMethod(MyProject.java:18)\n at com.myproject.module.MyProject.anotherMethod(MyProject.java:14)\n at com.myproject.module.MyProject.someMethod(MyProject.java:10)\n at com.myproject.module.MyProject.main(MyProject.java:6)", "ecs_cluster": "default", "ecs_task_arn": "arn:aws:ecs:us-east-1:xxxxxxxxxxxx:task/default/xxxxxx", "ecs_task_definition": "firelens-example-multiline:2" }
The following log snippet shows how the same message appears with just a single line if you run an Amazon ECS container that is not configured to concatenate multiline log messages.
{ "log": "Dec 14 06:41:08 Exception in thread \"main\" java.lang.RuntimeException: Something has gone wrong, aborting!", "container_id": "xxxxxx-xxxxxx", "container_name": "app", "source": "stdout", "ecs_cluster": "default", "ecs_task_arn": "arn:aws:ecs:us-east-1:xxxxxxxxxxxx:task/default/xxxxxx", "ecs_task_definition": "firelens-example-multiline:3" }
Example: Use a Fluent Bit built-in parser
In this example, you will complete the following steps:
-
Build and upload the image for a Fluent Bit container.
-
Build and upload the image for a demo multiline application that runs, fails, and generates a multiline stack trace.
-
Create the task definition and run the task.
-
View the logs to verify that messages that span multiple lines appear concatenated.
Build and upload the image for a Fluent Bit container
This image will include a configuration file that references the Fluent Bit parser.
-
Create a folder with the name
FluentBitDockerImage
. -
Within the
FluentBitDockerImage
folder, create a custom configuration file that references the Fluent Bit built-in parser file.For more information about the custom configuration file, see Specifying a custom configuration file in the Amazon Elastic Container Service Developer Guide
-
Paste the following contents in the file:
[FILTER] name multiline match * multiline.key_content log multiline.parser go
-
Save the file as
extra.conf
.
-
-
Within the
FluentBitDockerImage
folder, create the Dockerfile with the Fluent Bit image and the parser and configuration files that you created.-
Paste the following contents in the file:
FROM public.ecr.aws/aws-observability/aws-for-fluent-bit:latest ADD extra.conf /extra.conf
-
Save the file as
Dockerfile
.
-
-
Using the Dockerfile, build a custom Fluent Bit image with the custom configuration file included.
Note
You can place the configuration file anywhere in the Docker image except
/fluent-bit/etc/fluent-bit.conf
as this file path is used by FireLens.-
Build the image:
docker build -t fluent-bit-multiline-image .
Where:
fluent-bit-multiline-image
is the name for the image in this example. -
Verify that the image was created correctly:
docker images —filter reference=fluent-bit-multiline-image
If successful, the output shows the image and the
latest
tag.
-
-
Upload the custom Fluent Bit image to Amazon Elastic Container Registry.
-
Create an Amazon ECR repository to store the image:
aws ecr create-repository --repository-name fluent-bit-multiline-repo --region us-east-1
Where:
fluent-bit-multiline-repo
is the name for the repository andus-east-1
is the region in this example.The output gives you the details of the new repository.
-
Tag your image with the
repositoryUri
value from the previous output:docker tag
fluent-bit-multiline-image
repositoryUri
Example:
docker tag fluent-bit-multiline-image xxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/fluent-bit-multiline-repo
-
Run the docker image to verify it ran correctly:
docker images —filter reference=
repositoryUri
In the output, the repository name changes from fluent-bit-multiline-repo to the
repositoryUri
. -
Authenticate to Amazon ECR by running the
aws ecr get-login-password
command and specifying the registry ID you want to authenticate to:aws ecr get-login-password | docker login --username AWS --password-stdin
registry ID
.dkr.ecr.region
.amazonaws.comExample:
ecr get-login-password | docker login --username AWS --password-stdin xxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com
A successful login message appears.
-
Push the image to Amazon ECR:
docker push
registry ID
.dkr.ecr.region
.amazonaws.com/repository name
Example:
docker push xxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/fluent-bit-multiline-repo
-
Build and upload the image for a demo multiline application
This image will include a Python script file that runs the application and a sample log file.
-
Create a folder named
multiline-app
:mkdir multiline-app
-
Create a Python script file.
-
Within the
multiline-app
folder, create a file and name itmain.py
. -
Paste the following contents in the file:
import os import time file1 = open('/test.log', 'r') Lines = file1.readlines() count = 0 for i in range(10): print("app running normally...") time.sleep(1) # Strips the newline character for line in Lines: count += 1 print(line.rstrip()) print(count) print("app terminated.")
-
Save the
main.py
file.
-
-
Create a sample log file.
-
Within the
multiline-app
folder, create a file and name ittest.log
. -
Paste the following contents in the file:
panic: my panic goroutine 4 [running]: panic(0x45cb40, 0x47ad70) /usr/local/go/src/runtime/panic.go:542 +0x46c fp=0xc42003f7b8 sp=0xc42003f710 pc=0x422f7c main.main.func1(0xc420024120) foo.go:6 +0x39 fp=0xc42003f7d8 sp=0xc42003f7b8 pc=0x451339 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc42003f7e0 sp=0xc42003f7d8 pc=0x44b4d1 created by main.main foo.go:5 +0x58 goroutine 1 [chan receive]: runtime.gopark(0x4739b8, 0xc420024178, 0x46fcd7, 0xc, 0xc420028e17, 0x3) /usr/local/go/src/runtime/proc.go:280 +0x12c fp=0xc420053e30 sp=0xc420053e00 pc=0x42503c runtime.goparkunlock(0xc420024178, 0x46fcd7, 0xc, 0x1000f010040c217, 0x3) /usr/local/go/src/runtime/proc.go:286 +0x5e fp=0xc420053e70 sp=0xc420053e30 pc=0x42512e runtime.chanrecv(0xc420024120, 0x0, 0xc420053f01, 0x4512d8) /usr/local/go/src/runtime/chan.go:506 +0x304 fp=0xc420053f20 sp=0xc420053e70 pc=0x4046b4 runtime.chanrecv1(0xc420024120, 0x0) /usr/local/go/src/runtime/chan.go:388 +0x2b fp=0xc420053f50 sp=0xc420053f20 pc=0x40439b main.main() foo.go:9 +0x6f fp=0xc420053f80 sp=0xc420053f50 pc=0x4512ef runtime.main() /usr/local/go/src/runtime/proc.go:185 +0x20d fp=0xc420053fe0 sp=0xc420053f80 pc=0x424bad runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc420053fe8 sp=0xc420053fe0 pc=0x44b4d1 goroutine 2 [force gc (idle)]: runtime.gopark(0x4739b8, 0x4ad720, 0x47001e, 0xf, 0x14, 0x1) /usr/local/go/src/runtime/proc.go:280 +0x12c fp=0xc42003e768 sp=0xc42003e738 pc=0x42503c runtime.goparkunlock(0x4ad720, 0x47001e, 0xf, 0xc420000114, 0x1) /usr/local/go/src/runtime/proc.go:286 +0x5e fp=0xc42003e7a8 sp=0xc42003e768 pc=0x42512e runtime.forcegchelper() /usr/local/go/src/runtime/proc.go:238 +0xcc fp=0xc42003e7e0 sp=0xc42003e7a8 pc=0x424e5c runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc42003e7e8 sp=0xc42003e7e0 pc=0x44b4d1 created by runtime.init.4 /usr/local/go/src/runtime/proc.go:227 +0x35 goroutine 3 [GC sweep wait]: runtime.gopark(0x4739b8, 0x4ad7e0, 0x46fdd2, 0xd, 0x419914, 0x1) /usr/local/go/src/runtime/proc.go:280 +0x12c fp=0xc42003ef60 sp=0xc42003ef30 pc=0x42503c runtime.goparkunlock(0x4ad7e0, 0x46fdd2, 0xd, 0x14, 0x1) /usr/local/go/src/runtime/proc.go:286 +0x5e fp=0xc42003efa0 sp=0xc42003ef60 pc=0x42512e runtime.bgsweep(0xc42001e150) /usr/local/go/src/runtime/mgcsweep.go:52 +0xa3 fp=0xc42003efd8 sp=0xc42003efa0 pc=0x419973 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc42003efe0 sp=0xc42003efd8 pc=0x44b4d1 created by runtime.gcenable /usr/local/go/src/runtime/mgc.go:216 +0x58 one more line, no multiline
-
Save the
test.log
file.
-
-
Within the
multiline-app
folder, create the Dockerfile.-
Paste the following contents in the file:
FROM public.ecr.aws/amazonlinux/amazonlinux:latest ADD test.log /test.log RUN yum upgrade -y && yum install -y python3 WORKDIR /usr/local/bin COPY main.py . CMD ["python3", "main.py"]
-
Save the
Dockerfile
file.
-
-
Using the Dockerfile, build an image.
-
Build the image:
docker build -t multiline-app-image .
Where:
multiline-app-image
is the name for the image in this example. -
Verify that the image was created correctly:
docker images —filter reference=multiline-app-image
If successful, the output shows the image and the
latest
tag.
-
-
Upload the image to Amazon Elastic Container Registry.
-
Create an Amazon ECR repository to store the image:
aws ecr create-repository --repository-name multiline-app-repo --region us-east-1
Where:
multiline-app-repo
is the name for the repository andus-east-1
is the region in this example.The output gives you the details of the new repository. Note the
repositoryUri
value as you will need it in the next steps. -
Tag your image with the
repositoryUri
value from the previous output:docker tag
multiline-app-image
repositoryUri
Example:
docker tag multiline-app-image xxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/multiline-app-repo
-
Run the docker image to verify it ran correctly:
docker images —filter reference=
repositoryUri
In the output, the repository name changes from
multiline-app-repo
to therepositoryUri
value. -
Push the image to Amazon ECR:
docker push
aws_account_id
.dkr.ecr.region
.amazonaws.com/repository name
Example:
docker push
xxxxxxxxxxxx
.dkr.ecr.us-east-1
.amazonaws.com/multiline-app-repo
-
Create the task definition and run the task
-
Create a task definition file with the file name
multiline-task-definition.json
. -
Paste the following contents in the
multiline-task-definition.json
file:{ "family": "firelens-example-multiline", "taskRoleArn": "
task role ARN
, "executionRoleArn": "execution role ARN
", "containerDefinitions": [ { "essential": true, "image": "aws_account_id
.dkr.ecr.us-east-1
.amazonaws.com/fluent-bit-multiline-image:latest", "name": "log_router", "firelensConfiguration": { "type": "fluentbit", "options": { "config-file-type": "file", "config-file-value": "/extra.conf" } }, "memoryReservation": 50 }, { "essential": true, "image": "aws_account_id
.dkr.ecr.us-east-1
.amazonaws.com/multiline-app-image:latest", "name": "app", "logConfiguration": { "logDriver": "awsfirelens", "options": { "Name": "cloudwatch_logs", "region": "us-east-1
", "log_group_name": "multiline-test/application", "auto_create_group": "true", "log_stream_prefix": "multiline-" } }, "memoryReservation": 100 } ], "requiresCompatibilities": ["FARGATE"], "networkMode": "awsvpc", "cpu": "256", "memory": "512" }Replace the following in the
multiline-task-definition.json
task definition:-
task role ARN
To find the task role ARN, go to the IAM console. Choose Roles and find the
ecs-task-role-for-firelens
task role that you created. Choose the role and copy the ARN that appears in the Summary section. -
execution role ARN
To find the execution role ARN, go to the IAM console. Choose Roles and find the
ecsTaskExecutionRole
role. Choose the role and copy the ARN that appears in the Summary section. -
aws_account_id
To find your
aws_account_id
, log into the AWS Management Console. Choose your user name on the top right and copy your Account ID. -
us-east-1
Replace the region if necessary.
-
-
Register the task definition file:
aws ecs register-task-definition --cli-input-json file://multiline-task-definition.json --region us-east-1
Open the console at https://console.aws.amazon.com/ecs/v2
. -
In the navigation pane, choose Task Definitions and then choose the
firelens-example-multiline
family because we registered the task definition to this family in the first line of the task definition above. -
Choose the latest version.
-
Choose the Deploy, Run task.
-
On the Run Task page, For Cluster, choose the cluster, and then under Networking, for Subnets, choose the available subnets for your task.
-
Choose Create.
Verify that multiline log messages in Amazon CloudWatch appear concatenated
Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/
. -
From the navigation pane, expand Logs and choose Log groups.
-
Choose the
multiline-test/applicatio
log group. -
Choose the log and view the messages. Lines that matched the rules in the parser file are concatenated and appear as a single message.
The following log snippet shows a Go stack trace that is concatenated into a single event:
{ "log": "panic: my panic\n\ngoroutine 4 [running]:\npanic(0x45cb40, 0x47ad70)\n /usr/local/go/src/runtime/panic.go:542 +0x46c fp=0xc42003f7b8 sp=0xc42003f710 pc=0x422f7c\nmain.main.func1(0xc420024120)\n foo.go:6 +0x39 fp=0xc42003f7d8 sp=0xc42003f7b8 pc=0x451339\nruntime.goexit()\n /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc42003f7e0 sp=0xc42003f7d8 pc=0x44b4d1\ncreated by main.main\n foo.go:5 +0x58\n\ngoroutine 1 [chan receive]:\nruntime.gopark(0x4739b8, 0xc420024178, 0x46fcd7, 0xc, 0xc420028e17, 0x3)\n /usr/local/go/src/runtime/proc.go:280 +0x12c fp=0xc420053e30 sp=0xc420053e00 pc=0x42503c\nruntime.goparkunlock(0xc420024178, 0x46fcd7, 0xc, 0x1000f010040c217, 0x3)\n /usr/local/go/src/runtime/proc.go:286 +0x5e fp=0xc420053e70 sp=0xc420053e30 pc=0x42512e\nruntime.chanrecv(0xc420024120, 0x0, 0xc420053f01, 0x4512d8)\n /usr/local/go/src/runtime/chan.go:506 +0x304 fp=0xc420053f20 sp=0xc420053e70 pc=0x4046b4\nruntime.chanrecv1(0xc420024120, 0x0)\n /usr/local/go/src/runtime/chan.go:388 +0x2b fp=0xc420053f50 sp=0xc420053f20 pc=0x40439b\nmain.main()\n foo.go:9 +0x6f fp=0xc420053f80 sp=0xc420053f50 pc=0x4512ef\nruntime.main()\n /usr/local/go/src/runtime/proc.go:185 +0x20d fp=0xc420053fe0 sp=0xc420053f80 pc=0x424bad\nruntime.goexit()\n /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc420053fe8 sp=0xc420053fe0 pc=0x44b4d1\n\ngoroutine 2 [force gc (idle)]:\nruntime.gopark(0x4739b8, 0x4ad720, 0x47001e, 0xf, 0x14, 0x1)\n /usr/local/go/src/runtime/proc.go:280 +0x12c fp=0xc42003e768 sp=0xc42003e738 pc=0x42503c\nruntime.goparkunlock(0x4ad720, 0x47001e, 0xf, 0xc420000114, 0x1)\n /usr/local/go/src/runtime/proc.go:286 +0x5e fp=0xc42003e7a8 sp=0xc42003e768 pc=0x42512e\nruntime.forcegchelper()\n /usr/local/go/src/runtime/proc.go:238 +0xcc fp=0xc42003e7e0 sp=0xc42003e7a8 pc=0x424e5c\nruntime.goexit()\n /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc42003e7e8 sp=0xc42003e7e0 pc=0x44b4d1\ncreated by runtime.init.4\n /usr/local/go/src/runtime/proc.go:227 +0x35\n\ngoroutine 3 [GC sweep wait]:\nruntime.gopark(0x4739b8, 0x4ad7e0, 0x46fdd2, 0xd, 0x419914, 0x1)\n /usr/local/go/src/runtime/proc.go:280 +0x12c fp=0xc42003ef60 sp=0xc42003ef30 pc=0x42503c\nruntime.goparkunlock(0x4ad7e0, 0x46fdd2, 0xd, 0x14, 0x1)\n /usr/local/go/src/runtime/proc.go:286 +0x5e fp=0xc42003efa0 sp=0xc42003ef60 pc=0x42512e\nruntime.bgsweep(0xc42001e150)\n /usr/local/go/src/runtime/mgcsweep.go:52 +0xa3 fp=0xc42003efd8 sp=0xc42003efa0 pc=0x419973\nruntime.goexit()\n /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc42003efe0 sp=0xc42003efd8 pc=0x44b4d1\ncreated by runtime.gcenable\n /usr/local/go/src/runtime/mgc.go:216 +0x58", "container_id": "xxxxxx-xxxxxx", "container_name": "app", "source": "stdout", "ecs_cluster": "default", "ecs_task_arn": "arn:aws:ecs:us-east-1:xxxxxxxxxxxx:task/default/xxxxxx", "ecs_task_definition": "firelens-example-multiline:2" }
The following log snippet shows how the same event appears if you run an ECS container that is not configured to concatenate multiline log messages. The log field contains a single line.
{ "log": "panic: my panic", "container_id": "xxxxxx-xxxxxx", "container_name": "app", "source": "stdout", "ecs_cluster": "default", "ecs_task_arn": "arn:aws:ecs:us-east-1:xxxxxxxxxxxx:task/default/xxxxxx", "ecs_task_definition": "firelens-example-multiline:3"
Note
If your logs go to log files instead of the standard output, we recommend
specifying the multiline.parser
and
multiline.key_content
configuration parameters in the Tail input plugin