Custom Log Routing
FireLens for Amazon ECS enables you to use task definition parameters to route logs to an AWS service or AWS Partner Network (APN) destination for log storage and analytics. FireLens works with Fluentd and Fluent Bit. We provide the AWS for Fluent Bit image or you can use your own Fluentd or Fluent Bit image.
Creating Amazon ECS task definitions with a FireLens configuration is supported using the AWS SDKs, AWS CLI, and AWS Management Console.
Considerations
The following should be considered when using FireLens for Amazon ECS:
-
FireLens for Amazon ECS is supported for tasks using both the Fargate and EC2 launch types.
-
For tasks that use the
bridgenetwork mode, the container with the FireLens configuration must start before any application containers that rely on it start. To control the start order of your containers, use dependency conditions in your task definition. For more information, see Container Dependency.Note
If you use dependency condition parameters in container definitions with a FireLens configuration, ensure that each container has a
STARTorHEALTHYcondition requirement.
Required IAM Permissions
To use this feature, you must create an IAM role for your tasks that provides the
permissions necessary to use any AWS services that the tasks require. For example,
if
a container is routing logs to Kinesis Data Firehose, then the task would require
permission to call the
firehose:PutRecordBatch API. For more information, see Adding and
Removing IAM Identity Permissions in the
IAM User Guide.
The following example IAM policy adds the required permissions for routing logs to Kinesis Data Firehose.
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "firehose:PutRecordBatch" ], "Resource": [ "*" ] } ] }
Your task may also require the Amazon ECS task execution role under the following conditions. For more information, see Amazon ECS Task Execution IAM Role.
-
If your task uses the Fargate launch type and you are pulling container images from Amazon ECR or referencing sensitive data from AWS Secrets Manager in your log configuration, then you must include the task execution IAM role.
-
If you are specifying a custom configuration file that is hosted in Amazon S3, your task execution IAM role must include the
s3:GetObjectpermission for the configuration file and thes3:GetBucketLocationpermission on the Amazon S3 bucket that the file is in. For more information, see Specifying Permissions in a Policy in the Amazon Simple Storage Service Console User Guide.The following example IAM policy adds the required permissions for retrieving a file from Amazon S3. Specify the name of your Amazon S3 bucket and configuration file name.
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:GetObject" ], "Resource": [ "arn:aws:s3:::examplebucket/folder_name/config_file_name" ] }, { "Effect": "Allow", "Action": [ "s3:GetBucketLocation" ], "Resource": [ "arn:aws:s3:::examplebucket" ] } ] }
Using the AWS for Fluent Bit Image
AWS provides a Fluent Bit image with plugins for both CloudWatch Logs and Kinesis Data Firehose. We recommend using Fluent Bit as your log router because it has a lower resource utilization rate than Fluentd. For more information, see CloudWatch Logs for Fluent Bit and Amazon Kinesis Firehose for Fluent Bit.
The AWS for Fluent Bit image is available on Docker Hub. However, we recommend that you use the following images in Amazon ECR because they provide higher availability.
| Region Name | Region | Image URI |
|---|---|---|
|
US East (N. Virginia) |
us-east-1 |
|
|
US East (Ohio) |
us-east-2 |
|
|
US West (N. California) |
us-west-1 |
|
|
US West (Oregon) |
us-west-2 |
|
|
Asia Pacific (Hong Kong) |
ap-east-1 |
|
|
Asia Pacific (Mumbai) |
ap-south-1 |
|
|
Asia Pacific (Seoul) |
ap-northeast-2 |
|
|
Asia Pacific (Singapore) |
ap-southeast-1 |
|
|
Asia Pacific (Sydney) |
ap-southeast-2 |
|
|
Asia Pacific (Tokyo) |
ap-northeast-1 |
|
|
Canada (Central) |
ca-central-1 |
|
|
EU (Frankfurt) |
eu-central-1 |
|
|
EU (Ireland) |
eu-west-1 |
|
|
EU (London) |
eu-west-2 |
|
|
EU (Paris) |
eu-west-3 |
|
|
EU (Stockholm) |
eu-north-1 |
|
|
Middle East (Bahrain) |
me-south-1 |
|
|
South America (Sao Paulo) |
sa-east-1 |
|
|
AWS GovCloud (US-East) |
us-gov-east-1 |
|
|
AWS GovCloud (US-West) |
us-gov-west-1 |
|
|
China (Beijing) |
cn-north-1 |
|
|
China (Ningxia) |
cn-northwest-1 |
|
Creating a Task Definition that Uses a FireLens Configuration
To use custom log routing with FireLens you must specify the following in your task definition:
-
A log router container containing a FireLens configuration. This container must be marked as essential.
-
One or more application containers that contain a log configuration specifying the
awsfirelenslog driver. -
A task IAM role ARN containing the permissions needed for the task to route the logs.
When creating a new task definition using the AWS Management Console, there is a FireLens integration section that makes it easy to add a log router container. For more information, see Creating a Task Definition.
Amazon ECS converts the log configuration and generates the Fluentd or Fluent Bit
output
configuration. The output configuration is mounted in the log routing container at
/fluent-bit/etc/fluent-bit.conf for Fluent Bit and
/fluentd/etc/fluent.conf for Fluentd.
To demonstrate how this works, the following is an example task definition example containing a log router container that uses Fluent Bit to route its logs to CloudWatch Logs and an application container that uses a log configuration to route logs to Amazon Kinesis Data Firehose.
{ "family": "firelens-example-firehose", "taskRoleArn": "arn:aws:iam::123456789012:role/ecs_task_iam_role", "containerDefinitions": [ { "essential": true, "image": "906394416424.dkr.ecr.us-west-2.amazonaws.com/aws-for-fluent-bit:latest", "name": "log_router", "firelensConfiguration": { "type": "fluentbit" }, "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "firelens-container", "awslogs-region": "us-west-2", "awslogs-create-group": "true", "awslogs-stream-prefix": "firelens" } }, "memoryReservation":50}, { "essential": true, "image": "httpd", "name": "app", "logConfiguration": { "logDriver":"awsfirelens", "options": { "Name": "firehose", "region": "us-west-2", "delivery_stream": "my-stream" } }, "memoryReservation":100} ] }
The
key-value
pairs specified as options in the logConfiguration object are used to
generate the Fluentd or Fluent Bit output configuration. The following is a code example
from a Fluent Bit output definition.
[OUTPUT] Name firehose Match app-firelens* regionus-west-2delivery_streammy-stream
Note
FireLens manages the match configuration. This configuration is not
specified in your task definition.
Using ECS Metadata
When specifying a FireLens configuration in a task definition, you can optionally
toggle the value for enable-ecs-log-metadata. By default, Amazon ECS adds
additional fields in your log entries that help identify the source of the logs. You
can disable this action by setting enable-ecs-log-metadata to
false.
-
ecs_cluster– The name of the cluster that the task is part of. -
ecs_task_arn– The full ARN of the task that the container is part of. -
ecs_task_definition– The task definition name and revision that the task is using. -
ec2_instance_id– The Amazon EC2 instance ID that the container is hosted on. This field is only valid for tasks using the EC2 launch type.
The following shows the syntax required when specifying an Amazon ECS log metadata setting value:
{ "containerDefinitions":[ { "essential":true, "image":"906394416424.dkr.ecr.us-west-2.amazonaws.com/aws-for-fluent-bit:latest", "name":"log_router", "firelensConfiguration":{ "type":"fluentbit", "options":{ "enable-ecs-log-metadata":"true|false" } } } ] }
Specifying a Custom Configuration File
In addition to the auto-generated configuration file that FireLens creates on your behalf, you can also specify a custom configuration file. The configuration file format is the native format for the log router you're using. For more information, see Fluentd Config File Syntax and Fluent Bit Configuration Schema.
In your custom configuration file, for tasks using the bridge or
awsvpc network mode, you should not set a Fluentd or Fluent Bit
forward input over TCP because FireLens will add it to the input
configuration.
Your FireLens configuration must contain the following options to specify a custom configuration file:
config-file-type-
The source location of the custom configuration file. The available options are
s3orfile. config-file-value-
The source for the custom configuration file. If the
s3config file type is used, the config file value is the full ARN of the Amazon S3 bucket and file. If thefileconfig file type is used, the config file value is the full path of the configuration file that exists either in the container image or on a volume that is mounted in the container.Important
When using a custom configuration file, you must specify a different path than the one FireLens uses. Amazon ECS reserves the
/fluent-bit/etc/fluent-bit.conffilepath for Fluent Bit and/fluentd/etc/fluent.conffor Fluentd.
The following example shows the syntax required when specifying a custom configuration.
Important
To specify a custom configuration file that is hosted in Amazon S3, ensure you have created a task execution IAM role with the proper permissions. For more information, see Required IAM Permissions.
The following shows the syntax required when specifying a custom configuration:
{ "containerDefinitions":[ { "essential":true, "image":"906394416424.dkr.ecr.us-west-2.amazonaws.com/aws-for-fluent-bit:latest", "name":"log_router", "firelensConfiguration":{ "type":"fluentbit", "options":{ "config-file-type":"s3|file", "config-file-value":"arn:aws:s3:::mybucket/fluent.conf|filepath" } } } ] }
Note
For tasks using the Fargate launch type, the only supported
config-file-type value is file.
Using Fluent Logger Libraries
When the awsfirelens log driver is specified in a task definition, the
ECS agent injects the following environment variables into the container:
FLUENT_HOST-
The IP address assigned to the FireLens container.
FLUENT_PORT-
The port that the Fluent Forward protocol is listening on.
The FLUENT_HOST and FLUENT_PORT environment variables enable
you to log directly to the log router from code instead of going through
stdout. For more information, see fluent-logger-golang on
GitHub.
Filtering Logs Using Regular Expressions
Fluentd and Fluent Bit both support filtering of logs based on their content. FireLens
provides a simple method for enabling this filtering. In the log configuration
options in a container definition, you can specify the special keys
exclude-pattern and include-pattern that take regular
expressions as their values. The exclude-pattern key causes all logs that
match its regular expression to be dropped. With include-pattern, only logs
that match its regular expression are sent. These keys can be used together.
The following example demonstrates how to use this filter.
{ "containerDefinitions":[ { "logConfiguration":{ "logDriver":"awsfirelens", "options":{ "@type":"cloudwatch_logs", "log_group_name":"firelens-testing", "auto_create_stream":"true", "use_tag_as_stream":"true", "region":"us-west-2", "exclude-pattern":"^[a-z][aeiou].*$", "include-pattern":"^.*[aeiou]$" } } } ] }
Example Task Definitions
The following are some example task definitions demonstrating common log routing options. For more examples, see Amazon ECS FireLens Examples on GitHub.
Topics
Forwarding Logs to CloudWatch Logs
The following task definition example demonstrates how to specify a log configuration that forwards logs to a CloudWatch Logs log group. For more information, see What Is Amazon CloudWatch Logs? in the Amazon CloudWatch Logs User Guide.
In the log configuration options, specify the log group name and the Region it
exists in. To have Fluent Bit create the log group on your behalf, specify
"auto_create_group":"true". You can also specify a log stream
prefix,
which assists in filtering. For more information, see Fluent Bit Plugin for CloudWatch Logs.
{ "family": "firelens-example-cloudwatch", "taskRoleArn": "arn:aws:iam::123456789012:role/ecs_task_iam_role", "containerDefinitions": [ { "essential": true, "image": "906394416424.dkr.ecr.us-west-2.amazonaws.com/aws-for-fluent-bit:latest", "name": "log_router", "firelensConfiguration": { "type": "fluentbit" }, "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "firelens-container", "awslogs-region": "us-west-2", "awslogs-create-group": "true", "awslogs-stream-prefix": "firelens" } }, "memoryReservation":50}, { "essential": true, "image": "nginx", "name": "app", "logConfiguration": { "logDriver":"awsfirelens", "options": { "Name": "cloudwatch", "region": "us-west-2", "log_group_name": "firelens-testing-fluent-bit", "auto_create_group": "true", "log_stream_prefix": "from-fluent-bit" } }, "memoryReservation":100} ] }
Forwarding Logs to an Amazon Kinesis Data Firehose Delivery Stream
The following task definition example demonstrates how to specify a log configuration that forwards logs to an Amazon Kinesis Data Firehose delivery stream. The Kinesis Data Firehose delivery stream must already exist. For more information, see Creating an Amazon Kinesis Data Firehose Delivery Stream in the Amazon Kinesis Data Firehose Developer Guide.
In the log configuration options, specify the delivery stream name and the Region it exists in. For more information, see Fluent Bit Plugin for Amazon Kinesis Firehose.
{ "family": "firelens-example-firehose", "taskRoleArn": "arn:aws:iam::123456789012:role/ecs_task_iam_role", "containerDefinitions": [ { "essential": true, "image": "906394416424.dkr.ecr.us-west-2.amazonaws.com/aws-for-fluent-bit:latest", "name": "log_router", "firelensConfiguration": { "type": "fluentbit" }, "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "firelens-container", "awslogs-region": "us-west-2", "awslogs-create-group": "true", "awslogs-stream-prefix": "firelens" } }, "memoryReservation":50}, { "essential": true, "image": "httpd", "name": "app", "logConfiguration": { "logDriver":"awsfirelens", "options": { "Name": "firehose", "region": "us-west-2", "delivery_stream": "my-stream" } }, "memoryReservation":100} ] }
Forwarding to an External Fluentd or Fluent Bit
The following task definition example demonstrates how to specify a log
configuration that forwards logs to an external Fluentd or Fluent Bit host. Specify
the host and port for your environment.
{ "family": "firelens-example-forward", "taskRoleArn": "arn:aws:iam::123456789012:role/ecs_task_iam_role", "containerDefinitions": [ { "essential": true, "image": "906394416424.dkr.ecr.us-west-2.amazonaws.com/aws-for-fluent-bit:latest", "name": "log_router", "firelensConfiguration": { "type": "fluentbit" }, "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "firelens-container", "awslogs-region": "us-west-2", "awslogs-create-group": "true", "awslogs-stream-prefix": "firelens" } }, "memoryReservation":50}, { "essential": true, "image": "httpd", "name": "app", "logConfiguration": { "logDriver":"awsfirelens", "options": { "Name": "forward", "Host": "fluentdhost", "Port": "24224" } }, "memoryReservation":100} ] }
