Custom log routing

FireLens for Amazon ECS enables you to use task definition parameters to route logs to an AWS service or AWS Partner Network (APN) destination for log storage and analytics. FireLens works with Fluentd and Fluent Bit. We provide the AWS for Fluent Bit image or you can use your own Fluentd or Fluent Bit image.

Creating Amazon ECS task definitions with a FireLens configuration is supported using the AWS SDKs, AWS CLI, and AWS Management Console.

Considerations

The following should be considered when using FireLens for Amazon ECS:

• FireLens for Amazon ECS is supported for tasks hosted on both AWS Fargate and Amazon EC2.

• FireLens for Amazon ECS is supported in AWS CloudFormation templates. For more information, see AWS::ECS::TaskDefinition FirelensConfiguration in the AWS CloudFormation User Guide

• For tasks that use the bridge network mode, the container with the FireLens configuration must start before any application containers that rely on it start. To control the start order of your containers, use dependency conditions in your task definition. For more information, see Container dependency.

  Note

  If you use dependency condition parameters in container definitions with a FireLens configuration, ensure that each container has a START or HEALTHY condition requirement.

Required IAM permissions

To use this feature, you must create an IAM role for your tasks that provides the permissions necessary to use any AWS services that the tasks require. For example, if a container is routing logs to Kinesis Data Firehose, then the task would require permission to call the firehose:PutRecordBatch API. For more information, see Adding and Removing IAM Identity Permissions in the IAM User Guide.

The following example IAM policy adds the required permissions for routing logs to Kinesis Data Firehose.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "firehose:PutRecordBatch"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}

Your task may also require the Amazon ECS task execution role under the following conditions. For more information, see Amazon ECS task execution IAM role.

• If your task is hosted on Fargate and you are pulling container images from Amazon ECR or referencing sensitive data from AWS Secrets Manager in your log configuration, then you must include the task execution IAM role.

• If you are specifying a custom configuration file that is hosted in Amazon S3, your task execution IAM role must include the s3:GetObject permission for the configuration file and the s3:GetBucketLocation permission on the Amazon S3 bucket that the file is in. For more information, see Specifying Permissions in a Policy in the Amazon Simple Storage Service Console User Guide.

  The following example IAM policy adds the required permissions for retrieving a file from Amazon S3. Specify the name of your Amazon S3 bucket and configuration file name.

  {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": [
          "s3:GetObject"
        ],
        "Resource": [
          "arn:aws:s3:::examplebucket/folder_name/config_file_name"
        ]
      },
      {
        "Effect": "Allow",
        "Action": [
          "s3:GetBucketLocation"
        ],
        "Resource": [
          "arn:aws:s3:::examplebucket"
        ]
      }
    ]
  }

Using Fluent logger libraries

When the awsfirelens log driver is specified in a task definition, the ECS agent injects the following environment variables into the container:

FLUENT_HOST

The IP address assigned to the FireLens container.

FLUENT_PORT

The port that the Fluent Forward protocol is listening on.

The FLUENT_HOST and FLUENT_PORT environment variables enable you to log directly to the log router from code instead of going through stdout. For more information, see fluent-logger-golang on GitHub.