Deploying Fluent Bit on Amazon ECS for Windows containers

Fluent Bit is a fast and flexible log processor and router supported by various operating systems. It can be used to route logs to various AWS destinations such as Amazon CloudWatch Logs, Firehose Amazon S3, and Amazon OpenSearch Service. Fluent Bit supports common partner solutions such as Datadog, Splunk, and custom HTTP servers. For more information about Fluent Bit, see the Fluent Bit website.

The AWS for Fluent Bit image is available on Amazon ECR on both the Amazon ECR Public Gallery and in an Amazon ECR repository in most Regions for high availability. For more information, see aws-for-fluent-bit on the GitHub website.

This tutorial walks you through how to deploy Fluent Bit containers on their Windows instances running in Amazon ECS to stream logs generated by the Windows tasks to Amazon CloudWatch for centralized logging.

This tutorial uses the following approach:

• Fluent Bit runs as a service with the Daemon scheduling strategy. This strategy ensures that a single instance of Fluent Bit always runs on the container instances in the cluster.

  • Listens on port 24224 using the forward input plug-in.

  • Expose port 24224 to the host so that the docker runtime can send logs to Fluent Bit using this exposed port.

  • Has a configuration which allows Fluent Bit to send the logs records to specified destinations.

• Launch all other Amazon ECS task containers using the fluentd logging driver. For more information, see Fluentd logging driver on the Docker documentation website.

  • Docker connects to the TCP socket 24224 on localhost inside the host namespace.

  • The Amazon ECS agent adds labels to the containers which includes the cluster name, task definition family name, task definition revision number, task ARN, and the container name. The same information is added to the log record using the labels option of the fluentd docker logging driver. For more information, see labels, labels-regex, env, and env-regex on the Docker documentation website.

  • Because the async option of the fluentd logging driver is set to true, when the Fluent Bit container is restarted, docker buffers the logs until the Fluent Bit container is restarted. You can increase the buffer limit by setting the fluentd-buffer-limit option. For more information, see fluentd-buffer-limit on the Docker documentation website.

The work flow is as follows:

• The Fluent Bit container starts and listens on port 24224 which is exposed to the host.

• Fluent Bit uses the task IAM role credentials specified in its task definition.

• Other tasks launched on the same instance use the fluentd docker logging driver to connect to the Fluent Bit container on port 24224.

• When the application containers generate logs, docker runtime tags those records, adds additional metadata specified in labels, and then forwards them on port 24224 in the host namespace.

• Fluent Bit receives the log record on port 24224 because it is exposed to the host namespace.

• Fluent Bit performs its internal processing and routes the logs as specified.

This tutorial uses the default CloudWatch Fluent Bit configuration which does the following:

• Creates a new log group for each cluster and task definition family.

• Creates a new log stream for each task container in above generated log group whenever a new task is launched. Each stream will be marked with the task id to which the container belongs.

• Adds additional metadata including the cluster name, task ARN, task container name, task definition family, and the task definition revision number in each log entry.

  For example, if you have task_1 with container_1 and container_2 and task_2 with container_3, then the following are the CloudWatch log streams:

  • /aws/ecs/windows.ecs_task_1

    task-out.TASK_ID.container_1

    task-out.TASK_ID.container_2

  • /aws/ecs/windows.ecs_task_2

    task-out.TASK_ID.container_3

Prerequisites

This tutorial assumes that the following prerequisites have been completed:

• The latest version of the AWS CLI is installed and configured. For more information, see Installing the AWS Command Line Interface.

• The aws-for-fluent-bit container image is available for the following Windows operating systems:

  • Windows Server 2019 Core

  • Windows Server 2019 Full

  • Windows Server 2022 Core

  • Windows Server 2022 Full

• The steps in Set up to use Amazon ECS have been completed.

• You have a cluster. In this tutorial, the cluster name is FluentBit-cluster.

• You have a VPC with a public subnet where the EC2 instance will be launched. You can use your default VPC. You can also use a private subnet that allows Amazon CloudWatch endpoints to reach the subnet. For more information about Amazon CloudWatch endpoints, see Amazon CloudWatch endpoints and quotas in the AWS General Reference. For information about how to use the Amazon VPC wizard to create a VPC, see Create a virtual private cloud.

Step 1: Create the IAM access roles

Create the Amazon ECS IAM roles.

1. Create the Amazon ECS container instance role named "ecsInstanceRole". For more information, see Amazon ECS container instance IAM role.

2. Create an IAM role for the Fluent Bit task named fluentTaskRole. For more information, see Amazon ECS task role.

   The IAM permissions granted in this IAM role are assumed by the task containers. In order to allow Fluent Bit to send logs to CloudWatch, you need to attach the following permissions to the task IAM role.

   {
       "Version": "2012-10-17",
       "Statement": [
       {
           "Effect": "Allow",
           "Action": [
               "logs:CreateLogStream",
               "logs:CreateLogGroup",
               "logs:DescribeLogStreams",
               "logs:PutLogEvents"
           ],
           "Resource": "*"
       }
       ]
   }

3. Attach the policy to the role.

   1. Save the above content in a file named fluent-bit-policy.json.

   2. Run the following command to attach the inline policy to fluentTaskRole IAM role.

      aws iam put-role-policy --role-name fluentTaskRole --policy-name fluentTaskPolicy --policy-document file://fluent-bit-policy.json

Step 2: Create an Amazon ECS Windows container instance

Create an Amazon ECS Windows container instance.

To create an Amazon ECS instance

1. Use the aws ssm get-parameters command to retrieve the AMI ID for the Region that hosts your VPC. For more information, see Retrieving Amazon ECS-Optimized AMI metadata.

2. Use the Amazon EC2 console to launch the instance.

   1. Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.

   2. From the navigation bar, select the Region to use.

   3. From the EC2 Dashboard, choose Launch instance.

   4. For Name, enter a unique name.

   5. For Application and OS Images (Amazon Machine Image), choose the AMI that you retrieved in the first step.

   6. For Instance type, choose t3.xlarge.

   7. For Key pair (login), choose a key pair.

   8. Under Network settings, for Security group, choose an existing security group, or create a new one.

   9. Under Network settings, for Auto-assign Public IP, select Enable.

   10. Under Advanced details, for IAM instance profile , choose ecsInstanceRole.

   11. Configure your Amazon ECS container instance with the following user data. Under Advanced Details, paste the following script into the User data field, replacing cluster_name with the name of your cluster.

       <powershell>
       Import-Module ECSTools
       Initialize-ECSAgent -Cluster cluster-name -EnableTaskENI -EnableTaskIAMRole -LoggingDrivers '["awslogs","fluentd"]'
       </powershell>

   12. When you are ready, select the acknowledgment field, and then choose Launch Instances.

   13. A confirmation page lets you know that your instance is launching. Choose View Instances to close the confirmation page and return to the console.

Step 3: Configure Fluent Bit

You can use the following default configuration provided by AWS to get quickly started:

• Amazon CloudWatch which is based on the Fluent Bit plug-in for Amazon CloudWatch on the Fluent Bit Official Manual.

Alternatively, you can use other default configurations provided by AWS. For more information, see Overriding the entrypoint for the Windows image on the aws-for-fluent-bit the Github website.

The default Amazon CloudWatch Fluent Bit configuration is shown below.

Replace the following variables:

• region with the Region where you want to send the Amazon CloudWatch logs.

[SERVICE]
    Flush               5
    Log_Level           info
    Daemon              off

[INPUT]
    Name                forward
    Listen              0.0.0.0
    Port                24224
    Buffer_Chunk_Size   1M
    Buffer_Max_Size     6M
    Tag_Prefix          ecs.

# Amazon ECS agent adds the following log keys as labels to the docker container.
# We would use fluentd logging driver to add these to log record while sending it to Fluent Bit.
[FILTER]
    Name                modify
    Match               ecs.*
    Rename              com.amazonaws.ecs.cluster ecs_cluster
    Rename              com.amazonaws.ecs.container-name ecs_container_name
    Rename              com.amazonaws.ecs.task-arn ecs_task_arn
    Rename              com.amazonaws.ecs.task-definition-family ecs_task_definition_family
    Rename              com.amazonaws.ecs.task-definition-version ecs_task_definition_version

[FILTER]
    Name                rewrite_tag
    Match               ecs.*
    Rule                $ecs_task_arn ^([a-z-:0-9]+)/([a-zA-Z0-9-_]+)/([a-z0-9]+)$  out.$3.$ecs_container_name false
    Emitter_Name        re_emitted

[OUTPUT]
    Name                cloudwatch_logs
    Match               out.*
    region              region
    log_group_name      fallback-group
    log_group_template  /aws/ecs/$ecs_cluster.$ecs_task_definition_family
    log_stream_prefix   task-
    auto_create_group   On

Every log which gets into Fluent Bit has a tag which you specify, or is automatically generated when you do not supply one. The tags can be used to route different logs to different destinations. For additional information, see Tag in the Fluent Bit Official Manual.

The Fluent Bit configuration described above has the following properties:

• The forward input plug-in listens for incoming traffic on TCP port 24224.

• Each log entry received on that port has a tag which the forward input plug-in modifies to prefix the record with ecs. string.

• The Fluent Bit internal pipeline routes the log entry to modify the filter using the Match regex. This filter replaces the keys in the log record JSON to the format which Fluent Bit can consume.

• The modified log entry is then consumed by the rewrite_tag filter. This filter changes the tag of the log record to the format out.TASK_ID.CONTAINER_NAME.

• The new tag will be routed to output cloudwatch_logs plug-in which creates the log groups and streams as described earlier by using the log_group_template and log_stream_prefix options of the CloudWatch output plug-in. For additional information, see Configuration parameters in the Fluent Bit Official Manual.

Step 4: Register a Windows Fluent Bit task definition which routes the logs to CloudWatch

Register a Windows Fluent Bit task definition which routes the logs to CloudWatch.

Note

This task definition exposes Fluent Bit container port 24224 to the host port 24224. Verify that this port is not open in your EC2 instance security group to prevent access from outside.

To register a task definition

1. Create a file named fluent-bit.json with the following contents.

   Replace the following variables:

   • task-iam-role with the Amazon Resource Name (ARN) of your task IAM role

   • region with the Region where your task runs

   {
     "family": "ecs-windows-fluent-bit",
     "taskRoleArn": "task-iam-role",
     "containerDefinitions": [
       {
         "name": "fluent-bit",
         "image": "public.ecr.aws/aws-observability/aws-for-fluent-bit:windowsservercore-latest",
         "cpu": 512,
         "portMappings": [
           {
             "hostPort": 24224,
             "containerPort": 24224,
             "protocol": "tcp"
           }
         ],
         "entryPoint": [
           "Powershell",
           "-Command"
         ],
         "command": [
           "C:\\entrypoint.ps1 -ConfigFile C:\\ecs_windows_forward_daemon\\cloudwatch.conf"
         ],
         "environment": [
           {
             "name": "AWS_REGION",
             "value": "region"
           }
         ],
         "memory": 512,
         "essential": true,
         "logConfiguration": {
           "logDriver": "awslogs",
           "options": {
             "awslogs-group": "/ecs/fluent-bit-logs",
             "awslogs-region": "region",
             "awslogs-stream-prefix": "flb",
             "awslogs-create-group": "true"
           }
         }
       }
     ],
     "memory": "512",
     "cpu": "512"
   }

2. Run the following command to register the task definition.

   aws ecs register-task-definition --cli-input-json file://fluent-bit.json --region region

   You can list the task definitions for your account by running the list-task-definitions command. The output of displays the family and revision values that you can use together with run-task or start-task.

Step 5: Run the ecs-windows-fluent-bit task definition as an Amazon ECS service using the daemon scheduling strategy

After you register a task definition for your account, you can run a task in the cluster. For this tutorial, you run one instance of the ecs-windows-fluent-bit:1 task definition in your FluentBit-cluster cluster. Run the task in a service which uses the daemon scheduling strategy, which ensures that a single instance of Fluent Bit always runs on each of your container instances.

To run a task

1. Run the following command to start the ecs-windows-fluent-bit:1 task definition (registered in the previous step) as a service.

   Note

   This task definition uses the awslogs logging driver, your container instance need to have the necessary permissions.

   Replace the following variables:

   • region with the Region where your service runs

   aws ecs create-service \
       --cluster FluentBit-cluster \
       --service-name FluentBitForwardDaemonService \
       --task-definition ecs-windows-fluent-bit:1 \
       --launch-type EC2 \
       --scheduling-strategy DAEMON \
       --region region

2. Run the following command to list your tasks.

   Replace the following variables:

   • region with the Region where your service tasks run

   aws ecs list-tasks --cluster FluentBit-cluster --region region

Step 6: Register a Windows task definition which generates the logs

Register a task definition which generates the logs. This task definition deploys Windows container image which will write a incremental number to stdout every second.

The task definition uses the fluentd logging driver which connects to port 24224 which the Fluent Bit plug-in listens to. The Amazon ECS agent labels each Amazon ECS container with tags including the cluster name, task ARN, task definition family name, task definition revision number and the task container name. These key-value labels are passed to Fluent Bit.

Note

This task uses the default network mode. However, you can also use the awsvpc network mode with the task.

To register a task definition

1. Create a file named windows-app-task.json with the following contents.

   {
     "family": "windows-app-task",
     "containerDefinitions": [
       {
         "name": "sample-container",
         "image": "mcr.microsoft.com/windows/servercore:ltsc2019",
         "cpu": 512,
         "memory": 512,
         "essential": true,
         "entryPoint": [
           "Powershell",
           "-Command"
         ],
         "command": [
           "$count=1;while(1) { Write-Host $count; sleep 1; $count=$count+1;}"
         ],
         "logConfiguration": {
           "logDriver": "fluentd",
           "options": {
             "fluentd-address": "localhost:24224",
             "tag": "{{ index .ContainerLabels \"com.amazonaws.ecs.task-definition-family\" }}",
             "fluentd-async": "true",
             "labels": "com.amazonaws.ecs.cluster,com.amazonaws.ecs.container-name,com.amazonaws.ecs.task-arn,com.amazonaws.ecs.task-definition-family,com.amazonaws.ecs.task-definition-version"
           }
         }
       }
     ],
     "memory": "512",
     "cpu": "512"
   }

2. Run the following command to register the task definition.

   Replace the following variables:

   • region with the Region where your task runs

   aws ecs register-task-definition --cli-input-json file://windows-app-task.json --region region

   You can list the task definitions for your account by running the list-task-definitions command. The output of displays the family and revision values that you can use together with run-task or start-task.

Step 7: Run the windows-app-task task definition

After you register the windows-app-task task definition, run it in your FluentBit-cluster cluster.

To run a task

1. Run the windows-app-task:1 task definition you registered in the previous step.

   Replace the following variables:

   • region with the Region where your task runs

   aws ecs run-task --cluster FluentBit-cluster --task-definition windows-app-task:1 --count 2 --region region

2. Run the following command to list your tasks.

   aws ecs list-tasks --cluster FluentBit-cluster

Step 8: Verify the logs on CloudWatch

In order to verify your Fluent Bit setup, check for the following log groups in the CloudWatch console:

• /ecs/fluent-bit-logs - This is the log group which corresponds to the Fluent Bit daemon container which is running on the container instance.

• /aws/ecs/FluentBit-cluster.windows-app-task - This is the log group which corresponds to all the tasks launched for windows-app-task task definition family inside FluentBit-cluster cluster.

  task-out.FIRST_TASK_ID.sample-container - This log stream contains all the logs generated by the first instance of the task in the sample-container task container.

  task-out.SECOND_TASK_ID.sample-container - This log stream contains all the logs generated by the second instance of the task in the sample-container task container.

The task-out.TASK_ID.sample-container log stream has fields similar to the following:

{
    "source": "stdout",
    "ecs_task_arn": "arn:aws:ecs:region:0123456789012:task/FluentBit-cluster/13EXAMPLE",
    "container_name": "/ecs-windows-app-task-1-sample-container-cEXAMPLE",
    "ecs_cluster": "FluentBit-cluster",
    "ecs_container_name": "sample-container",
    "ecs_task_definition_version": "1",
    "container_id": "61f5e6EXAMPLE",
    "log": "10",
    "ecs_task_definition_family": "windows-app-task"
}

To verify the Fluent Bit setup

1. Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/.

2. In the navigation pane, choose Log groups. Make sure that you're in the Region where you deployed Fluent Bit to your containers.

   In the list of log groups in the AWS Region, you should see the following:

   • /ecs/fluent-bit-logs

   • /aws/ecs/FluentBit-cluster.windows-app-task

   If you see these log groups, the Fluent Bit setup is verified.

Step 9: Clean up

When you have finished this tutorial, clean up the resources associated with it to avoid incurring charges for resources that you aren't using.

To clean up the tutorial resources

1. Stop the windows-simple-task task and the ecs-fluent-bit task. For more information, see Stop an Amazon ECS task.

2. Run the following command to delete the /ecs/fluent-bit-logs log group. For more information, about deleting log groups see delete-log-group in the AWS Command Line Interface Reference.

   aws logs delete-log-group --log-group-name /ecs/fluent-bit-logs
   aws logs delete-log-group --log-group-name /aws/ecs/FluentBit-cluster.windows-app-task

3. Run the following command to terminate the instance.

   aws ec2 terminate-instances --instance-ids instance-id

4. Run the following commands to delete the IAM roles.

   aws iam delete-role --role-name ecsInstanceRole
   aws iam delete-role --role-name fluentTaskRole

5. Run the following command to delete the Amazon ECS cluster.

   aws ecs delete-cluster --cluster FluentBit-cluster