Example task definitions - Amazon ECS

Example task definitions

The following are some example task definitions demonstrating common custom log routing options. For more examples, see Amazon ECS FireLens examples on GitHub.

Forwarding logs to CloudWatch Logs

Note

For more examples, see Amazon ECS FireLens examples on GitHub.

The following task definition example demonstrates how to specify a log configuration that forwards logs to a CloudWatch Logs log group. For more information, see What Is Amazon CloudWatch Logs? in the Amazon CloudWatch Logs User Guide.

In the log configuration options, specify the log group name and the Region it exists in. To have Fluent Bit create the log group on your behalf, specify "auto_create_group":"true", to set the fluentd-buffer-limit use log-driver-buffer-limit. You can also specify the task ID as the log stream prefix, which assists in filtering. For more information, see Fluent Bit Plugin for CloudWatch Logs.

{ "family": "firelens-example-cloudwatch", "taskRoleArn": "arn:aws:iam::123456789012:role/ecs_task_iam_role", "containerDefinitions": [ { "essential": true, "image": "906394416424.dkr.ecr.us-west-2.amazonaws.com/aws-for-fluent-bit:latest", "name": "log_router", "firelensConfiguration": { "type": "fluentbit" }, "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "firelens-container", "awslogs-region": "us-west-2", "awslogs-create-group": "true", "awslogs-stream-prefix": "firelens" } }, "memoryReservation": 50 }, { "essential": true, "image": "httpd", "name": "app", "logConfiguration": { "logDriver":"awsfirelens", "options": { "Name": "cloudwatch", "region": "us-west-2", "log_group_name": "firelens-blog", "auto_create_group": "true", "log_stream_prefix": "from-fluent-bit", "log-driver-buffer-limit": "8192" } }, "memoryReservation": 100 } ] }

Forwarding logs to an Amazon Kinesis Data Firehose delivery stream

Note

For more examples, see Amazon ECS FireLens examples on GitHub.

The following task definition example demonstrates how to specify a log configuration that forwards logs to an Amazon Kinesis Data Firehose delivery stream. The Kinesis Data Firehose delivery stream must already exist. For more information, see Creating an Amazon Kinesis Data Firehose Delivery Stream in the Amazon Kinesis Data Firehose Developer Guide.

In the log configuration options, specify the delivery stream name and the Region it exists in. For more information, see Fluent Bit Plugin for Amazon Kinesis Firehose.

{ "family": "firelens-example-firehose", "taskRoleArn": "arn:aws:iam::123456789012:role/ecs_task_iam_role", "containerDefinitions": [ { "essential": true, "image": "906394416424.dkr.ecr.us-west-2.amazonaws.com/aws-for-fluent-bit:stable", "name": "log_router", "firelensConfiguration": { "type": "fluentbit" }, "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "firelens-container", "awslogs-region": "us-west-2", "awslogs-create-group": "true", "awslogs-stream-prefix": "firelens" } }, "memoryReservation": 50 }, { "essential": true, "image": "httpd", "name": "app", "logConfiguration": { "logDriver":"awsfirelens", "options": { "Name": "firehose", "region": "us-west-2", "delivery_stream": "my-stream" } }, "memoryReservation": 100 } ] }

Forwarding logs to an Amazon OpenSearch Service domain

Note

For more examples, see Amazon ECS FireLens examples on GitHub.

The following task definition example demonstrates how to specify a log configuration that forwards logs to an Amazon OpenSearch Service; domain. The Amazon OpenSearch Service domain must already exist. For more information, see What is Amazon OpenSearch Service in the Amazon OpenSearch Service Developer Guide.

In the log configuration options, specify the log options required for OpenSearch Service integration. For more information, see Fluent Bit for Amazon OpenSearch Service.

{ "family": "firelens-example-opensearch", "taskRoleArn": "arn:aws:iam::123456789012:role/ecs_task_iam_role", "containerDefinitions": [ { "essential": true, "image": "906394416424.dkr.ecr.us-west-2.amazonaws.com/aws-for-fluent-bit:stable", "name": "log_router", "firelensConfiguration": { "type": "fluentbit" }, "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "firelens-container", "awslogs-region": "us-west-2", "awslogs-create-group": "true", "awslogs-stream-prefix": "firelens" } }, "memoryReservation": 50 }, { "essential": true, "image": "httpd", "name": "app", "logConfiguration": { "logDriver": "awsfirelens", "options": { "Name": "es", "Host": "vpc-fake-domain-ke7thhzo07jawrhmz6mb7ite7y.us-west-2.es.amazonaws.com", "Port": "443", "Index": "my_index", "Type": "my_type", "AWS_Auth": "On", "AWS_Region": "us-west-2", "tls": "On" } }, "memoryReservation": 100 } ] }

Parsing container logs that are serialized JSON

Note

For more examples, see Amazon ECS FireLens examples on GitHub.

Beginning with AWS for Fluent Bit version 1.3, there is a JSON parser included in the AWS for Fluent Bit image. The following example shows how to reference the JSON parser in the FireLens configuration of your task definition.

"firelensConfiguration": { "type": "fluentbit", "options": { "config-file-type": "file", "config-file-value": "/fluent-bit/configs/parse-json.conf" } },

The Fluent Bit config file will parse any logs that are in JSON. For example, if the logs at your destination looked like the following without JSON parsing:

{ "source": "stdout", "log": "{\"requestID\": \"b5d716fca19a4252ad90e7b8ec7cc8d2\", \"requestInfo\": {\"ipAddress\": \"204.16.5.19\", \"path\": \"/activate\", \"user\": \"TheDoctor\"}}", "container_id": "e54cccfac2b87417f71877907f67879068420042828067ae0867e60a63529d35", "container_name": "/ecs-demo-6-container2-a4eafbb3d4c7f1e16e00" "ecs_cluster": "mycluster", "ecs_task_arn": "arn:aws:ecs:us-east-2:01234567891011:task/mycluster/3de392df-6bfa-470b-97ed-aa6f482cd7a6", "ecs_task_definition": "demo:7" "ec2_instance_id": "i-06bc83dbc2ac2fdf8" }

With the JSON parsing, the log will look like the following:

{ "source": "stdout", "container_id": "e54cccfac2b87417f71877907f67879068420042828067ae0867e60a63529d35", "container_name": "/ecs-demo-6-container2-a4eafbb3d4c7f1e16e00" "ecs_cluster": "mycluster", "ecs_task_arn": "arn:aws:ecs:us-east-2:01234567891011:task/mycluster/3de392df-6bfa-470b-97ed-aa6f482cd7a6", "ecs_task_definition": "demo:7" "ec2_instance_id": "i-06bc83dbc2ac2fdf8" "requestID": "b5d716fca19a4252ad90e7b8ec7cc8d2", "requestInfo": { "ipAddress": "204.16.5.19", "path": "/activate", "user": "TheDoctor" } }

The serialized JSON is expanded into top level fields in the final JSON output. For more information on JSON parsing, see Parser in the Fluent Bit documentation.

Forwarding to an external Fluentd or Fluent Bit

Note

For more examples, see Amazon ECS FireLens examples on GitHub.

The following task definition example demonstrates how to specify a log configuration that forwards logs to an external Fluentd or Fluent Bit host. Specify the host and port for your environment.

{ "family": "firelens-example-forward", "taskRoleArn": "arn:aws:iam::123456789012:role/ecs_task_iam_role", "containerDefinitions": [ { "essential": true, "image": "906394416424.dkr.ecr.us-west-2.amazonaws.com/aws-for-fluent-bit:stable", "name": "log_router", "firelensConfiguration": { "type": "fluentbit" }, "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "firelens-container", "awslogs-region": "us-west-2", "awslogs-create-group": "true", "awslogs-stream-prefix": "firelens" } }, "memoryReservation": 50 }, { "essential": true, "image": "httpd", "name": "app", "logConfiguration": { "logDriver":"awsfirelens", "options": { "Name": "forward", "Host": "fluentdhost", "Port": "24224" } }, "memoryReservation": 100 } ] }