Service event messages - Amazon ECS

Service event messages

When troubleshooting a problem with a service, the first place you should check for diagnostic information is the service event log. You can view service events using the DescribeServices API, the AWS CLI, or by using the AWS Management Console.

When viewing service event messages using the Amazon ECS API, only the events from the service scheduler are returned. These include the most recent task placement and instance health events. However, the Amazon ECS console displays service events from the following sources.

  • Task placement and instance health events from the Amazon ECS service scheduler. These events will have a prefix of service (service-name). To ensure that this event view is helpful, we only show the 100 most recent events and duplicate event messages are omitted until either the cause is resolved or six hours passes. If the cause is not resolved within six hours, you will receive another service event message for that cause.

  • Service Auto Scaling events. These events will have a prefix of Message. The 10 most recent scaling events are shown. These events only occur when a service is configured with an Application Auto Scaling scaling policy.

Use the following steps to view your current service event messages.

To view the service event log in the Amazon ECS console

  1. Open the Amazon ECS console at https://console.aws.amazon.com/ecs/.

  2. On the Clusters page, select the cluster in which your service resides.

  3. On the Cluster : clustername page, select the service to inspect.

  4. On the Service : servicename page, choose Events.

    
                            Service event messages
  5. Examine the Message column for errors or other helpful information.

Use the describe-services command to view the service event messages for a specified service.

The following AWS CLI example describes the service-name service in the default cluster, which will provide the latest service event messages.

aws ecs describe-services \ --cluster default \ --services service-name \ --region us-west-2

Service event messages

The following are examples of service event messages you may see in the Amazon ECS console.

This service contains tasks that have failed to start after consecutive attempts. At this point, the service scheduler begins to incrementally increase the time between retries. You should troubleshoot why your tasks are failing to launch. For more information, see Service throttle logic.

After the service is updated, for example with an updated task definition, the service scheduler resumes normal behavior.

This service is unable to launch more tasks due to API throttling limits. Once the service scheduler is able to launch more tasks, it will resume.

To request an API rate limit quota increase, open the AWS Support Center page, sign in if necessary, and choose Create case. Choose Service limit increase. Complete and submit the form.

This service is unable to stop or start tasks during a service deployment due to the deployment configuration. The deployment configuration consists of the minimumHealthyPercent and maximumPercent values which are defined when the service is created, but can also be updated on an existing service.

The minimumHealthyPercent represents the lower limit on the number of tasks that should be running for a service during a deployment or when a container instance is draining, as a percent of the desired number of tasks for the service. This value is rounded up. For example if the minimum healthy percent is 50 and the desired task count is four, then the scheduler can stop two existing tasks before starting two new tasks. Likewise, if the minimum healthy percent is 75% and the desired task count is two, then the scheduler can't stop any tasks due to the resulting value also being two.

The maximumPercent represents the upper limit on the number of tasks that should be running for a service during a deployment or when a container instance is draining, as a percent of the desired number of tasks for a service. This value is rounded down. For example if the maximum percent is 200 and the desired task is four then scheduler can start four new tasks before stopping four existing tasks. Likewise, if the maximum percent is 125 and the desired task count is three, the scheduler can't start any tasks due to the resulting value also being three.

When setting a minimum healthy percent or a maximum percent, you should ensure that the scheduler can stop or start at least one task when a deployment is triggered.

The follwing are the list of error messages:

  • This service is unable to place the task because you have reached the limit on the number of tasks.

  • This service is unable to place a task. Reason: The requested CPU configuration is above your limit.

  • This service is unable to place a task. Reason: The requested MEMORY configuration is above your limit.

You can request a quota increase for the resource that caused the error. For more information, see Amazon ECS service quotas. To request a quota increase, see Requesting a quota increase in the Service Quotas User Guide.