Monitor Labeling Job Status

To monitor the status of your labeling jobs, you can set up an Amazon CloudWatch Events (CloudWatch Events) rule for Amazon SageMaker Ground Truth (Ground Truth) to send an event to CloudWatch Events when a labeling job status changes to Completed, Failed, or Stopped or when a worker accepts, declines, submits, or returns a task.

Once you create a rule, you can add a target to it. CloudWatch Events uses this target to invoke another AWS service to process the event. For example, you can create a target using a Amazon Simple Notification Service (Amazon SNS) topic to send a notification to your email when a labeling job status changes.

Prerequisites:

To create a CloudWatch Events rule, you will need an AWS Identity and Access Management (IAM) role with an events.amazonaws.com trust policy attached. The following is an example of an events.amazonaws.com trust policy.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": [
          "events.amazonaws.com"
        ]
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Send Events to CloudWatch Events

To configure a CloudWatch Events rule to get status updates, or events, for your Ground Truth labeling jobs, use the AWS Command Line Interface (AWS CLI) put-rule command. You can filter events that are sent to your rule by status change. For example, you can create a rule that notifies you only if a labeling job status changes to Completed. When using the put-rule command, specify the following to receive labeling job statuses:

• \"source\":[\"aws.sagemaker\"]

• \"detail-type\":[\"SageMaker Ground Truth Labeling Job State Change\"]

To configure a CloudWatch Events rule to watch for all status changes, use the following command and replace the placeholder text. For example, replace "GTLabelingJobStateChanges" with a unique CloudWatch Events rule name and "arn:aws:iam::111122223333:role/MyRoleForThisRule" with the Amazon Resource Number (ARN) of an IAM role with an events.amazonaws.com trust policy attached.

aws events put-rule --name "GTLabelingJobStateChanges" 
    --event-pattern "{\"source\":[\"aws.sagemaker\"],\"detail-type\":[\"SageMaker Ground Truth Labeling Job State Change\"]}" 
    --role-arn "arn:aws:iam::111122223333:role/MyRoleForThisRule" 
    --region "region"

To filter by job status, use the \"detail\":{\"LabelingJobStatus\":[\"Status\"]}}" syntax. Valid values for Status are Completed, Failed, and Stopped.

The following example creates a CloudWatch Events rule that notifies you when a labeling job in us-west-2 (Oregon) changes to Completed.

aws events put-rule --name "LabelingJobCompleted" 
    --event-pattern "{\"source\":[\"aws.sagemaker\"],\"detail-type\":[\"SageMaker Ground Truth Labeling Job State Change\"], \"detail\":{\"LabelingJobStatus\":[\"Completed\"]}}"  
    --role-arn "arn:aws:iam::111122223333:role/MyRoleForThisRule" 
    --region us-west-2

The following example creates a CloudWatch Events rule that notifies you when a labeling job in us-east-1 (Virginia) changes to Completed or Failed.

aws events put-rule --name "LabelingJobCompletedOrFailed" 
    --event-pattern "{\"source\":[\"aws.sagemaker\"],\"detail-type\":[\"SageMaker Ground Truth Labeling Job State Change\"], \"detail\":{\"LabelingJobStatus\":[\"Completed\", \"Failed\"]}}"  
    --role-arn "arn:aws:iam::111122223333:role/MyRoleForThisRule" 
    --region us-east-1

To learn more about the put-rule request, see Event Patterns in CloudWatch Events in the Amazon CloudWatch Events User Guide.

Set Up a Target to Process Events

After you have created a rule, events similar to the following are sent to CloudWatch Events. In this example, the labeling job test-labeling-job's status changed to Completed.

{
    "version": "0",
    "id": "111e1111-11d1-111f-b111-1111b11dcb11",
    "detail-type": "SageMaker Ground Truth Labeling Job State Change",
    "source": "aws.sagemaker",
    "account": "111122223333",
    "time": "2018-10-06T12:26:13Z",
    "region": "us-east-1",
    "resources": [
        "arn:aws:sagemaker:us-east-1:111122223333:labeling-job/test-labeling-job"
    ],
    "detail": {      
        "LabelingJobStatus": "Completed"
    }
}

To process events, you need to set up a target. For example, if you want to receive an email when your labeling job status changes, use a procedure in Setting Up Amazon SNS Notifications in the Amazon CloudWatch User Guide to set up an Amazon SNS topic and subscribe your email to it. Once you have create a topic, you can use it to create a target.

To add a target to your CloudWatch Events rule

1. Open the CloudWatch console: https://console.aws.amazon.com/cloudwatch/home

2. In the navigation pane, choose Rules.

3. Choose the rule that you want to add a target to.

4. Choose Actions, and then choose Edit.

5. Under Targets, choose Add Target and choose the AWS service you want to act when a labeling job status change event is detected.

6. Configure your target. For instructions, see the topic for configuring a target in the AWS documentation for that service.

7. Choose Configure details.

8. For Name, enter a name and, optionally, provide details about the purpose of the rule in Description.

9. Make sure that the check box next to State is selected so that your rule is listed as Enabled.

10. Choose Update rule.

Labeling Job Expiration

If your labeling job is not completed after 30 days, it will expire. If your labeling job expires, you can chain the job to create a new labeling job that will only send unlabeled data to workers. For more information, and to learn how to create a labeling job using chaining, see Chaining labeling jobs.

Declining Tasks

Workers are able to decline tasks.

Workers decline a task if the instructions are not clear, input data is not displaying correctly, or if they encounter some other issue with the task. If the number of workers per dataset object (NumberOfHumanWorkersPerDataObject) decline the task, the data object is marked as expired and will not be sent to additional workers.