Monitor and Manage Your Human Loop - Amazon SageMaker

Monitor and Manage Your Human Loop

Once you've started a human review loop, you can check the results of and manage the loop using the Amazon Augmented AI Runtime API. Additionally, Amazon A2I integrates with Amazon EventBridge (also known as Amazon CloudWatch Events) to alert you when a human review loop changes status.

Use the procedures below to learn how to use the Amazon Augmented AI Runtime API to monitor and manage your human loops. See Use Amazon CloudWatch Events in Amazon Augmented AI to learn how Amazon A2I integrates with Amazon EventBridge.

To check your output data:

  1. Check the results of your human loop by calling the DescribeHumanLoop operation. The result of this API operation contains information about the reason for and outcome of the loop activation.

  2. Check the output data from your human loop in Amazon Simple Storage Service (Amazon S3). The path to the data uses the following pattern where YYYY/MM/DD/hh/mm/ss represents the human loop creation date with year (YYYY), month (MM) and day (DD) and the creation time with hour (hh), minute (mm) and second (ss).

    s3://customer-output-bucket-specified-in-flow-definition/flow-definition-name/YYYY/MM/DD/hh/mm/ss/human-loop-name/output.json

You can integrate this structure with AWS Glue or Amazon Athena to partition and analyze your output data. For more information, see Managing Partitions for ETL Output in AWS Glue.

In your output data, you can use the following to track individual workers:

  • The workerId is unique to each worker.

  • In workerMetadata, you will see the following.

    • identityProviderType – The service which Augmented AI uses to manage the private workforce.

    • issuer – The Cognito user pool associated with the work team assigned to this human review task.

    • sub – A unique identifier that refers to the worker. You can retrieve details about this worker (such as the name or username) using this ID using Amazon Cognito. To learn how, see Managing and Searching for User Accounts in Amazon Cognito Developer Guide.

"workerId": "a12b3cdefg4h5i67", "workerMetadata": { "identityData": { "identityProviderType": "Cognito", "issuer": "https://cognito-idp.aws-region.amazonaws.com/aws-region_123456789", "sub": "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee"

To stop and delete your human loop:

  1. Once a human loop has been started, you can stop your human loop by calling the StopHumanLoop operation using the HumanLoopName. If a human loop was successfully stopped, the server sends back an HTTP 200 response.

  2. To delete a human loop for which the status equals Failed, Completed, or Stopped, use the DeleteHumanLoop operation.

To list human loops:

  1. You can list all active human loops by calling the ListHumanLoops operation. You can filter human loops by the creation date of the loop using the CreationTimeAfter and CreateTimeBefore parameters.

  2. If successful, ListHumanLoops will return HumanLoopSummaries and NextToken objects in the response element. HumanLoopSummaries contains information about a single human loop. For example, it will list a loop's status and if applicable, failure reason.

    Use the string returned in NextToken as an input in a subsequent call to ListHumanLoops to see the next page of human loops.