Review Policies - Amazon Mechanical Turk


                WARNING

You are browsing the documentation for a deprecated version ('2014-08-15') of the Amazon Mechanical Turk Requester API. This version of the API will be deprecated and will be rendered unusable as of June 1st, 2019.

If you request against a legacy API version (https://docs.aws.amazon.com/AWSMechTurk/latest/AWSMturkAPI-legacy/Welcome.html) on or after June 1, 2019, you will receive the following response:

This Requester API is no longer supported. Please use the latest API using the official AWS SDK. https://aws.amazon.com/getting-started/tools-sdks

The latest version of our API ('2017-01-17') provides you with additional tool choices and enables you to select from nine AWS Software Development Kits (SDKs) that have been widely adopted across the AWS community. This API can be accessed using the following AWS SDKs: Python/Boto (Boto3), Javascript (NodeJS or Browser), Java, .NET, Go, Ruby, PHP or C++. This version also makes it easier for customers to connect MTurk with other AWS services like S3, Lambda, Step Functions, Lex, Polly, Rekognition, Amazon SageMaker, AWS Batch, EC2, and more.

This version also updates naming conventions used in the API and adopts the AWS standard of Signature Version 4 to authenticate requests securely. The API uses REST requests and no longer requires that developers be familiar with SOAP protocol. These changes make the MTurk API consistent with AWS APIs, simplifying the on-boarding process for both new and existing AWS developers. The legacy MTurk Command Line Tools and .NET, Java, Ruby, and Perl SDKs were marked as deprecated in January 2018. We will be deprecating the legacy APIs as of June 1, 2019.

If you are on a legacy API, you must migrate to the latest version of our API. You can find documentation for the latest API here and the AWS SDKs here. Please check whether you need to migrate and review the technical migration guide.

For support, contact requestor-apilegacydeprecation-support@amazon.com.

 

Review Policies

Using Amazon Mechanical Turk Review Policies you can evaluate Worker submissions against a defined set of criteria. You specify the Review Policy(s) that you want to use when you call the CreateHIT operation.

There are two types of Review Policies, Assignment-level and HIT-level:

  • An Assignment-level Review Policy is applied as soon as a Worker submits an assignment. For more information, see Assignment Review Policies.

  • A HIT-level Review Policy is applied when a HIT becomes reviewable. For more information, see HIT Review Policies.

You can select from a set of pre-defined Review Policies. One Review Policy leverages known answers or gold standards within a Human Intelligence Task (HIT) and has Mechanical Turk calculate a Worker’s performance on these known answers. You can specify an action for Mechanical Turk to take automatically based on Worker performance against the known answer.

Mechanical Turk has Review Policies that calculate consensus/agreement among multiple Workers performing the same HITs. For instance, you can specify a Review Policy that measures agreement on work items within the HIT and authorizes Mechanical Turk to keep asking additional Workers to work on the HIT, until a certain level of agreement is achieved. Once the required level of agreement is achieved, the results are returned to you for immediate use.

Review Policies that track Worker performance on your known answers and agreement with other Workers give you information you can use to manage your Workers. For more information about using Review Policies, see Review Policy Use Cases.