Review Policies

Using Amazon Mechanical Turk Review Policies you can evaluate Worker submissions against a defined set of criteria. You specify the Review Policy(s) that you want to use when you call the CreateHIT operation.

There are two types of Review Policies, Assignment-level and HIT-level:

• An Assignment-level Review Policy is applied as soon as a Worker submits an assignment. For more information, see Assignment Review Policies.

• A HIT-level Review Policy is applied when a HIT becomes reviewable. For more information, see HIT Review Policies.

You can select from a set of pre-defined Review Policies. One Review Policy leverages known answers or gold standards within a Human Intelligence Task (HIT) and has Mechanical Turk calculate a Worker’s performance on these known answers. You can specify an action for Mechanical Turk to take automatically based on Worker performance against the known answer.

Mechanical Turk has Review Policies that calculate consensus/agreement among multiple Workers performing the same HITs. For instance, you can specify a Review Policy that measures agreement on work items within the HIT and authorizes Mechanical Turk to keep asking additional Workers to work on the HIT, until a certain level of agreement is achieved. Once the required level of agreement is achieved, the results are returned to you for immediate use.

Review Policies that track Worker performance on your known answers and agreement with other Workers give you information you can use to manage your Workers. For more information about using Review Policies, see Review Policy Use Cases.