Amazon Mechanical Turk
Developer Guide (API Version 2017-01-17)

Creating and Managing Qualifications

Qualifications and Qualification Requirements

A Qualification is a property of a Worker that represents a Worker's skill, ability or reputation. You can use Qualifications to control which Workers can perform your HITs. A HIT can have Qualification requirements that a Worker's Qualifications must meet before the Worker is allowed to accept the HIT. A requirement can also state that a Worker must meet the requirement to see the HIT's question data when previewing the HIT.

You can create and maintain your own Qualifications using the web service API. You create a Qualification type that describes what Qualifications of that type are for, and how Qualifications will be assigned.

A Qualification can have an optional value, such as a number. A HIT's Qualification requirement can specify a condition that the value must meet for the Qualification requirement to be met. A Worker can only accept a HIT if all of the HIT's Qualification requirements are met by the Worker's Qualifications.

You can use any Qualification type as the basis for your HIT's Qualification requirements, even if you did not create the type. For example, the Amazon Mechanical Turk system maintains a set of Qualification types that represent a Worker's account history, such as how many HITs the Worker has submitted in the lifetime of her account. You can create HIT requirements based on these statistics using the corresponding Qualification types.

You can create a new Qualification type by calling CreateQualificationType. Once a Qualification type has been created, it is available for use in a Qualification requirement, and can be searched or browsed at the Amazon Mechanical Turk website.

Tip

Any Requester can use any Qualification type for a Qualification requirement, regardless of who created the type. However, only the creator of the Qualification type can grant requests for the type and assign values.

Qualification Requests

A Worker discovers a Qualification type by browsing HITs that require a Qualification of that type, or by searching or browsing the Qualification types directly. The Worker can view a description of the type, and request a Qualification of the type from the type's creator.

By default, a Worker can only request a Qualification for a given type once. The creator of the type can allow a Worker to re-request a Qualification by specifying a value for the RetryDelayInSeconds parameter when creating the type. If set, the Worker must wait until the delay has elapsed after the first request before requesting the Qualification again.

A Qualification type may include an optional Qualification test, a form similar to a HIT that the Worker must complete to request the Qualification. The data entered into this form can be collected by your application, used to determine whether or not the Qualification request should be granted, or used to determine the Qualification's score.

Qualification requests can be granted in three different ways:

  • Your application can retrieve and process the requests for the Qualification. Your application downloads the Worker's answers for the Qualification test, and grants or rejects the request using the web service API.

  • You can include an answer key with the Qualification test, which specifies desired answers and corresponding scores. Amazon Mechanical Turk processes Qualification requests for types with answer keys automatically.

    Note

    If a Qualification test has an answer key, test answers are not available to your application, the requests are granted immediately. For more information on answer keys, see AnswerKey Data Structure.

  • You can specify that the Qualification type should grant every request for the Qualification automatically, using a default score. You could use this to set up Qualifications your application will adjust later, based on the Worker's performance. (Auto-granted Qualifications do not have tests, and the requests are granted immediately.)

Tip

If you have a prior relationship with a Worker and know the Worker's ID, you can assign Qualifications directly to the Worker without the Worker first making a request. See the following discussion.

Granting and Rejecting Qualification Requests

Your application can retrieve pending requests for your Qualification types using ListQualificationRequests. To grant the request and (optionally) assign the Qualification's value, call AcceptQualificationRequest. To reject the request, call RejectQualificationRequest.

Qualification Tests

A Qualification type may include a Qualification test, a set of questions similar to a HIT that a Worker must answer when requesting the Qualification. The test answers are returned with the request from a call to ListQualificationRequests. You can use the answers to determine the value to assign when granting the request.

The Qualification type may also include an answer key for the test, with answers and score values for each question in the test. In addition to score values for each matching answer, the answer key can specify how the Qualification value is calculated from the sum of the scores. For example, the Qualification value may be a percentage of correct answers.

Note

If a Qualification test has an answer key, the test may only contain multiple choice questions. An answer key cannot grade questions with free-text answers. For more information on answer keys, see AnswerKey Data Structure.

Assigning Qualifications Without Requests

If you have a prior relationship with a Worker, such as if the Worker has submitted results for your HITs in the past, and you know the Worker's ID, you can assign a Qualification directly to the Worker without the Worker first making a request. AssociateQualificationWithWorker gives a Worker a Qualification, and can send an email message to the Worker saying that the Qualification was assigned. You can also tell Amazon Mechanical Turk to not send an email message, and just assign the Qualification.

AssociateQualificationWithWorker is useful for managing Qualifications for people that already do work for you on a regular basis. If you wish to establish new Qualifications for Workers that have never done work for you in the past, you can use HITs without Qualification requirements, or requestable Qualifications, to establish a work history.

Updating and Retrieving Qualification Scores

After you have granted a Qualification to a Worker and the Worker has completed a HIT for you, you can update the Worker's Qualification score with UpdateQualificationScore. The operation requires the Worker's user ID, which is included in the assignment data the Worker submits for your HITs. You can use this operation to revoke a qualifying score if the Worker is not performing to expectations, or promote a score if the Worker has earned it with good results.

You can query a user's Qualification score for a type you created using GetQualificationScore.

Note

If your Qualification type has a Qualification test, an answer key, and allows test retakes, a Worker who has been granted the Qualification can take the test again to get a new score, even if you have updated the score since it was first granted. To query a user's current score, use the GetQualificationScore. You can configure the ability to retake a Qualification test when you create the Qualification type.

Revoking Qualifications

You can revoke the Qualification at any time by calling DisassociateQualificationFromWorker. A revoked Qualification behaves as if the Qualification were never granted: The Worker cannot qualify for HITs that require the Qualification be present. Also, calling the GetQualificationScore operation on a revoked Qualification will return an error.

A Worker may be able to request that a revoked Qualification be granted again if allowed by the Qualification type's retry policy. If the type does not have a retry delay, the Worker will not be able to request the Qualification a second time.

Querying All Qualifications For a Type

You can query all of the Qualifications you have granted for a type that you created using ListQualificationsForQualificationType. The operation returns the Worker ID and current Qualification score for every Qualification of the type that you have granted. You can request either all granted (active) Qualifications of the given type, or all revoked Qualifications of the given type.

Querying All HITs That Use a Qualification Type

You can query all of your HITs that use a given Qualification type in a Qualification requirement using ListHITsForQualificationType.

Note

You can only query your own HITs with ListHITsForQualificationType. The operation will not return HITs created by other Requesters that use a Qualification type.

You can query for HITs using any Qualification type, not only Qualification types that you created.

Updating a Qualification Type's Test

It is sometimes useful to make modifications to a Qualification type's test, to improve the accuracy of the resulting Qualification scores, or to replace old test questions with new ones. You can replace the test for a Qualification type using UpdateQualificationType.

UpdateQualificationType can also be used to replace an automatically graded test (with an answer key) with a manually graded one (without an answer key), and vice versa. It can also be used to change the test duration, the amount of time the Worker has to complete the test.

Deactivating a Qualification Type

Once you create a Qualification type, you are responsible for granting Qualification requests for the type (or having requests granted automatically using a Qualification test and answer key). You can temporarily deactivate a Qualification type by using UpdateQualificationType to update the status of the type. The type can be activated again with the same operation.

An inactive Qualification type cannot be requested by a Worker. It does not appear on the Amazon Mechanical Turk website, and does not appear in search results. While the type is inactive, a HIT that requires a Qualification of the type cannot be created. Workers with Qualifications of the type will continue to have those Qualifications, and will continue to qualify for existing HITs that require the Qualification.

Deleting a Qualification Type

You can permanently remove a Qualification type by using DeleteQualificationType. Once a Qualification type has been disposed, you cannot use it to create HITs or HIT types. It does not appear on the website and does not appear in search results. You can reuse the old Qualification type name and parameters to create a new Qualification type.

Note

A QualificationType may be auto-disposed if a HIT using the QualificationType in a QualificationRequirement has not been created for 120 days.

HITs that require a disposed Qualification type cannot be extended. If there are active HITs related to a disposed Qualification type, the HITs remain active and Workers who have qualified for that type can continue to work on those HITs. However, HIT types that use a disposed Qualification type can no longer be used to create new HITs.

Workers who have the Qualification of the disposed type keep those Qualifications, and will continue to be qualified for existing HITs that require the Qualification. If a Worker has a request pending to receive a Qualification for a disposed type, Amazon Mechanical Turk rejects the request.

System and Master Qualifications

Amazon Mechanical Turk provides a special set of Worker Qualifications available to all Requesters. System-assigned qualifications include approval rate and location. The system assigns these Qualifications to every Worker and updates their values as Workers use the system. For a list of system Qualifications and more information about locale-based Qualification requirements, go to the QualificationRequirement Data Structure, in the Amazon Mechanical Turk API Reference.

Mechanical Turk Masters

Mechanical Turk also provides Requesters access to Mechanical Turk Masters. Masters are an elite group of Workers who have demonstrated superior performance in completing HITs across the Mechanical Turk marketplace. Masters must maintain this high level of performance or they may lose this distinction.

For a more information about the Masters Qualification types, go to the QualificationRequirement Data Structure, in the Amazon Mechanical Turk API Reference.

Searching Qualification Types

ListQualificationTypes returns Qualification types whose names or descriptions match a search query. The results are similar to what is returned when performing such a search on the Amazon Mechanical Turk website or Requester Console, except that system Qualifications may also be included in results.