Assignment Review Policies

Assignment-level Review Policies are applied as soon as a Worker submits an assignment.

ScoreMyKnownAnswers/2011-09-01

ScoreMyKnownAnswers/2011-09-01 is an Assignment-level Review Policy.

Description

You can use the ScoreMyKnownAnswers/2011-09-01 Review Policy for QuestionForm (QAP) HITs and for ExternalQuestion (iframe) HITs. You provide an answer key when you call the CreateHIT operation. The answer key is a collection of QuestionIds, where each QuestionId has a set of zero or more values that represent the correct response for that QuestionId. For more information about QuestionForm and ExternalQuestion HITs, see QuestionForm and ExternalQuestion.

You can specify if one question in your HIT has a known answer or if many questions in your HIT have known answers. When a Worker submits an assignment Mechanical Turk examines the Worker's answers and compares them against the set of known answers that you provide when you create the HIT. Mechanical Turk then calculates a score, for example, 4 out of 10 known answers were correct.

Based on how the Worker's level of agreement with the known answers compares with various configurable thresholds, Mechanical Turk can automatically take actions you requested to approve the assignment, automatically reject the assignment, or automatically extend the HIT to publish an assignment for another Worker.

A Worker’s performance on known answers within a specific assignment are returned from calling the GetReviewResultsForHIT operation. You can get a Worker’s Life to Date (LTD), 7 day, and 30 day known answer statistics using the GetRequesterWorkerStatistic operation.

Mechanical Turk evaluates answers and considers the following answers as not matching:

• The Worker left an empty value set in the answer key.

• The answer key has an empty value set but the Worker supplied an answer.

• The Worker provides an answer that is the wrong case or has incorrect punctuation that doesn't match the answer exactly. You can either use structured HTML form elements to restrict the values a Worker can submit, or use JavaScript to validate and normalize the submitted values.

• The answer key says a question's answer is A and B but the Worker's value is A.

• The answer key says a question's answer is A and the Worker selected both A and B.

When comparing answers for a match, Mechanical Turk removes any whitespace from before and after the Worker's answer, and from before and after the answer you provide.

Parameters

The following parameters are specified in the AssignmentReviewPolicy element when calling the CreateHIT operation. You must also specify the PolicyName ScoreYourKnownAnswers/2011-09-01 as part of the AssignmentReviewPolicy element. For an example of how to structure the AssignmentReviewPolicy element, see the HIT Review Policy data structure.

Name	Description	Required
AnswerKey	Question IDs and the answers to the questions. Type: MapEntry, see the HIT Review Policy data structure. Default: None	Yes
ApproveIfKnownAnswerScoreIsAtLeast	Approve the assignment if the KnownAnswerScore is equal to or greater than this value. If not specified, assignments are left in the submitted state and are not approved or rejected. Type: Integer Constraints: Minimum value 0 (always approve), maximum 101 (never approve)	No
ApproveReason	A description provided to the Worker about the reason the assignment was approved. If not specified, the reason is left blank. Type: String	No
RejectIfKnownAnswerScoreIsLessThan	Reject the assignment if the KnownAnswerScore is equal to or less than this value. If not specified, assignments are left in the submitted state and are not approved or rejected. Type: Integer Constraints: Minimum value 0 (never reject), maximum 101 (always reject).	No
RejectReason	A description provided to the Worker about the reason the assignment was rejected. If not specified, the reason is left blank. Type: String	No
ExtendIfKnownAnswerScoreIsLessThan	Extend the HIT by one assignment to allow one more Worker to complete it if the known answer score is less than this value. Ordinarily this is done to replace an assignment that is being rejected or that is not usable because the Worker didn't answer the known answer correctly. If omitted the HIT is not extended. Type: String Constraint: Minimum value 0 (never extend), maximum 101 (always extend).	No
ExtendMaximumAssignments	The maximum number of assignments the HIT can be extended. Note that if you use the ExtendHIT operation and specify a maximum assignment count greater than this value, ScoreMyKnownAnswers will not extend the HIT. Note: If a HIT is created with fewer than 10 assignments, it will not extend to have 10 or more assignments. Type: Integer Constraint: Minimum value 2, maximum 25. Default: 5	No
ExtendMinimumTimeInSeconds	The additional time in seconds to let other Workers complete the extended assignment. Type: Integer Constraints: Minimum of 3600 (one hour), Maximum of 31536000 (one year). Default: 0	No