Review Policy Use Cases - Amazon Mechanical Turk
Photo Moderation Use Case – Single Worker with Known AnswersPhoto Moderation Use Case – Multiple Workers with AgreementCategorization and Tagging Use Case – Multiple Workers

Review Policy Use Cases

The following use cases show you how to apply ScoreYourKnownAnswers and SimplePlurality policies when you call the CreateHIT operation.

Photo Moderation Use Case – Single Worker with Known Answers

In this scenario, you want Workers to moderate photos and screen the photos for inappropriate content. You place 20 photos in a single HIT and 5 of the 20 photos are your known answers. You are using Master Workers and have created the HIT with only one initial assignment. You want to use the answers based on the Worker getting at least 4 of the 5 known answers (80% Answer Agreement Score) correct. If the first Worker does not meet the Answer Agreement score of 80%, then you want to extend the HIT to another Worker. But, in this scenario, you only want to extend the HIT to a maximum of three Workers.

Elements and Parameters

The following is a list of elements and parameters you need to specify in the CreateHIT operation to execute the above scenario and allow Mechanical Turk to automatically calculate the known answer score. Note that this CreateHIT example assumes you have already created a HIT Type.

Element Parameter Value

AssignmentReviewPolicy

PolicyName

ScoreMyKnownAnswers/2011/09/01

AssignmentReviewPolicy

AnswerKey

List of questionIDs and answers.

AssignmentReviewPolicy

ExtendIfKnownAnswerScoreIsLessThan

80

AssignmentReviewPolicy

ExtendMaximumAssignments

3

Examples

The following example shows how to use the above elements and parameters with the CreateHIT operation.

Sample CreateHIT Request

The following example shows a CreateHIT request.

<CreateHITRequest>
    <HITTypeId>T100CN9P324W00EXAMPLE</HITTypeId>
    <Question>[CDATA block or XML Entity encoded]</Question>
    <LifetimeInSeconds>604800</LifetimeInSeconds>
    <AssignmentReviewPolicy>
        <PolicyName>ScoreMyKnownAnswers/2011-09-01</PolicyName>
        <Parameter>
            <Key>AnswerKey</Key>
            <MapEntry>
                <Key>QuestionId3</Key>   <!—correct answer is “B” -->
                <Value>B</Value>
            </MapEntry
            <MapEntry>
                <Key>QuestionId7</Key>  <!—correct answer is “A” -->
                <Value>A</Value>
            </MapEntry>
            <MapEntry>
                <Key>QuestionId15</Key>   <!—correct answer is “F” -->
                <Value>F</Value>
            </MapEntry>
            <MapEntry>
                <Key>QuestionId17</Key>   <!—correct answer is “C” -->
                <Value>C</Value>
            </MapEntry>
            <MapEntry>
                <Key>QuestionId18</Key>   <!—correct answer is “A” -->
                <Value>A</Value>
            </MapEntry>
        </Parameter>
        <Parameter>
           <Key>ExtendIfKnownAnswerScoreIsLessThan</Key>
           <Value>80</Value>
        </Parameter>
        <Parameter>
           <Key>ExtendMaximumAssignments</Key>
           <Value>3</Value>
        </Parameter>
    </AssignmentReviewPolicy>
</CreateHITRequest>

Photo Moderation Use Case – Multiple Workers with Agreement

In this scenario, you want Workers to moderate photos and screen the photos for inappropriate content. You place 20 photos in a single HIT and 5 of the 20 photos are your known answers. You want to approve the assignment if the Worker completes at least 4 of the 5 known answers correct (at least 80% Answer Agreement Score).

You want 3 Workers to complete each HIT and you want to calculate the HIT Agreement Score for the 15 photos you don’t know the answer to. Also, you want to disregard the Worker's answer in the Agreement Score if they don't get 4 of 5 of the known answers correct.

Elements and Parameters

The following is a list of elements and parameters you need to specify in the CreateHIT operation to execute the above scenario and allow Mechanical Turk to automatically approve the assignments. Note that this CreateHIT example assumes you have already created a HIT Type.

Element Parameter Value

AssignmentReviewPolicy

PolicyName

ScoreMyKnownAnswers/2011/09/01

AssignmentReviewPolicy

Answer

List of questionIDs and answers.

AssignmentReviewPolicy

ApproveIfKnownAnswerScoreIsAtLeast

80

AssignmentReviewPolicy

ExtendIfKnownAnswerScoreIsLessThan

80

AssignmentReviewPolicy

ExtendMaximumAssignments

3

HITReviewPolicy

PolicyName

SimplePlurality/2011-09-01

HITReviewPolicy

QuestionIDs

Your list of 15 question IDs.

HITReviewPolicy

QuestionAgreementThreshold

100

HITReviewPolicy

DisregardAssignmentIfKnownAnswerScoreIsLessThan

80

Examples

The following example shows how to use the above elements and parameters with the CreateHIT operation.

Sample CreateHIT Request

The following example shows a CreateHIT request.

<CreateHITRequest>
    <HITTypeId>T100CN9P324W00EXAMPLE</HITTypeId>
    <Question>[CDATA block or XML Entity encoded]</Question>
    <LifetimeInSeconds>604800</LifetimeInSeconds>
    <AssignmentReviewPolicy>
        <PolicyName>ScoreMyKnownAnswers/2011-09-01</PolicyName>
        <Parameter>
            <Key>AnswerKey</Key>
            <MapEntry>
                <Key>QuestionId3</Key>   <!—correct answer is “B” -->
                <Value>B</Value>
            </MapEntry
            <MapEntry>
                <Key>QuestionId4</Key>  <!—correct answer is “A” -->
                <Value>A</Value>
            </MapEntry>
            <MapEntry>
                <Key>QuestionId13</Key>   <!—correct answer is “F” -->
                <Value>F</Value>
            </MapEntry>
            <MapEntry>
                <Key>QuestionId14</Key>   <!—correct answer is “C” -->
                <Value>C</Value>
            </MapEntry>
            <MapEntry>
                <Key>QuestionId19</Key>   <!—correct answer is “A” -->
                <Value>A</Value>
            </MapEntry>
         </Parameter>
        <Parameter>
           <Key>ApproveIfKnownAnswerScoreIsAtLeast</Key>
           <Value>80</Value>
        </Parameter>
        <Parameter>
           <Key>ExtendIfKnownAnswerScoreIsLessThan</Key>
           <Value>80</Value>
        </Parameter>
        <Parameter>
           <Key>ExtendMaximumAssignments</Key>
           <Value>3</Value>
        </Parameter>
    </AssignmentReviewPolicy>
    <HITReviewPolicy>
        <PolicyName>SimplePlurality/2011-09-01</PolicyName>
        <Parameter>
           <Key>QuestionIDs</Key>
           <Value>questionid1</Value>
           <Value>questionid2</Value>
           <Value>questionid5</Value>
           <Value>questionid6</Value>
           <Value>questionid7</Value>
                 ..... <! Add your additional 10 questionIDs for a total of 15 questions. Different from your known answer questionIDs.>
        </Parameter>
        <Parameter>
           <Key>QuestionAgreementThreshold</Key>
           <Value>100</Value>
        </Parameter>
        <Parameter>
           <Key>DisregardAssignmentIfKnownAnswerScoreIsLessThan</Key>
           <Value>80</Value>
        </Parameter>
    </HITReviewPolicy>
</CreateHITRequest>

Categorization and Tagging Use Case – Multiple Workers

In this scenario, you want Workers to categorize a product and provide multiple tags for the product in a HIT. You also want the Workers to be able to comment on your HIT and give you feedback.

You want to calculate the Answer Agreement Score for only the categorization question. If two Workers do not agree on the product categorization question, you want to extend the HIT to a third Worker. Also, you want to extend the assignment by an hour so the third Worker has time to work on the assignment.

Elements and Parameters

The following is a list of elements and parameters you need to specify in the CreateHIT operation to execute the above scenario and allow Mechanical Turk to automatically calculate agreement and approve or reject the assignments. Note that this CreateHIT example assumes you have already created a HIT Type.

Element Parameter Value

HITReviewPolicy

PolicyName

SimplePlurality/2011-09-01

HITReviewPolicy

QuestionIDs

questionID1

HITReviewPolicy

QuestionAgreementThreshold

100

HITReviewPolicy

ExtendMinimumTimeInSeconds

3600

HITReviewPolicy

ExtendMaximumAssignments

3

Examples

The following example shows how to use the above elements and parameters with the CreateHIT operation.

Sample CreateHIT Request

The following example shows a CreateHIT request.

<CreateHITRequest>
    <HITTypeId>T100CN9P324W00EXAMPLE</HITTypeId>
    <Question>[CDATA block or XML Entity encoded]</Question>
    <LifetimeInSeconds>604800</LifetimeInSeconds>
    <HITReviewPolicy>
        <PolicyName>SimplePlurality/2011-09-01</PolicyName>
        <Parameter>
           <Key>QuestionIDs</Key>
           <Value>questionID1</Value>
        </Parameter>
        <Parameter>
           <Key>QuestionAgreementThreshold</Key>
           <Value>100</Value>
        </Parameter>
        <Parameter>
           <Key>ExtendMaximumAssignments</Key>
           <Value>3</Value>
        </Parameter>
        <Parameter>
           <Key>ExtendMinimumTimeInSeconds</Key>
           <Value>3600</Value>
        </Parameter>
    </HITReviewPolicy>
</CreateHITRequest>