GetReviewResultsForHIT

Description

The GetReviewResultsForHIT operation retrieves the computed results and the actions taken in the course of executing your Review Policies during a CreateHIT operation. For information about how to apply Review Policies when you call CreateHIT, see Review Policies. The GetReviewResultsForHIT operation can return results for both Assignment-level and HIT-level review results. You can also specify to only return results pertaining to a particular Assignment.

Request Parameters

The GetReviewResultsForHIT operation accepts parameters common to all operations. Some common parameters are required. For more information, see Common Parameters.

The following parameters are specific to the GetReviewResultsForHIT operation:

Name	Description	Required
Operation	The name of the operation. Type: String Valid Values: GetReveiwResultsForHIT Default: None	Yes
HITId	The unique identifier of the HIT to retrieve review results for. Type: String Default: None	Yes
PolicyLevel	The Policy Level(s) to retrieve review results for. Type: String Default: HIT and Assignments Constraints: HIT and Assignment. If omitted, the default behavior is to retrieve all data for both policy levels. For a list of all the described policies, see Review Policies.	No
AssignmentId	If supplied, the results are limited to those pertaining directly to this Assignment ID. Type: String Default: None Note: Both the HIT-level and the Assignment-level Review Policies may take actions and return results concerning an Assignment, so both will be reported unless filtered by the PolicyLevel parameter.	No
RetrieveActions	Retrieves a list of the actions taken executing the Review Policies and their outcomes. Type: String Default: T Constraints: T or F	No
RetrieveResults	Retrieves a list of the results computed by the Review Policies. Type: String Default: T Constraints: T or F	No
PageSize	The number of results to include in a page of results. The complete results set is divided into pages of this many HITs. Type: positive integer Valid Values: any integer between 100 and 65535 Default: 400	Optional
PageNumber	The page of results to return. After the results are divided into pages of size PageSize, the operation returns the page corresponding to the PageNumber. Type: positive integer Default: 1	Optional

Response Elements

A successful request for the GetReviewResultsForHIT operation has a GetReviewResultsForHITResult element in the response. The GetReviewResultsForHITResult element contains the name of the Review Policy applied as well as the AssignmentReviewReport element and the HITReviewReport element. Additional elements called ReviewAction and ReviewResult provide the results of Review Policy specified when the HIT was created.

The GetReviewResultsForHITResult element contains the following elements:

Name	Description
HITId	The HIT ID. Type: String
AssignmentReviewPolicy	The name of the Assignment-level Review Policy. This contains only the PolicyName element. Type: String Constraint: Name of the applied policy. For a list of available named policies, see Review Policies.
HITReviewPolicy	The name of the HIT-level Review Policy. This contains only the PolicyName element. Type: String Constraint: Name of the applied policy. For a list of available named policies, see Review Policies.
AssignmentReviewReport	Contains both ReviewResult and ReviewAction elements for an Assignment. Type: ReviewResult data structure and ReviewAction data structure. See below for details.
HITReviewReport	Contains bothReviewResult and ReviewAction elements for a particular HIT. Type: ReviewResult data structure and ReviewAction data structure. See below for details.

ReviewResult Data Structure

Both the AssignmentReviewReport element and theHITReviewReport element contain the ReviewResult data structure. The ReviewResult structure is returned multiple times for each result specified in the Review Policy.

Note

A HIT-level Review Policy may be enacted multiple times if the HIT is extended by the Review Policy or by extending a reviewable HIT using the ExtendHIT operation.

GetReviewResultsForHITResult includes the results from each of the review steps, in chronological order. The first-produced results appear first in the list, while other results with the same SubjectId/QuestionId/Key may appear later in the list, reflecting later reviews. Assignment-level review policies are enabled precisely once per submitted Assignment, so the Assignment-level results will only contain duplicate SubjectId/QuestionId/Key elements if the Assignment-level policy contains information about the HIT itself.

ReviewResult Elements

The ReviewResult data structure contains the following elements:

Name	Description
ActionId	A unique identifier of the Review action result. Type: String
SubjectId	The HITID or AssignmentId about which this result was taken. Note that HIT-level Review Policies will often emit results about both the HIT itself and its Assignments, while Assignment-level review policies generally only emit results about the Assignment itself. Type: String
ObjectType	The type of the object from the SubjectId field. Type: String Constraint: Assignment or HIT
QuestionId	Specifies the QuestionId the result is describing. Depending on whether the ObjectType is a HIT or Assignment this results could specify multiple values. See the next table for possible values and their meaning. If ObjectType is HIT and QuestionId is absent, then the result describes results of the HIT, including the HIT agreement score. If ObjectType is Assignment and QuestionId is absent, then the result describes the Worker's performance on the HIT. Type: String
Key	Key identifies the particular piece of reviewed information. See the next table for details. Type: String
Value	The values of Key provided by the review policies you have selected. Type: String

Result Keys

The QuestionId element can contain the following result values:

Key	SubjectType	Meaning
AgreedAnswerFound	Question	A boolean to describe whether an agreed answer was found. Constraints: true or false
AgreedAnswer	Question	If AgreedAnswerFound is true, contains the agreed answer for the question. If no agreed answer is found then this value is not returned.
AnswerAgreementScore	Question	If AgreedAnswerFound is true, contains the Agreement Score for the question. If no agreed answer is found then this value is not returned.
WorkerAgreementScore	Assignment	Contains the computed Worker Agreement Score for the Worker that completed the assignment.
PluralityAnswersCorrect	Assignment	Contains the number of answers in the assignment that agreed with the majority.
PluralityAnswersIncorrect	Assignment	Contains the number of answers in the assignment that disagreed with the majority.
HitAgreementScore	HIT	Contains the computed HIT Agreement Score for the HIT.

ReviewAction Data Structure

Both the AssignmentReviewReport and the HITReviewReport elements contains the ReviewAction data structure. This structure is returned multiple times for each action specified in the Review Policy.

ReviewAction Elements

The ReviewAction data structure element contains the following elements:

Name	Description
ActionId	The unique identifier for the action. Type: String
ActionName	The nature of the action itself. The Review Policy is responsible for examining the HIT and Assignments, emitting results, and deciding which other actions will be necessary. Type: String Constraints: Approve, Reject, or Extend
ObjectId	The specific HITId or AssignmentID targeted by the action. Type: String Constraint: none
ObjectType	The type of object in ObjectId. Type: String Constraint: AssignmentID or HITID
Status	The current disposition of the action. See below for a description of the statuses. Type: String Constraints: INTENDED, SUCCEEDED, FAILED, or CANCELLED
CompleteTime	The date when the action was completed. Type: Date/Time Constraints: none
Results	A description of the outcome of the review. Note Do not to make assumptions about the format of this string or the data it contains. Type: String
ErrorCode	Present only when the Results have a FAILED Status. Type: String

ReviewAction Elements Returned Status Values

The following table lists the types of Status values returned in the ReviewAction Status element:

Status	Description
INTENDED	The action is either queued for execution or currently being worked on.
SUCCEEDED	The action was taken successfully. Note that the SUCCESS of an action does not necessarily imply that the object has actually been acted upon. For example, an action trying to approve an already-approved Assignment will report itself as SUCCESSFUL even though the Assignment was not further altered by the second approval. Similarly, if the policy was configured to extend a HIT to a maximum of 5 assignments, then an extend action attempted when the HIT has already been extended to have a maximum of 5 assignments will take no action and report SUCCESS, because it has successfully performed the action in accordance with the configured limits of the Review Policy.
FAILED	The action could not be taken because of the state of its object. For example, an Assignment that is already approved cannot later be rejected or a HIT cannot be extended if the Requester lacks sufficient funds to cover the increased liability. Whenever an action is "FAILED", an additional ErrorCode element is present.
CANCELLED	The action was prevented by the intervention of Mechanical Turk.

Examples

The following example shows how to use the GetReviewResultsForHIT operation.

Sample Request

The following is an example request.

https://mechanicalturk.amazonaws.com/?Service=AWSMechanicalTurkRequester
&AWSAccessKeyId=[the Requester's Access Key ID]
&Version=2011-XX-XX
&Operation=GetReviewResultsForHIT
&Signature=[signature for this request]
&Timestamp=[your system's local time]
&HITId=1AAAAAAAAABBBBBBBBBBCCCCCCCCCC				
			

Sample Response

The following is an example response.

<GetReviewResultsForHITResult>
  <HITId>1AAAAAAAAABBBBBBBBBBCCCCCCCCCC</HITId>
  <AssignmentReviewPolicy>
    <PolicyName>ScoreMyKnownAnswers/2011-09-01</PolicyName>
  </AssignmentReviewPolicy>
  <HITReviewPolicy>
    <PolicyName>SimplePlurality/2011-09-01</PolicyName>
  </HITReviewPolicy>
  <AssignmentReviewReport>
    <ReviewResult>
      <SubjectId>1DDDDDDDDDEEEEEEEEEEFFFFFFFFFF</SubjectId>
      <ObjectType>Assignment</ObjectType>
      <QuestionId>Question_2</QuestionId>
      <Key>KnownAnswerCorrect</Key>
      <Value>1</Value>
    </ReviewResult>
    <ReviewResult>
      <SubjectId>1DDDDDDDDDEEEEEEEEEEFFFFFFFFFF</SubjectId>
      <ObjectType>Assignment</ObjectType>
      <Key>KnownAnswerScore</Key>
      <Value>100</Value>
    </ReviewResult>
    <ReviewResult>
      <SubjectId>1GGGGGGGGGHHHHHHHHHHIIIIIIIIII</SubjectId>
      <ObjectType>Assignment</ObjectType>
      <QuestionId>Question_2</QuestionId>
      <Key>KnownAnswerCorrect</Key>
      <Value>0</Value>
    </ReviewResult>
    <ReviewResult>
      <SubjectId>1GGGGGGGGGHHHHHHHHHHIIIIIIIIII</SubjectId>
      <ObjectType>Assignment</ObjectType>
      <Key>KnownAnswerScore</Key>
      <Value>0</Value>
    </ReviewResult>
    <ReviewAction>
      <ActionName>review</ActionName>
      <ObjectId>1DDDDDDDDDEEEEEEEEEEFFFFFFFFFF</ObjectId>
      <ObjectType>Assignment</ObjectType>
      <Status>SUCCEEDED</Status>
      <Result>Reviewed one known answer; 1/1 correct</Result>
    </ReviewAction>
    <ReviewAction>
      <ActionName>approve</ActionName>
      <ObjectId>1DDDDDDDDDEEEEEEEEEEFFFFFFFFFF</ObjectId>
      <ObjectType>Assignment</ObjectType>
      <Status>SUCCEEDED</Status>
      <Result>Approved</Result>
    </ReviewAction>
    <ReviewAction>
      <ActionName>review</ActionName>
      <ObjectId>1GGGGGGGGGHHHHHHHHHHIIIIIIIIII</ObjectId>
      <ObjectType>Assignment</ObjectType>
      <Status>SUCCEEDED</Status>
      <Result>Reviewed one known answer; 0/1 correct</Result>
    </ReviewAction>
    <ReviewAction>
      <ActionName>reject</ActionName>
      <ObjectId>1GGGGGGGGGHHHHHHHHHHIIIIIIIIII</ObjectId>
      <ObjectType>Assignment</ObjectType>
      <Status>SUCCEEDED</Status>
      <Result>Rejected</Result>
    </ReviewAction>
  </AssignmentReviewReport>
  <HITReviewReport>
    <ReviewResult>
      <SubjectId>1DDDDDDDDDEEEEEEEEEEFFFFFFFFFF</SubjectId>
      <ObjectType>Assignment</ObjectType>
      <QuestionId>Question_1</QuestionId>
      <Key>AgreedWithPlurality</Key>
      <Value>1</Value>
    </ReviewResult>
    <ReviewResult>
      <SubjectId>1GGGGGGGGGHHHHHHHHHHIIIIIIIIII</SubjectId>
      <ObjectType>Assignment</ObjectType>
      <QuestionId>Question_1</QuestionId>
      <Key>AgreedWithPlurality</Key>
      <Value>1</Value>
    </ReviewResult>
    <ReviewResult>
      <SubjectId>1AAAAAAAAABBBBBBBBBBCCCCCCCCCC</SubjectId>
      <ObjectType>HIT</ObjectType>
      <QuestionId>Question_1</QuestionId>
      <Key>PluralityAnswer</Key>
      <Value>true</Value>
    </ReviewResult>
    <ReviewResult>
      <SubjectId>1AAAAAAAAABBBBBBBBBBCCCCCCCCCC</SubjectId>
      <ObjectType>HIT</ObjectType>
      <QuestionId>Question_1</QuestionId>
      <Key>PluralityLevel</Key>
      <Value>100</Value>
    </ReviewResult>
    <ReviewAction>
      <ActionName>approve</ActionName>
      <ObjectId>1DDDDDDDDDEEEEEEEEEEFFFFFFFFFF</ObjectId>
      <ObjectType>Assignment</ObjectType>
      <Status>SUCCEEDED</Status>
      <Result>Already approved</Result>
    </ReviewAction>
    <ReviewAction>
      <ActionName>approve</ActionName>
      <ObjectId>1GGGGGGGGGHHHHHHHHHHIIIIIIIIII</ObjectId>
      <ObjectType>Assignment</ObjectType>
      <Status>FAILED</Status>
      <Result>Assignment was in an invalid state for this operation.</Result>
      <ErrorCode>AWS.MechanicalTurk.InvalidAssignmentState</ErrorCode>
    </ReviewAction>
  </HITReviewReport>
</GetReviewResultsForHITResult>