You are browsing the documentation for a deprecated version ('2014-08-15') of the Amazon Mechanical Turk Requester API. This version of the API will be deprecated and will be rendered unusable as of June 1st, 2019. If you request against a legacy API version (https://docs.aws.amazon.com/AWSMechTurk/latest/AWSMturkAPI-legacy/Welcome.html) on or after June 1, 2019, you will receive the following response:
The latest version of our API ('2017-01-17') provides you with additional tool choices and enables you to select from nine AWS Software Development Kits This version also updates naming conventions used in the API and adopts the AWS standard of Signature Version 4 to authenticate requests securely. The API uses REST requests and no longer requires that developers be familiar with SOAP protocol. These changes make the MTurk API consistent with AWS APIs, simplifying the on-boarding process for both new and existing AWS developers. The legacy MTurk Command Line Tools and .NET, Java, Ruby, and Perl SDKs were marked as deprecated in January 2018. We will be deprecating the legacy APIs as of June 1, 2019. If you are on a legacy API, you must migrate to the latest
version of our API. You can find documentation for the latest API here and the AWS SDKs here For support, contact requestor-apilegacydeprecation-support@amazon.com |
GetReviewResultsForHIT
Description
The GetReviewResultsForHIT
operation retrieves the computed results and
the actions taken in the course of executing your Review Policies during a CreateHIT operation. For information about
how to apply Review Policies when you call CreateHIT
, see Review Policies. The
GetReviewResultsForHIT
operation can return results for both
Assignment-level and HIT-level review results. You can also specify to only return
results pertaining to a particular Assignment.
Request Parameters
The GetReviewResultsForHIT
operation accepts parameters common to all
operations. Some common parameters are required. For more information, see Common Parameters.
The following parameters are specific to the GetReviewResultsForHIT
operation:
Name | Description | Required |
---|---|---|
|
The name of the operation. Type: String Valid Values: GetReveiwResultsForHIT Default: None |
Yes |
|
The unique identifier of the HIT to retrieve review results for. Type: String Default: None |
Yes |
|
The Policy Level(s) to retrieve review results for. Type: String Default: HIT and Assignments Constraints: HIT and Assignment. If omitted, the default behavior is to retrieve all data for both policy levels. For a list of all the described policies, see Review Policies. |
No |
|
If supplied, the results are limited to those pertaining directly to this Assignment ID. Type: String Default: None Note: Both the HIT-level and the Assignment-level Review Policies may take actions
and return results concerning an Assignment, so both will be
reported unless filtered by the |
No |
|
Retrieves a list of the actions taken executing the Review Policies and their outcomes. Type: String Default: T Constraints: T or F |
No |
|
Retrieves a list of the results computed by the Review Policies. Type: String Default: T Constraints: T or F |
No |
|
The number of results to include in a page of results. The complete results set is
divided into pages of this many Type: positive integer Valid Values: any integer between 100 and 65535 Default: 400 |
Optional |
|
The page of results to return. After the results are divided into pages of size
Type: positive integer Default: 1 |
Optional |
Response Elements
A successful request for the GetReviewResultsForHIT
operation has a
GetReviewResultsForHITResult
element in the response. The
GetReviewResultsForHITResult
element contains the name of the
Review Policy applied as well as the AssignmentReviewReport
element and the HITReviewReport
element. Additional elements
called ReviewAction
and ReviewResult
provide the results of Review Policy specified when the HIT was created.
The GetReviewResultsForHITResult
element contains the following
elements:
Name | Description |
---|---|
|
The HIT ID. Type: String |
|
The name of the Assignment-level Review Policy. This contains only the
Type: String Constraint: Name of the applied policy. For a list of available named policies, see Review Policies. |
|
The name of the HIT-level Review Policy. This contains only the
Type: String Constraint: Name of the applied policy. For a list of available named policies, see Review Policies. |
|
Contains both Type: |
|
Contains both Type: |
ReviewResult Data Structure
Both the AssignmentReviewReport
element and theHITReviewReport
element contain the
ReviewResult data structure. The ReviewResult structure is returned multiple times for
each result specified in the Review Policy.
Note
A HIT-level Review Policy may be enacted multiple times if the HIT is extended by the Review Policy or by extending a reviewable HIT using the ExtendHIT operation.
GetReviewResultsForHITResult
includes the results from each of the
review steps, in chronological order. The first-produced results appear first in the
list, while other results with the same SubjectId/QuestionId/Key may appear later in the
list, reflecting later reviews. Assignment-level review policies are enabled precisely
once per submitted Assignment, so the Assignment-level results will only contain
duplicate SubjectId/QuestionId/Key elements if the Assignment-level policy contains
information about the HIT itself.
ReviewResult Elements
The ReviewResult
data structure contains the following elements:
Name | Description |
---|---|
|
A unique identifier of the Review action result. Type: String |
|
The HITID or AssignmentId about which this result was taken. Note that HIT-level Review Policies will often emit results about both the HIT itself and its Assignments, while Assignment-level review policies generally only emit results about the Assignment itself. Type: String |
|
The type of the object from the SubjectId field. Type: String Constraint: Assignment or HIT |
|
Specifies the QuestionId the result is describing. Depending on whether the ObjectType is a HIT or Assignment this results could specify multiple values. See the next table for possible values and their meaning. If ObjectType is HIT and QuestionId is absent, then the result describes results of the HIT, including the HIT agreement score. If ObjectType is Assignment and QuestionId is absent, then the result describes the Worker's performance on the HIT. Type: String |
|
Key identifies the particular piece of reviewed information. See the next table for details. Type: String |
|
The values of Key provided by the review policies you have selected. Type: String |
Result Keys
The QuestionId
element can contain the following result values:
Key | SubjectType | Meaning |
---|---|---|
AgreedAnswerFound |
Question |
A boolean to describe whether an agreed answer was found. Constraints: true or false |
AgreedAnswer |
Question |
If AgreedAnswerFound is true, contains the agreed answer for the question. If no agreed answer is found then this value is not returned. |
AnswerAgreementScore |
Question |
If AgreedAnswerFound is true, contains the Agreement Score for the question. If no agreed answer is found then this value is not returned. |
WorkerAgreementScore |
Assignment |
Contains the computed Worker Agreement Score for the Worker that completed the assignment. |
PluralityAnswersCorrect |
Assignment |
Contains the number of answers in the assignment that agreed with the majority. |
PluralityAnswersIncorrect |
Assignment |
Contains the number of answers in the assignment that disagreed with the majority. |
HitAgreementScore |
HIT |
Contains the computed HIT Agreement Score for the HIT. |
ReviewAction Data Structure
Both the AssignmentReviewReport
and the
HITReviewReport
elements contains the
ReviewAction
data structure. This structure is returned
multiple times for each action specified in the Review Policy.
ReviewAction Elements
The ReviewAction
data structure element contains the following
elements:
Name | Description |
---|---|
|
The unique identifier for the action. Type: String |
|
The nature of the action itself. The Review Policy is responsible for examining the HIT and Assignments, emitting results, and deciding which other actions will be necessary. Type: String Constraints: Approve, Reject, or Extend |
|
The specific HITId or AssignmentID targeted by the action. Type: String Constraint: none |
|
The type of object in ObjectId. Type: String Constraint: AssignmentID or HITID |
|
The current disposition of the action. See below for a description of the statuses. Type: String Constraints: INTENDED, SUCCEEDED, FAILED, or CANCELLED |
|
The date when the action was completed. Type: Date/Time Constraints: none |
|
A description of the outcome of the review. NoteDo not to make assumptions about the format of this string or the data it contains. Type: String |
|
Present only when the Results have a FAILED Status. Type: String |
ReviewAction Elements Returned Status Values
The following table lists the types of Status values returned in
the ReviewAction
Status element:
Status | Description |
---|---|
INTENDED |
The action is either queued for execution or currently being worked on. |
SUCCEEDED |
The action was taken successfully. Note that the SUCCESS of an action does not necessarily imply that the object has actually been acted upon. For example, an action trying to approve an already-approved Assignment will report itself as SUCCESSFUL even though the Assignment was not further altered by the second approval. Similarly, if the policy was configured to extend a HIT to a maximum of 5 assignments, then an extend action attempted when the HIT has already been extended to have a maximum of 5 assignments will take no action and report SUCCESS, because it has successfully performed the action in accordance with the configured limits of the Review Policy. |
FAILED |
The action could not be taken because of the state of its object. For example, an Assignment that is already approved cannot later be rejected or a HIT cannot be extended if the Requester lacks sufficient funds to cover the increased liability. Whenever an action is "FAILED", an additional ErrorCode element is present. |
CANCELLED |
The action was prevented by the intervention of Mechanical Turk. |
Examples
The following example shows how to use the GetReviewResultsForHIT
operation.
Sample Request
The following is an example request.
https://mechanicalturk.amazonaws.com/?Service=AWSMechanicalTurkRequester &AWSAccessKeyId=[
the Requester's Access Key ID
] &Version=2011-XX-XX &Operation=GetReviewResultsForHIT &Signature=[signature for this request
] &Timestamp=[your system's local time
] &HITId=1AAAAAAAAABBBBBBBBBBCCCCCCCCCC
Sample Response
The following is an example response.
<GetReviewResultsForHITResult> <HITId>1AAAAAAAAABBBBBBBBBBCCCCCCCCCC</HITId> <AssignmentReviewPolicy> <PolicyName>ScoreMyKnownAnswers/2011-09-01</PolicyName> </AssignmentReviewPolicy> <HITReviewPolicy> <PolicyName>SimplePlurality/2011-09-01</PolicyName> </HITReviewPolicy> <AssignmentReviewReport> <ReviewResult> <SubjectId>1DDDDDDDDDEEEEEEEEEEFFFFFFFFFF</SubjectId> <ObjectType>Assignment</ObjectType> <QuestionId>Question_2</QuestionId> <Key>KnownAnswerCorrect</Key> <Value>1</Value> </ReviewResult> <ReviewResult> <SubjectId>1DDDDDDDDDEEEEEEEEEEFFFFFFFFFF</SubjectId> <ObjectType>Assignment</ObjectType> <Key>KnownAnswerScore</Key> <Value>100</Value> </ReviewResult> <ReviewResult> <SubjectId>1GGGGGGGGGHHHHHHHHHHIIIIIIIIII</SubjectId> <ObjectType>Assignment</ObjectType> <QuestionId>Question_2</QuestionId> <Key>KnownAnswerCorrect</Key> <Value>0</Value> </ReviewResult> <ReviewResult> <SubjectId>1GGGGGGGGGHHHHHHHHHHIIIIIIIIII</SubjectId> <ObjectType>Assignment</ObjectType> <Key>KnownAnswerScore</Key> <Value>0</Value> </ReviewResult> <ReviewAction> <ActionName>review</ActionName> <ObjectId>1DDDDDDDDDEEEEEEEEEEFFFFFFFFFF</ObjectId> <ObjectType>Assignment</ObjectType> <Status>SUCCEEDED</Status> <Result>Reviewed one known answer; 1/1 correct</Result> </ReviewAction> <ReviewAction> <ActionName>approve</ActionName> <ObjectId>1DDDDDDDDDEEEEEEEEEEFFFFFFFFFF</ObjectId> <ObjectType>Assignment</ObjectType> <Status>SUCCEEDED</Status> <Result>Approved</Result> </ReviewAction> <ReviewAction> <ActionName>review</ActionName> <ObjectId>1GGGGGGGGGHHHHHHHHHHIIIIIIIIII</ObjectId> <ObjectType>Assignment</ObjectType> <Status>SUCCEEDED</Status> <Result>Reviewed one known answer; 0/1 correct</Result> </ReviewAction> <ReviewAction> <ActionName>reject</ActionName> <ObjectId>1GGGGGGGGGHHHHHHHHHHIIIIIIIIII</ObjectId> <ObjectType>Assignment</ObjectType> <Status>SUCCEEDED</Status> <Result>Rejected</Result> </ReviewAction> </AssignmentReviewReport> <HITReviewReport> <ReviewResult> <SubjectId>1DDDDDDDDDEEEEEEEEEEFFFFFFFFFF</SubjectId> <ObjectType>Assignment</ObjectType> <QuestionId>Question_1</QuestionId> <Key>AgreedWithPlurality</Key> <Value>1</Value> </ReviewResult> <ReviewResult> <SubjectId>1GGGGGGGGGHHHHHHHHHHIIIIIIIIII</SubjectId> <ObjectType>Assignment</ObjectType> <QuestionId>Question_1</QuestionId> <Key>AgreedWithPlurality</Key> <Value>1</Value> </ReviewResult> <ReviewResult> <SubjectId>1AAAAAAAAABBBBBBBBBBCCCCCCCCCC</SubjectId> <ObjectType>HIT</ObjectType> <QuestionId>Question_1</QuestionId> <Key>PluralityAnswer</Key> <Value>true</Value> </ReviewResult> <ReviewResult> <SubjectId>1AAAAAAAAABBBBBBBBBBCCCCCCCCCC</SubjectId> <ObjectType>HIT</ObjectType> <QuestionId>Question_1</QuestionId> <Key>PluralityLevel</Key> <Value>100</Value> </ReviewResult> <ReviewAction> <ActionName>approve</ActionName> <ObjectId>1DDDDDDDDDEEEEEEEEEEFFFFFFFFFF</ObjectId> <ObjectType>Assignment</ObjectType> <Status>SUCCEEDED</Status> <Result>Already approved</Result> </ReviewAction> <ReviewAction> <ActionName>approve</ActionName> <ObjectId>1GGGGGGGGGHHHHHHHHHHIIIIIIIIII</ObjectId> <ObjectType>Assignment</ObjectType> <Status>FAILED</Status> <Result>Assignment was in an invalid state for this operation.</Result> <ErrorCode>AWS.MechanicalTurk.InvalidAssignmentState</ErrorCode> </ReviewAction> </HITReviewReport> </GetReviewResultsForHITResult>