|« PreviousNext »|
|Did this page help you? Yes | No | Tell us about it...|
When your application creates a HIT using the CreateHITOperation operation, the HIT becomes available for Workers to find on the Amazon Mechanical Turk website. If a Worker has Qualifications that meet the HIT's Qualification requirements, the Worker can preview the HIT, then select the "Accept HIT" button to begin work.
When a Worker accepts a HIT, Amazon Mechanical Turk creates an
assignment to track the work to completion. The assignment belongs
exclusively to the Worker, and guarantees that the Worker will be allowed to submit results
any time until the HIT's
AssignmentDurationInSeconds has elapsed,
and still be eligible for the reward.
By default, any Worker can accept a maximum of 10 assignments for your HITs. Once any Worker has accepted 10 of your assignments, they must complete one of the 10 assignments before Amazon Mechanical Turk lets them accept another one of your assignments.
By default, a HIT has at most one assignment. When a Worker accepts the HIT, an assignment is created, and the HIT is no longer available for other Workers to accept. If the Worker returns or abandons the HIT, the assignment is removed, and the HIT becomes available again.
A HIT can be created to accept multiple assignments by specifying a
MaxAssignments parameter greater than 1 to the CreateHITOperation operation. Such a HIT will remain available for Workers to
accept as long as the number of assignments, in progress or submitted, is less than
A HIT is only available for Workers to accept until the HIT's
LifetimeInSeconds elapses, from the time the HIT was created. Once this time
elapses, the HIT expires. Such a HIT is no longer available, even if the number of assignments
is less than
MaxAssignments. Workers with assignments in progress are
allowed to continue working on the assignments as long as the
AssignmentDurationInSeconds has not elapsed, even after the HIT expires. If a
Worker returns or abandons the HIT after the HIT has expired, the HIT is not made available to
You can retrieve a list of all of your HITs at any time using SearchHITs.
You can also retrieve the submitted assignments for a HIT at any time using GetAssignmentsForHIT. If your HIT has multiple assignments and has not expired, but some of the assignments have been submitted, GetAssignmentsForHIT will return the submitted assignments.
A Worker completes an assignment by entering values into the question form and selecting the "Submit HIT" button. The results are stored for later retrieval by your application. The Worker continues working on other HITs, or ends the session.
If a Worker decides not to complete a HIT after accepting it, the Worker may select the
"Return HIT" button. Unless the HIT has expired-- that is, unless the HIT's
LifetimeInSeconds has elapsed since the HIT was created-- the
returned HIT becomes available for another qualified Worker to accept. The returned assignment
ends, and is no longer accessible using the Requester API.
If a Worker does not submit results before the assignment's deadline-- that is,
before the HIT's
AssignmentDurationInSeconds has elapsed since the
Worker accepted the HIT-- the HIT is considered abandoned. The effect is similar to if the
Worker explicitly returned the HIT, except that abandonments and returns are tracked as
separate statistics in the Worker's profile.
A Worker's HIT return and abandonment rates are tracked by Amazon Mechanical Turk as system Qualifications. You can create HITs that use Qualification requirements based on these numbers. Abandonments and returns are tracked as separate Qualifications.
Normally, a HIT will remain available for Workers to accept as long as
MaxAssignments results have not been submitted and the HIT has not expired (the
LifetimeInSeconds has not elapsed).
You can cause a HIT to expire early by calling ForceExpireHIT. This has the same effect as the HIT expiring naturally: The HIT becomes no longer available for new Workers to accept. Assignments in progress are allowed to complete, either with the Worker submitting results, or the Worker returning or abandoning the assignment. After the HIT has expired, returned or abandoned assignments are not made available to other Workers.
ForceExpireHIT is your "stop" button. If you have submitted incorrect data for a HIT, or otherwise have changed your mind about the HIT, you can force it to expire to prevent Workers from completing it. You will still be responsible for approving assignments that have already been submitted, but ForceExpireHIT can minimize the damage.
When either all of a HIT's assignments have been submitted by Workers, or the HIT has expired and all assignments have either been submitted, returned or abandoned, the HIT is considered "reviewable." Once a HIT is reviewable, the Requester can retrieve and process the results.
GetReviewableHITs returns the HIT IDs for all of your HITs that are reviewable. GetAssignmentsForHIT takes the ID of a reviewable HIT and returns the completed assignments, with the answers submitted by Workers. Your application can use these operations to poll for and retrieve results for HITs.
Once results for an assignment have been retrieved, you approve or reject the assignment. Your application calls ApproveAssignment to approve the assignment, or RejectAssignment to reject the assignment.
A call to
ApproveAssignment will return an error if the
Requester's account does not have sufficient funds to pay the Worker and the listing
fee at the time of the call.
An assignment should always be approved, unless it is clear the Worker did not follow the instructions in the HIT. The method your application uses to decide if an assignment should be approved will depend on the nature of your application. For example, you may request 3 assignments for each HIT then check to see if the results match, and if they do, approve all 3 assignments. If they don't match, have those results checked by a human operator. In some cases, it may be appropriate to automatically approve all assignments, then control the quality of answers using Qualification requirements.
Every HIT has an
AutoApprovalDelayInSeconds, an amount of time
after an assignment is submitted at which point Amazon Mechanical Turk will automatically
approve the assignment if it has not already been explicitly approved or rejected. If not
specified in the call to CreateHITOperation , this is set to the maximum, equivalent to 30 days. The maximum
value assures that the Worker will get paid even if the Requester is unavailable. You may wish
to specify a shorter auto-approval delay if the Workers are trusted (such as by having been
vetted with Qualification requirements) and to assure Workers that they will be paid in a
When you call ApproveAssignment, Amazon Mechanical Turk automatically processes payment of the reward to the Worker, and payment of the HIT listing fee, using the money in your Amazon.com account. You will not be charged for the reward, nor for the listing fee, if the results are rejected.
When you approve or reject an assignment, you can include a
RequesterFeedback message, a string that the Worker can see on his or her HIT
status screen. Including feedback message when rejecting a result may improve the quality of
the results the Worker submits for your other HITs.
When a HIT becomes reviewable, its assignments are ready for your application to retrieve and process. Your application can approve or reject assignments, then either dispose of the HIT, or extend it to collect more assignments. GetReviewableHITs returns the IDs of your HITs that have the "reviewable" status.
If your application does not want to make an immediate decision about approving or rejecting assignments, or disposing of or extending the HIT, the application can promote the status of the HIT to the "reviewing" status. "Reviewing" HITs are not returned by a call to GetReviewableHITs by default, so your application can continue to poll for "reviewable" HITs while "reviewing" HITs are awaiting processing. You can promote a HIT to the "reviewing" status using SetHITAsReviewing.
The "reviewing" status is useful for answer validation techniques that require waiting for additional information to approve an answer. For example, you can create a "verification" HIT based on the answers for a completed HIT that asks a Worker to compare the submitted answers and verify that they meet the HIT's requirements. Your application can give the original completed HIT a status of "reviewing" while it waits for the verification HIT to be completed, and continue to poll for other HITs as they become "reviewable".
Since your task HITs and your verification HITs will have different HIT types, your application can poll for task and verification results separately by passing the appropriate HIT type IDs to GetReviewableHITs.
You can retrieve a list of IDs of HITs with the "reviewing" status by calling
GetReviewableHITs with an appropriate value for the
StatusMatchOption parameter. You can revert a "reviewing" HIT back to
the "reviewable" status by calling
SetHITAsReviewing with an
appropriate value for the
When you approve an assignment a Worker has submitted, Amazon Mechanical Turk automatically processes the payment of the HIT's reward from your account to the Worker's account. By approving the assignment, you pay the Worker the amount of money specified in the original HIT.
In addition to the HIT reward, you can pay the Worker a "bonus" amount of money by calling GrantBonus. You might pay a bonus to reward Workers that submit high quality results. You could use the promise of a bonus payment to encourage Workers to complete your HITs and also to develop a following of Workers.
You can grant a bonus to any Worker who has submitted an assignment for one of your HITs any time after you approve or reject the assignment up until the HIT has been disposed.
Amazon Mechanical Turk collects a fee for bonus payments separate from (and in addition to) the HIT listing fee. The charge for awarding a bonus (currently) is 10% of the bonus with a minimum charge of $0.005. For example, if a Worker has a $0.01 assignment and is awarded a $0.01 bonus, the Requester pays the Worker $0.02 ($0.01 assignment+$0.01 bonus) and MTurk $0.01 ($0.005 + $0.005). Or, for example, if the Worker has a $1 assignment and is awarded a $1 bonus, the Requester pays the Worker $2 and MTurk $0.20.
For more information about Amazon Mechanical Turk pricing, go to the Amazon Mechanical Turk website at Amazon Web Services.
There is no way to pay a Worker for completing a HIT an amount less than the reward for the HIT. When you approve the assignment, the Worker is paid the full HIT reward. Rejecting an assignment impacts your account statistics as well as the statistics of the Worker, so you should always approve an assignment if the Worker completed the instructions in the HIT successfully.
If you want to offer Workers a reward within a range of amounts, you can post the HIT with a reward equal to the lowest amount in the range, then mention the offer of a bonus payment in the description of the HIT. For example, to offer a HIT with a reward between $1 and $5, post the HIT with a reward of $1, then mention in the description that you will pay a bonus of up to $4.
You can retrieve a list of bonuses paid for a particular HIT or assignment by calling the
Once all assignments have been approved or rejected, your application calls DisposeHIT. This removes the HIT from the list of HITs returned by GetReviewableHITs, and tells Amazon Mechanical Turk the data is no longer needed.
Once you dispose of a HIT, you can no longer approve the HIT's rejected assignments.
If a HIT has expired, or the maximum number of assignments has all been submitted, the HIT will no longer be available for Workers to accept. If the HIT has not gathered a satisfactory result, you may extend either the expiration date or the number of assignments, or both, using ExtendHIT. With appropriate values, extending the HIT will make it available again.
The DisableHIT operation allows you to completely withdraw a HIT from the system, even before it has expired. Assignments that have been submitted, but not yet explicitly approved or rejected, will be approved automatically. Assignments in progress are allowed to complete, then are approved automatically if submitted by the Worker. Assignments returned or abandoned after the HIT is disabled are simply discarded. The HIT and all submitted assignment data are disposed. A disabled HIT cannot be re-enabled.
Like ForceExpireHIT, disabling a HIT is useful if the HIT's question data is incorrect and would cause Workers to submit bad results. Workers expect to be paid for correctly answering the question, even if the question is not the one you intended to ask. DisableHIT removes a HIT from the Amazon Mechanical Turk marketplace, approves any submitted assignments pending approval or rejection, and disposes of the HIT and all assignment data. Assignment results data cannot be retrieved for a HIT that has been disposed. Assignments in progress at the time of the call to the DisableHIT operation are approved once the assignments are submitted. You will be charged for approval of these assignments.
DisableHIT is like ForceExpireHIT, but much more drastic. It provides an automatic method to
completely settle a HIT and all of its assignments with one service call.
Assignment data cannot be retrieved until the HIT enters the "reviewable" status, when it is no longer possible for Workers to submit more results. Because it is only necessary to disable a HIT to prevent Workers from submitting results, it is likely you will only wish to disable a HIT prior to it becoming "reviewable". This means you will not get to see the results for submitted assignments prior to disabling the HIT.