Creating tasks that work well on Amazon Mechanical Turk - Amazon Mechanical Turk

Creating tasks that work well on Amazon Mechanical Turk

Amazon Mechanical Turk (Mechanical Turk) can be used for an exceptionally wide range of tasks. Tasks that work well on Mechanical Turk generally meet the following criteria:

  • Can be completed from within a web browser

  • Can be broken into distinct, bite-sized tasks

  • Can support clear instructions and outcomes

Most tasks that meet these criteria can be completed on Mechanical Turk, assuming you provide workers with a task interface that allows them to successfully perform the task. You should also keep in mind that Mechanical Turk workers excel at tasks that rely on general human knowledge and skills. While some workers have specialized experience such as legal or medical backgrounds, most do not. As a result, while Mechanical Turk can enable tasks such as labeling the location of people or animals in images, you are likely to have less success asking workers to apply expertise that would be associated with a radiologist.

Note that tasks must also conform to the rules in the Mechanical Turk Acceptable Use Policy. Prohibited uses cover a range of activities such as violating the privacy or security of workers or others, abusive behavior, or any illegal activities.

Tasks can be completed within a web browser

Mechanical Turk tasks are built using HTML and presented to workers via the Mechanical Turk website. Most workers complete tasks on their computer without the need to use other devices or specialized software. Tasks that require workers to visit physical locations or leverage other devices aren't recommended.

Work can be broken into distinct, bite-sized tasks

Most Mechanical Turk tasks take less than five minutes to complete and almost all can be completed within an hour. This lets workers try new tasks without needing to commit a lot of time. Most workers appreciate the flexibility that Mechanical Turk provides in moving from task to-task without being locked in for an extended period of time.

Task supports clear instructions and outcomes

The most successful tasks on Mechanical Turk are those that provide the necessary information for a worker to imagine what a successful response would look like. Avoid tasks that are open-ended and could have multiple possible outcomes. For example, a task that asks workers to identify all of the competitors of company X would be frustrating for workers. By specifying that you want all competitors, workers are left wondering at what point they should draw a line and stop their research. It would also leave them wondering if you will reject their work if they aren't as comprehensive as you want them to be. In this example, you should instead be specific about the data that you need by describing your task as identify the top 5 competitors of company X.

Examples of common uses of Mechanical Turk

The following are examples of common Mechanical Turk use-cases:

  • Audio transcription: Transcribe an audio clip.

  • Categorization: Categorize products.

  • Data collection: Identify the website for a business.

  • Writing: wWrite a description of a product based on an image and details.

  • Market research: Complete a market research survey.

  • Rating: Evaluate and rate the quality of an image.

  • Usability testing: Visit a website and complete a set of steps, providing feedback on each step.

  • Research study: Participate in a study by responding to questions surrounding a scenario.

  • Computer vision: Draw bounding boxes around animals in images.

  • Natural language processing: Identify the named entities within a statement.

  • Matching: Review two data records and confirm they relate to the same business.

  • Moderation: Evaluate a set of images and identify any that don't meet the provided criteria.

  • Ranking: Rank a list of products based on their relevance to a search query.

  • Data extraction: Extract the names and prices of products in a receipt.

  • Text transcription: Transcribe handwritten text.

  • Video transcription: Transcribe a video clip.