Quality Management at Amazon's Mechanical Turk

One interesting quality management topic is ensuring quality work in crowd sourcing such as Amazon's Mechanical Turk.  Crowd sourcing can be especially beneficial in research and developing software.  Requesters can gather large amounts of data for a relatively low cost.  Ensuring quality data can seem like a very difficult task because anyone at any knowledge, skill, and intent level can perform the work. 

Amazon's Mechanical Turk is a marketplace for work (Amazon).  It enables individuals or businesses to post tasks that they would like completed to an on-demand scalable workforce (Human Intelligence, which means that anyone can choose to perform that task from their computer that is connected to the internet. These tasks are tasks that usually involve simple tasks that require human judgement and intelligence.  For instance, A9's BlockView pictures show street-level pictures of businesses and people are asked to select the best representation of the business from multiple photographs (Amazon).  It seems a daunting task to make sure that the requestors are getting quality information from the tasks that they are paying people.

Malicious workers often take advantage of this and submit answers of low quality.  Most requestors rely on redundancy to identify the correct answers, however, it is expensive to have enough submittals to ensure the correct answer.  One group worked on an algorithm for multiple choice type work that incorporates estimating the correct answer by getting answers from multiple workers while accounting for each workers quality, and estimating the quality of the works by comparing their answers with the inferred correct answer (Ipeirotis).  The output of this algorithm would be able to show the correct answer for each question and the quality rating of each worker.  

Amazon addresses the question of quality data in their Frequently Asked Questions page for Amazon Mechanical Turk.  Requestors have the option to automatically approve work completed without viewing, automatically approving the work completed when at least one other person has the matching answers, and manually approving them.  Requestors can also specify that the workers complete a qualification test and they have to meet a certain level in order to start on the work.   Also, workers have statistics associated with their account that track how many jobs they have completed and how many of those were approved (Amazon).  Amazon is saying that they do have some means of quality control, but it is up to the customer to decide on how much control they want to ensure accurate data.  There could be much effort on Amazon's side for quality management. 

Amazon should develop this quality management more, to ensure further quality.  Amazon is dealing with hundreds of thousands of "employees" with no direct supervisor besides the rules they set in place.  Amazon could try to set more rules and try to develop their relationship with their workers.  It is a very difficult situation because the idea of crowdsourcing is to be able to get information from a lot of workers to get a confident answer and it is important to get quality answers, however that would increase costs and time.   It would be good to use an algorithm to help ensure the answers are quality and the workers are quality.  Workers should have developed profiles that give more criteria for statistics.  They should also include work type and categories, as well as, some personal and demographic type of information.  This could help requesters to help determine marketing strategies for different demographics, for example.  The more advanced Amazon Mechanical Turk's quality management system is, the more inclined businesses will be willing to use the crowd sourcing effort to help their business.

Amazon Mechanical Turk. Amazon Web Services. 2013 http://aws.amazon.com/mturk/faqs/

Ipeirotis, P; Wang, Jing; Provost, Foster.  "Quality Management on Amazon Mechanical Turk." Department of Information, Operations and Management Sciences.  Leonard N. Stern School of Business, New York University.  July 2010.