Demographics of Mechanical Turk

Discussion in 'MTurk Help' started by clickhappier, Jul 28, 2014.

  1. clickhappier ★★Ⰼ₳ՖŦξᚱ⌚ Contributor

    Joined:
    Jul 1, 2014
    Messages:
    5,600
    Likes Received:
    9,527
    Here is a ton of information I've found about MTurk worker demographics and related interesting statistics. Some of these academic research sources (and others) are also listed in Tribune's Articles and Academic Papers About Turking thread.

    I'm making separate posts for each major source (paper/article/post). Whenever possible (exceptions include maps, a complex table of languages, and a few complex charts/graphs), information being included is presented here as text rather than images (for searchability, easier comparison of different sources, etc); charts/graphs that had value labels are typed out with those exact values, and charts/graphs that had unlabeled values are typed out with approximated values. To maintain direct links/references, posts #50+ will not necessarily appear in chronological order.


    Index of topics with post #s:

    Basic personal characteristics:
    - gender: 5, 6, 8, 12, 13, 15, 19, 20, 21, 22, 23, 27, 29, 33, 35, 37, 39, 40, 41, 43, 48, 49, 50, 52, 53, 54
    - orientation: 40, 52
    - age: 5, 6, 8, 12, 13, 15, 19, 20, 21, 22, 23, 27, 29, 33, 35, 37, 39, 41, 43, 46, 48, 50, 52, 53, 54
    - race: 12, 19, 20, 29, 37, 40, 45, 52, 53
    - education: 6, 8, 15, 19, 20, 21, 27, 33, 37, 39, 40, 41, 43, 48, 50, 51, 53
    - income: 5, 6, 12, 13, 15, 27, 35, 53

    Household/family:
    - children: 12, 19, 20, 40
    - marital status: 19, 20, 37, 40
    - household size: 19, 20
    - household income: 19, 20, 21, 40, 46

    Location-related:
    - countries: 3, 5, 6, 13, 15, 18, 19, 20, 21, 22, 25, 29, 33, 38, 47
    - IP geocoding: 22, 25, 32, 37
    - IP address sharing: 45
    - languages: 28, 44
    - map: 28, 36, 44
    - US region: 37
    - US states: 54
    - urban/rural: 49

    MTurk usage:
    - total accounts: 2, 4, 11, 14, 24, *30*, 34
    - total HITs available: 24
    - day/time fluctuations: 22, 25, 39
    - motivations: 6, 15, 18, 19, 20, 21, 27, 33, 34, 50
    - earnings per week: 9, 15, 19, 20
    - hours per week: 15, 19, 20, 27
    - HITs per week: 19, 27
    - quantity of HITs: 34
    - tenure [how long using MTurk]: 15, 19, 20
    - percent done by top workers: 16, *34*, 37, 45, 46 [see also #20]
    - reasons for choosing HITs: 33
    - fair pay rates: 46

    MTurk performance:
    - completion: 7, 10, 23, 27
    - follow-up response rate: 37, 45
    - quality (honesty/consistency): 7, 23, 26, 31, 32, 35, 37
    - quality (attention checks): 17, 23, 27, 33, *42*
    - quality (effect of approval rate requirement): 17
    - quality (effect of country): 38
    - quality (accuracy at complex tasks): 51
    - work environment: 45
    - non-mturk study costs: 37
    - quantity of papers: 34, 37, 45

    Work outside MTurk:
    - job industry/profession: 10, 43
    - job occupation/role: 10, 23, 41, 49
    - student status: 15
    - employment status: 15, 40, 50

    Misc:
    - daily hours of internet use: 41
    - turker community use: 45, 51
    - social media use: 5, 48
    - privacy positions: 41, 48
    - politics: 37, 39, 49
    - religion: 37, 54
    - mental health: 40
    - substance abuse: 40
    - disability: 46

    List of posts:
    1. > this index/intro post
    2. > Nov 2, 2006 - Christian Science Monitor news article by Gregory M. Lamb: "When workers turn into 'turkers' " - total accounts
    3. > Dec 1, 2006 - Sherwood Stranieri blog post: "Mechanical Turk Workers Mostly US-Based" - countries
    4. > Mar 25, 2007 - New York Times (NYT) news article by Jason Pontin: "Artificial Intelligence, With Help From the Humans" - total accounts
    5. > Apr to ~Jun 2007 - Jeffrey Harmon blog posts about his MTurk survey - social media use, income, gender, age, countries
    6. > Mar 19, 2008 - Panos Ipeirotis blog post: "Mechanical Turk: The Demographics" - countries, gender, age, education, income, motivations
    7. > Apr 2008 - paper by Aniket Kittur, Ed H. Chi, and Bongwon Suh: "Crowdsourcing user studies with Mechanical Turk" - completion, quality (honesty/consistency)
    8. > Aug 30, 2008 - Tom Roberts commented on Dan Rockwell blog post: "Valley of the Turks" - gender, education, age
    9. > Sep 12, 2008 - Panos Ipeirotis blog post: "How Much Turking Pays?" - earnings per week
    10. > Oct 2008 - paper by Brynn M. Evans and Ed H. Chi: "Towards a Model of Understanding Social Search" - job industry/profession, job occupation/role, completion
    11. > Mar 3, 2009 - Wall Street Journal (WSJ) news article by Elizabeth Holmes: "Selling Expertise On the Internet For Extra Cash"; plus June 26, 2009 Puget Sound Business Journal news article by Eric Engleman: "Amazon's Sharon Chiarella on crowdsourcing and cookie sales" - total accounts
    12. > Mar 16, 2009 - Panos Ipeirotis blog post: "Turker Demographics vs Internet Demographics" - age, gender, income, children, race
    13. > Jun 2009 - paper by Winter Mason and Duncan J. Watts: "Financial Incentives and the 'Performance of Crowds' " - countries, gender, age, income
    14. > Sep 15, 2009 - whitepaper by Brent Frei of Smartsheet: "Paid Crowdsourcing, Current State & Progress toward Mainstream Business Use" - total accounts
    15. > ~Sep 2009 - paper by Joel Ross, Andrew Zaldivar, Lilly Irani, and Bill Tomlinson: "Who are the Turkers? Worker Demographics in Amazon Mechanical Turk" - countries, gender, age, education, student status, employment status, income, tenure, hours per week, earnings per week, motivations
    16. > Dec 13, 2009 - Greg Little blog post: "How many turkers are there?" - percent done by top workers
    17. > Jan 5, 2010 - Raphael Mudge blog post: "Be a Good Turk Boss – Mechanical Turk Tips", plus an agreeing opinion from Michael Buhrmester's "Amazon Mechanical Turk Guide for Social Scientists", and a somewhat disagreeing opinion from Eyal Peer's blog post "Reputation as a Sufficient Condition for High Data Quality on MTurk" - quality (effect of approval rate requirement, attention checks)
    18. > Feb 27, 2010 - John Horton blog post: "Why People Participate on Mechanical Turk, Now as a Mosaic Plot" - countries, motivations
    19. > Mar 10, 2010 - paper and blog post by Panos Ipeirotis: "(New) Demographics of Mechanical Turk" - countries, gender, age, education, household income, marital status, children, household size, race, tenure, earnings per week, HITs per week, hours per week, motivations
    20. > My (clickhappier) addendum to Panos Ipeirotis's March 2010 "Demographics of Mechanical Turk", with details of the top 10% 'Super Turkers' - countries, gender, age, education, household income, marital status, children, household size, race, tenure, earnings per week, hours per week, motivations
    21. > Apr 2010 - paper by Joel Ross, Lilly Irani, M. Six Silberman, Andrew Zaldivar, and Bill Tomlinson: "Who are the Crowdworkers? Shifting Demographics in Mechanical Turk" - countries, gender, age, education, household income, motivations
    22. > Apr 2010 - paper by Robert Kosara and Caroline Ziemkiewicz: "Do Mechanical Turks Dream of Square Pie Charts?" - age, gender, day/time fluctuations, quality (honesty/consistency), countries, IP geocoding
    23. > Apr 2010 - paper by by Julie S. Downs, Mandy B. Holbrook, Steve Sheng, and Lorrie Faith Cranor: "Are Your Participants Gaming the System? Screening Mechanical Turk Workers" - quality (attention checks, honesty/consistency), completion, gender, age, job occupation
    24. > May 2010 - paper by Alek L. Felstiner (J.D.): "Working the Crowd: Employment and Labor Law in the Crowdsourcing Industry" - total accounts, total HITs available
    25. > May 12, 2010 - John Le blog post: "Amazon Mechanical Turk Survey" - countries, IP geocoding, day/time fluctuations
    26. > ~Jun 2010 - paper by Patrick Gage Kelley: "Conducting Usable Privacy & Security Studies with Amazon’s Mechanical Turk" - quality (honesty/consistency)
    27. > Aug 2010 - paper by Gabriele Paolacci, Jesse Chandler, and Panagiotis G. (Panos) Ipeirotis: "Running experiments on Amazon Mechanical Turk" - gender, age, education, income, motivations, hours per week, HITs per week, quality (attention checks), completion
    28. > Oct 18, 2010 - Rob Munro blog post: "What languages are spoken by crowdsourced workers?", plus his October 19, 2013 post about the same study: "Crowdsourcing and worker diversity" - languages, map
    29. > Jan 2011 - paper by Michael Buhrmester, Tracy Kwang, and Samuel D. Gosling: "Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?" - countries, gender, race, age
    30. > Jan 26, 2011 - requester forum post by Amazon employee '[email protected]': "Re: MTurk CENSUS: About how many workers were on Mechanical Turk in 2010?", plus related commentary - total accounts
    31. > Mar 2011 - paper by David G. Rand: "The promise of Mechanical Turk: How online labor markets can help theorists run behavioral experiments" - quality (honesty/consistency)
    32. > Mar 14, 2011 - Panos Ipeirotis blog post: "Do Mechanical Turk workers lie about their location?" - IP geocoding, quality (honesty/consistency)
    33. > Jun 2011 - paper by Thimo Schulze, Stefan Seedorf, David Geiger, Nicolas Kaufmann, and Martin Schader: "Exploring Task Properties in Crowdsourcing - An Empirical Study on Mechanical Turk" - reasons for choosing HITs, quality (attention checks), countries, gender, age, education, motivations
    34. > Jun 2011 - paper by Karën Fort, Gilles Adda, and K. Bretonnel Cohen: "Amazon Mechanical Turk: Gold Mine or Coal Mine?" - quantity of papers, total accounts, quantity of HITs, percent done by top workers, motivations
    35. > Jun 30, 2011 - paper by Winter Mason and Siddharth Suri: "Conducting behavioral research on Amazon’s Mechanical Turk" - gender, age, income, quality (honesty/consistency)
    36. > ~Sep 2011 - Dahn Tamir of Techlist created an interactive Google Map of MTurk workers' locations: "50,000 Worl(d)wide Mechanical Turk Workers" - map
    37. > Mar 2012 - paper by Adam J. Berinsky, Gregory A. Huber, and Gabriel S. Lenz: "Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk" - gender, age, education, non-mturk study costs, quantity of papers, follow-up response rate, IP geocoding, politics, race, marital status, religion, US region, quality (honesty/consistency), percent done by top workers
    38. > May (Spring) 2012 - paper by Adam Berinsky, Kai Quek, and Michael Sances: "Conducting Online Experiments on Mechanical Turk" - gender, age, education, countries, quality (effect of country)
    39. > Dec 19, 2012 - Sean Richey and J. Benjamin Taylor blog post: "How Representative Are Amazon Mechanical Turk Workers?" - politics, age, gender, education, day/time fluctuations
    40. > Apr 2013 (written in 2012) - paper by Danielle N. Shapiro, Jesse Chandler, Pam A. Mueller: "Using Mechanical Turk to Study Clinical Populations" - race, gender, orientation, marital status, children, education, employment status, household income, mental health, substance abuse
    41. > Jul 2013 - paper by Pedro Giovanni Leon, et al: "What Matters to Users? Factors that Affect Users' Willingness to share Information with Online Advertisers" - age, gender, education, job occupation, daily hours of internet use, privacy positions
    42. > Jul 15, 2013 - paper by Adam Berinsky, Michele F. Margolis, and Michael Sances: "Separating the Shirkers from the Workers? Making Sure Respondents Pay Attention on Self-Administered Surveys" - quality (attention checks)
    43. > Jul 20, 2013 - Amy Quarton blog post: "Mechanical Turk 101: Worker Demographics" - age, gender, job industry, education
    44. > Feb 2014 - paper by Ellie Pavlick, Matt Post, Ann Irvine, Dmitry Kachaev, and Chris Callison-Burch: "The Language Demographics of Amazon Mechanical Turk" - languages, map
    45. > Mar 2014 (written in 2013) - paper by Jesse Chandler, Pam Mueller, and Gabriele Paolacci: "Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers" - IP address sharing, percent done by top workers, follow-up response rate, race, work environment, turker community use, quantity of papers
    46. > May 8, 2014 - paper by Vanessa Williamson: "On the Ethics of Crowd-sourced Research" (working paper) - percent done by top workers, age, disability, household income, fair pay rates
    47. > Jun 5, 2014 - My (clickhappier) Reddit comment: "re: Is it worth posting to mturk for non-US jobs?", plus my addendum with prior years for comparison - countries (estimates from Alexa data for 2011-2014)
    48. > Jul 2014 - paper by Ruogu Kang, Stephanie Brown, Laura Dabbish, and Sara Kiesler: "Privacy Attitudes of Mechanical Turk Workers and the U.S. Public" - age, gender, education, social media use, privacy positions
    49. > Jul 15, 2014 draft - paper by Connor Huff and Dustin Tingley: " 'Who are These People?': Evaluating the Demographic Characteristics and Political Preferences of MTurk Survey Respondents" - job occupation, urban/rural, gender, politics
    50. > Feb 10, 2014 - forum post by Dahn Tamir of Techlist, and addendum by 'RippedWarrior': "MTurk Work and Education Survey (Results)" - age, gender, education, employment status, motivations
    51. > Jun 11, 2014 - paper by Matthew Staffelbach, et al: "Lessons Learned from an Experiment in Crowdsourcing Complex Citizen Engineering Tasks with Amazon Mechanical Turk" - quality (accuracy at complex tasks), education, turker community use
    52. > Aug 5, 2014 - paper abstract by JoEllen Blass and George Chavez: "Are You Transphobic?: How Biological Views Influence Attitudes" - gender, orientation, race, age
    53. > Feb 2015 - statistics from PickFu (2008-2015): "Demographic Information for PickFu Poll Responders" - gender, age, race, education, income
    54. > Sep-Oct 2014 - statistics from Stephen Rapier's "Online Search Use" survey demographic questions - gender, age, US states, religion

    This thread is referenced in Lilly Irani's Jan 2015 article "Justice for 'Data Janitors' ", and the Dynamo Guidelines for Academic Requesters (Fair Payment).


    Of related interest: Turkopticon Usage Statistics; Grinders' Weekly Activity Reports Totals
     
    Last edited: Mar 14, 2015
    • Like Like x 6
  2. clickhappier ★★Ⰼ₳ՖŦξᚱ⌚ Contributor

    Joined:
    Jul 1, 2014
    Messages:
    5,600
    Likes Received:
    9,527
    November 2, 2006 - Christian Science Monitor news article by Gregory M. Lamb: "When workers turn into 'turkers' " , also republished by USA Today

    "For nearly a year, Amazon.com's Mechanical Turk (mturk.com) has paid amounts ranging from one cent to several dollars for tasks that take a few seconds to a few minutes to complete."

    " "It looks as though there are about 5,000 to 10,000 people registered" at Mechanical Turk and about 500 are online looking for assignments at any given time, says Sherwood Stranieri, a search engine marketing consultant who has posted work on the Turk and writes a Web log about Turking at www.paylancers.blogspot.com . Though not huge, that's already a pretty useful number of Turkers to draw on, he says. Mr. Stranieri posted an advertising copy-writing assignment on the Turk and was impressed by the quality of work returned. "There obviously are some very educated people floating around in that system," he says. "The vocabulary they used was pretty impressive." "

    "Amazon requires Turkers to be paid with US dollars deposited into an American bank account. Though Amazon's Mr. Cohen (Peter Cohen, the director of Amazon's Web Services software unit, which includes the Mechanical Turk) says Turkers already come from 100 countries, the vast majority are in the US. The true impact on wages of large numbers of, say, Indians or Chinese becoming Turkers has yet to be felt."

    This is apparently the first official 'number of countries' number.
     
    • Like Like x 1
  3. clickhappier ★★Ⰼ₳ՖŦξᚱ⌚ Contributor

    Joined:
    Jul 1, 2014
    Messages:
    5,600
    Likes Received:
    9,527
    December 1, 2006 - Sherwood Stranieri of Skypromote wrote a blog post: "Mechanical Turk Workers Mostly US-Based"

    "I've got some results to share from a questionaire deployed on Mechanical Turk to gather stats and feedback from the Turk workforce. Here's a taste [looks like this 'taste' was all he ever posted] of the info gathered from the first 100 responses: a breakdown of country of origin. As suspected, Amazon's U.S.-centric payment system has resulted in a mostly-U.S. workforce, but a few workers from elsewhere are participating."

    79% United States
    4% Canada
    2% UK
    2% Israel
    2% France
    1% each from Cyprus, Finland, Germany, India, Italy, Malta, Netherlands, Romania, Singapore, Thailand, and (yes!) Turkey.

    An anonymous comment clarified, "The reason there aren't more workers in the UK and els[e]where is because MTurk pays in Amazon.com credit. You can't transfer the credit to Amazon.co.uk [or other regional Amazon sites], and you can only bank the money [transfer it to a bank account] if you have a US account."
     
    • Like Like x 1
  4. clickhappier ★★Ⰼ₳ՖŦξᚱ⌚ Contributor

    Joined:
    Jul 1, 2014
    Messages:
    5,600
    Likes Received:
    9,527
    March 25, 2007 - New York Times (NYT) news article by Jason Pontin: "Artificial Intelligence, With Help From the Humans"

    "The company opened Mechanical Turk as a public site in November 2005. Today, there are more than 100,000 “Turk Workers” in more than 100 countries"

    Since Jeff Bezos spoke with the reporter for this article, this is apparently the first official 'total accounts' number.
     
    • Like Like x 1
  5. clickhappier ★★Ⰼ₳ՖŦξᚱ⌚ Contributor

    Joined:
    Jul 1, 2014
    Messages:
    5,600
    Likes Received:
    9,527
    April to ~June 2007 - Jeffrey Harmon of now-defunct Pyxlin ran a survey through MTurk, primarily intended to gather information to gauge people's interest in his online journal service, but is also the earliest example I've found so far of detailed mTurk demographic gathering (although the data may not be purely from MTurk users, since he posted a link on his blog from which visitors could take the survey).

    He made blog posts about the survey on April 11, April 20, and June 6, 2007.

    The blog posts included a link to see a 'Real-Time Summary Report' of the results, but the survey had since been removed/deactivated from its host's records, so the link no longer works. I retrieved the data from archived copies; the June 9, 2007 copy is probably most representative of MTurk results, as the June 6 blog post sounded like he was done running it. There were 2,155 responses as of June 9, but by Jan 29, 2008, there were 3,349 responses, despite no indication in the blog that he continued offering it through MTurk HITs.

    Results of interest, as of June 9, 2007:
    Completed: 2155 [country responses add up to 2147]

    "Do you use any social sites? (ie FaceBook, Digg, MySpace, Reddit, YouTube, etc...):"

    yes: 1454, 67.41%
    no: 663, 30.74%
    other: 40, 1.85%

    "What is your annual income (US Currency)?:" [This of course doesn't account for the different countries' cost of living / average income. Since the whole range of possibilities was covered by the other choices, presumably the 'Other's either didn't know what their income total is or didn't know how to convert from their currency to US dollars.]

    less than $12,000: 524, 24.41%
    $12,000-$25,000: 356, 16.58%
    $25,000-$40,000: 423, 19.70%
    $40,000-$60,000: 352, 16.39%
    $60,000-$100,000: 284, 13.23%
    $100,000+: 154, 7.17%
    Other: 54, 2.52%

    "Your Gender?:"

    Male: 1169, 54.45%
    Female: 978, 45.55%

    "Your Age Is:" [Presumably the '13-17's and 'Other's (who must either be under 13 or didn't understand the options) came from blog post survey link contamination, but I don't know what Amazon's verification system was like back then, or could've been using a relative's account.]

    13-17: 34, 1.58%
    18-22: 428, 19.86%
    23-28: 693, 32.16%
    29-32: 332, 15.41%
    33-39: 333, 15.45%
    40-59: 289, 13.41%
    60+: 27, 1.25%
    Other: 19, 0.88%

    "What country are you from?:" [The list had a total of 185 countries + 'other' to choose from; I omitted the ones with 0 responses, sorted the remaining 75 from most to fewest responses, and put the 66 ones with less than 10 responses in a spoiler.]

    United States of America: 1379, 64.23%
    India: 352, 16.39%
    United Kingdom and Northern Ireland: 77, 3.59%
    Canada: 68, 3.17%
    Australia: 22, 1.02%
    Germany: 22, 1.02%
    Philippines: 16, 0.75%
    Turkey: 16, 0.75%
    France: 10, 0.47%
    Netherlands: 9, 0.42%
    Argentina: 7, 0.33%
    Brazil: 7, 0.33%
    Israel: 7, 0.33%
    New Zealand: 7, 0.33%
    Malaysia: 6, 0.28%
    Singapore: 6, 0.28%
    Austria: 5, 0.23%
    China: 5, 0.23%
    Italy: 5, 0.23%
    Mexico: 5, 0.23%
    Romania: 5, 0.23%
    Switzerland: 5, 0.23%
    Venezuela: 5, 0.23%
    Andorra: 4, 0.19%
    Belgium: 4, 0.19%
    Bulgaria: 4, 0.19%
    Denmark: 4, 0.19%
    Japan: 4, 0.19%
    Norway: 4, 0.19%
    South Africa: 4, 0.19%
    Afghanistan: 3, 0.14%
    Bosnia and Herzegovina: 3, 0.14%
    Colombia: 3, 0.14%
    Finland: 3, 0.14%
    Greece: 3, 0.14%
    Latvia: 3, 0.14%
    Slovakia: 3, 0.14%
    Sweden: 3, 0.14%
    Uruguay: 3, 0.14%
    Angola: 2, 0.09%
    Cuba: 2, 0.09%
    Hungary: 2, 0.09%
    Indonesia: 2, 0.09%
    Iran: 2, 0.09%
    Lithuania: 2, 0.09%
    Poland: 2, 0.09%
    Slovenia: 2, 0.09%
    Spain: 2, 0.09%
    Tanzania: 2, 0.09%
    Albania: 1, 0.05%
    Antigua and Barbuda: 1, 0.05%
    Armenia: 1, 0.05%
    Azerbaijan: 1, 0.05%
    Bangladesh: 1, 0.05%
    Belarus: 1, 0.05%
    Bhutan: 1, 0.05%
    Brunei Darussalam: 1, 0.05%
    Burkina Faso: 1, 0.05%
    Chile: 1, 0.05%
    Croatia: 1, 0.05%
    Guatemala: 1, 0.05%
    Iceland: 1, 0.05%
    Ireland: 1, 0.05%
    Kuwait: 1, 0.05%
    Malta: 1, 0.05%
    Morocco: 1, 0.05%
    Portugal: 1, 0.05%
    Republic of Korea: 1, 0.05%
    Taiwan: 1, 0.05%
    Thailand: 1, 0.05%
    Trinidad and Tobago: 1, 0.05%
    Ukraine: 1, 0.05%
    Uzbekistan: 1, 0.05%
    VietNam: 1, 0.05%
    Yemen: 1, 0.05%
     
    • Like Like x 1
  6. clickhappier ★★Ⰼ₳ՖŦξᚱ⌚ Contributor

    Joined:
    Jul 1, 2014
    Messages:
    5,600
    Likes Received:
    9,527
    March 19, 2008 - Panos Ipeirotis at New York University wrote a blog post: "Mechanical Turk: The Demographics"

    "The next survey that I conducted focused more on the demographics of the Turkers. ... Turkers are a pretty representative sample of the online population, perhaps with a slight bias towards females and towards young participants."

    "First, I would start with the country breakdown."
    United States: 76.25%
    India: 8.03%
    United Kingdom: 3.34%
    Canada: 2.34%
    [Other: 10.04%]

    "The clear result is that most of the participants are coming from the US and not from a third world country, despite the common misconception. This is due to the fact that in order to get paid, someone has to have a US bank account, or be willing to be paid using Amazon gift certificates."

    "Then, the gender breakdown:" [chart approximated as text:]

    Female: ~58-59%
    Male: ~41-42%

    "Next, the age distribution:" [chart approximated as text:]
    21-30: ~43%
    31-40: ~34%
    41-50: ~17%
    51-60: ~4%
    over 60: ~2%

    "And what about education?:" [chart approximated as text:]
    High school: ~28%
    Bachelor's: ~52%
    Master's: ~16%
    PhD: ~4%

    "Turkers are a pretty representative sample. Most of them have a college education, and some of them even have PhDs! In fact, the distribution seems pretty similar to the distribution for the overall US population."

    "Similarly, the income distribution also follows closely the income distribution in the US:" [chart approximated as text:]

    <$10K: ~16%
    $10K-$25K: ~21%
    $25K-$40K: ~22%
    $40K-$55K: ~15%
    $55K-$70K: ~10%
    $70K-$85K: ~6%
    $85K-$100K: ~5%
    $100K-$150K: ~3%
    $150K-$200K: ~1%
    >$200K: ~1%

    "Finally, why people participate in Mechanical Turk? ... (See also the detailed responses.) Here is the breakdown of the responders when they had to choose (not exclusively) between the choices "for money," "for fun," and "for killing time":" [chart approximated as text and percents added (total = ~300):]

    For money (only): ~110, ~37%
    For fun (only): ~30, ~10%
    Killing time (only): ~25, ~8%
    For money and fun: ~85, ~28%
    For money & kill time: ~10, ~3%
    For fun & kill time: ~5, ~2%
    For all three: ~35, ~12%
     
    • Like Like x 1
  7. clickhappier ★★Ⰼ₳ՖŦξᚱ⌚ Contributor

    Joined:
    Jul 1, 2014
    Messages:
    5,600
    Likes Received:
    9,527
    April 2008 - academic paper: "Crowdsourcing user studies with Mechanical Turk" (pdf)
    by Aniket Kittur, Ed H. Chi, and Bongwon Suh, at Palo Alto Research Center (PARC)

    "In March 2007, Amazon claimed the user base of Mechanical Turk (who commonly refer to themselves as “turkers”) consisted of over 100,000 users from over 100 countries."

    "We conducted two experiments to test the utility of Mechanical Turk as a user study platform. We used tasks that collected quantitative user ratings as well as qualitative feedback regarding the quality of Wikipedia articles. ... In the first experiment ... 58 users provided 210 ratings for 14 articles (i.e., 15 ratings per article). User response was extremely fast, with 93 of the ratings received in the first 24 hours after the task was posted, and the remaining 117 received in the next 24 hours. Many tasks were completed within minutes of entry into the system"

    "Out of the total of 210 free-text responses regarding how the article could be improved, 102 (48.6%) consisted of uninformative responses including semantically empty (e.g., “None”), non-constructive (e.g., “well written”), or copy-and-paste responses (e.g., “More pictures to break up the text” given for all articles rated by a user). ... 64 ratings [were] completed in less than 1 minute (less time than likely needed for reading the article, let along rating it). 123 (58.6%) ratings were flagged as potentially invalid based either on their comments or duration. ... However, many of the invalid responses were due to a small minority of users. Only 8 users gave 5 or more responses flagged as potentially invalid based on either comments or time; yet these same users accounted for 73% (90 responses) of all flagged responses. Thus it appeared that, rather than widespread gaming, a small group of users were trying to take advantage of the system multiple times."

    "In experiment 2, we tried a different method of collecting user responses ... In the new rating task, users were required to complete four questions that had verifiable, quantitative answers before rating the quality of the article. ... 124 users provided 277 ratings for 14 articles (i.e., 19-20 ratings per article). The number of ratings per user was significantly smaller than for Experiment 1 (2.2 vs. 3.6 ...) and also more distributed across users (only 5% of users rated 8 or more pages, vs. 16% in Experiment 1). ... In addition to the improved match to expert ratings, there were dramatically fewer responses that appeared invalid. Only 7 responses had meaningless, incorrect, or copy-and-paste summaries, versus 102 in Experiment 1. Also, only 18 responses were completed in less than one minute, and the median completion time was much higher than in Experiment 1 (4:06 vs. 1:30 ...)."

    In March 2008, co-author Ed H. Chi also wrote a blog post about this study, "How to reduce the cost of doing user studies with Crowdsourcing".
     
    Last edited: Jan 18, 2015
    • Like Like x 1
  8. clickhappier ★★Ⰼ₳ՖŦξᚱ⌚ Contributor

    Joined:
    Jul 1, 2014
    Messages:
    5,600
    Likes Received:
    9,527
    August 30, 2008 - Tom Roberts commented on a blog post by Dan Rockwell: "Valley of the Turks"

    "Inspired by being a participant in a recent HIT you published, I created a HIT ... I paid for 100 HITs (25 cents each) and was surprised by the demographics:"

    Gender:
    Female: 68%
    Male: 32%

    Highest Education Achieved:
    Some High School: 3%
    High School Graduate: 6%
    Some College, no degree: 27%
    Associates degree: 7%
    Bachelor's degree: 42%
    Graduate degree (Master's or Doctorate): 15%

    Age:
    19-23: 13%
    24-30: 34%
    31-40: 24%
    41-50: 17%
    51+: 12%

    "I was surprised and pleased by this breakdown. Again, not a scientific survey of all Turk users, but I will include those questions on all of my HITs to better interpret the results."
     
    • Like Like x 1
  9. clickhappier ★★Ⰼ₳ՖŦξᚱ⌚ Contributor

    Joined:
    Jul 1, 2014
    Messages:
    5,600
    Likes Received:
    9,527
    September 12, 2008 - Panos Ipeirotis at New York University wrote a blog post: "How Much Turking Pays?"

    "One of the most common questions was about the compensation of Turkers ... I posted the very same question on MTurk, asking people about their average compensation per week. ... here are the results:" [chart approximated as text:]

    "Reported Turker Compensation (per week)"
    <$1: ~8%
    $1- $5: ~31%
    $5- $10: ~24%
    $10- $20: ~17%
    $20- $50: ~14%
    $50-$100: ~5%
    $100-$200: ~1%

    "The current survey paid 5 cents per HIT, and received responses for 200 Turkers. I will keep running the survey to collect 1,000 responses and will report if I see any significant changes. But so far the results seem remarkably stable."
     
    • Like Like x 1
  10. clickhappier ★★Ⰼ₳ՖŦξᚱ⌚ Contributor

    Joined:
    Jul 1, 2014
    Messages:
    5,600
    Likes Received:
    9,527
    October 2008 - academic paper: "Towards a Model of Understanding Social Search" (pdf)
    by Brynn M. Evans at University of California, San Diego; and Ed H. Chi at Palo Alto Research Center

    "We surveyed users about their most recent searching behaviors ... We collected 150 complete survey responses from anonymous individuals on Mechanical Turk."

    "Table 1. The most frequently occurring professions and job roles reported by users in our sample."

    Profession:
    Education: 9.3%
    Financial: 8.7%
    Healthcare: 6.7%
    Govt. Agency: 6.0%
    Retail: 6.0%
    Software: 6.0%
    Research: 5.3%
    [Others: 52.0%]

    Job Role:
    Manager: 19.3%
    Assistant: 18.7%
    CEO/Director: 8.0%
    Customer Support: 7.3%
    Teacher: 6.0%
    Programmer: 6.0%
    Analyst: 4.0%
    [Others: 30.7%]

    Co-author Ed H. Chi wrote a blog posts about this study on October 2, 2008, and co-author Brynn Evans did likewise on July 9, 2008 and October 15, 2008.

    From the former of Brynn Evans's posts:
    " we put this survey out on Mechanical Turk from March 7–April 9 (almost exactly 1 month). Here’s how we fared:
    Total submitted responses: 164
    Total responses that I accepted and compensated: 157 (95.7% of the submitted responses)
    Total accepted responses that were actually usable (good): 150 (95.5% of the accepted responses). "

    From the latter of Brynn Evans's posts:
    "Ed Chi & I ran a survey on Mechanical Turk asking 150 users to recount their most recent search experience ... We collected responses from 150 anonymous, English-speaking users with diverse backgrounds and occupations. In fact, there was so much diversity in our sample that the most highly represented professions were in Education (9%) and Financial Services (9%). The next ranking professions were Healthcare (7%) and Government Agency (6%) positions. We were quite surprised by the range of companies people worked for: from 1-person companies run out of people’s homes to LexisNexis, Liberty Mutual, EA Games, and the IRS!"

    Also, on November 20, 2008, Brynn Evans wrote a comment in response to a blog post by Andy Baio of Kickstarter, "The Faces of Mechanical Turk":

    "This is really fantastic! Putting a face to a "turk" is quite reassuring in some ways. I've had such great success with the Turk community though, that I've wondered less about who they are. I also have 300 replies from Turkers stating their professions, which I really should post on my blog to add another dimension to these mysterious people. For now here's a small sampling of their occupations (much more than simply students and part-time workers): spanish translator; United Nations worker; webmaster of dogtoys.com; researcher at Duke Medical Center; paralegal at Murray Law Office; manager of a Bait & Tackle shop in Florida..."

    (Unfortunately, it appears she never got around to making that additional blog post.)
     
    • Like Like x 1
  11. clickhappier ★★Ⰼ₳ՖŦξᚱ⌚ Contributor

    Joined:
    Jul 1, 2014
    Messages:
    5,600
    Likes Received:
    9,527
    March 3, 2009 - Wall Street Journal (WSJ) news article by Elizabeth Holmes: "Selling Expertise On the Internet For Extra Cash"

    "Amazon says that MTurk now has 200,000 workers from 100 different countries, but it doesn't keep track of past figures."


    Update: Although the WSJ is still the earliest source of this number I've seen, I've identified another mention of it with clearer authoritativeness.

    June 26, 2009 - Puget Sound Business Journal news article by Eric Engleman: "Amazon's Sharon Chiarella on crowdsourcing and cookie sales"

    "Sharon Chiarella is vice president of Amazon Mechanical Turk ... Chiarella, a veteran of Kodak, Microsoft and Yahoo, joined Amazon in 2007, and is overseeing the company's efforts to expand the Mechanical Turk service. She talked to TechFlash about the power of crowdsourcing ..."

    " On Mechanical Turk’s workforce:
    We have over 200,000 workers in 100 different countries, so we really are a 24-by-7 workforce. It follows the sun, if you will. "
     
    • Like Like x 1
  12. clickhappier ★★Ⰼ₳ՖŦξᚱ⌚ Contributor

    Joined:
    Jul 1, 2014
    Messages:
    5,600
    Likes Received:
    9,527
    March 16, 2009 - Panos Ipeirotis at New York University wrote a blog post: "Turker Demographics vs Internet Demographics"

    " One of the common questions that I see is: "How these demographics compare to the general Internet population?" ... I found some data from ComScore, dating back to June 2008. I also had data about Mechanical Turk from two separate surveys that I ran on October 2008 and on December 2008 (both asking 1000 Turkers). The results across the two MTurk surveys were rather consistent, indicating that the results are rather trustworthy.

    So, how Turkers compare to the general US Internet population? The short answers:
    • Turkers are younger. 54% of Turkers are between 21-35 years old, compared to 22% of the general population.
    • Turkers are mainly female. 70% of the Turkers are female, compared to 50% of the general population.
    • Turkers have lower income. 65% of Turkers have household income less than $60K, compared to 45% of the general population.
    • Turkers have smaller families. 55% of Turkers do not have children, compared to the 40% of the general population.
    • Geographical distribution of Turkers and Internet users is similar.
    • Race composition of Turkers and Internet users is similar, although there are slightly more Asians on Mechanical Turk.
    Of course, the last two bullets may be simply the result of the first: Younger people have lower income, do not have children, and live in smaller households. "
     
    • Like Like x 1
  13. clickhappier ★★Ⰼ₳ՖŦξᚱ⌚ Contributor

    Joined:
    Jul 1, 2014
    Messages:
    5,600
    Likes Received:
    9,527
    June 2009 - academic paper: "Financial Incentives and the 'Performance of Crowds' " (pdf)
    by Winter Mason and Duncan J. Watts at Yahoo! Research aka Yahoo! Labs

    "Currently, several hundred requests may be available on any given day, representing tens of thousands of HITs (i.e. a single request may comprise hundreds or even thousands of individual HITs); thus while AMT is only one particular instantiation of the crowd sourcing model, its size and diversity make it an attractive object of study."

    " we created a task in which participants sorted a set of images taken from a traffic camera at 2-second intervals into chronological order ... Over all conditions, the experiment involved 611 participants, who sorted a total of 36,425 image sets.
    Participants were asked to report their current location, and 594 participants identified 43 different countries. The majority (82.7%) was from the United States, and the next four highest responding countries were India (6.4%), Canada (1.5%), Vietnam (1.2%), and United Kingdom (1%).
    Asked to report their gender, 563 participants responded, and of these 58.8% were female and 41.2% were male.
    Of the 568 reporting age, the average response was 33.3 years and the median age was 31 years.
    Participants were given a choice of five income levels to report, and of the 598 who offered the information, 18.6% reported an income less than $7000, 22.6% reported an income between $7[k]-$30k, 34.5% reported an income between $30[k]-$70k, 21.1% reported an income between $70[k]-$160k, and 3.2% reported an income greater than $160k.
    The subject pool was therefore reasonably diverse, consistent with previous user surveys of the AMT population "

    " we performed another experiment ... changing the task to finding words hidden in a random array of letters ... Over all conditions, 320 participants solved a total of 2736 puzzles, finding 23,440 words.
    Participants were asked to report their current location, and 309 participants identified 19 different countries. The majority (83.9%) was from the United States, and the next four highest responding countries were India (4.8%), Philippines (2.5%), Canada (1.9%), and United Kingdom (1.3%).
    Of the 303 ages reported, the average was 34.6 years and the median age was 32 years.
    Participants were again given a choice of income levels to report, and of the 303 reporting, 6.8% reported an income less than $7000, 30.7% reported an income between $7[k]-$30k, 40.8% reported an income between $30[k]-$70k, 19.7% reported an income between $70[k]-$160k, and 1.9% reported an income greater than $160k.
    These self-reported descriptive statistics, in other words, were generally consistent with those from Study 1, which encourages us to believe that they are reliable.
    A striking difference with Study 1, however, was that of the 290 participants who reported their gender 74.1% were female, as opposed to 58.8%. "
     
    • Like Like x 1
  14. clickhappier ★★Ⰼ₳ՖŦξᚱ⌚ Contributor

    Joined:
    Jul 1, 2014
    Messages:
    5,600
    Likes Received:
    9,527
    September 15, 2009 - whitepaper: "Paid Crowdsourcing, Current State & Progress toward Mainstream Business Use" (pdf) (switched to archive links because this content was removed from Smartsheet's live site in ~Nov 2014, details here)
    by Brent Frei of Smartsheet

    This general report about the crowdsourcing industry includes, on page 6, an unexplained 'estimate' of Amazon Mechanical Turk's number of 'Registered Workers' to be 200,000.

    (The report also oddly places MTurk on a timeline as having started in paid crowdsourcing in 2001; it didn't open to the public until November 2005, and I haven't found any date specifics of its earlier internal-only use by Amazon employees, but that period wouldn't exactly have been 'paid crowdsourcing'.)
     
    • Like Like x 1
  15. clickhappier ★★Ⰼ₳ՖŦξᚱ⌚ Contributor

    Joined:
    Jul 1, 2014
    Messages:
    5,600
    Likes Received:
    9,527
    ~September 2009 - academic paper: "Who are the Turkers? Worker Demographics in Amazon Mechanical Turk" (pdf or pdf)
    by Joel Ross (formerly at University of Puget Sound), Andrew Zaldivar, Lilly Irani (of Turkopticon), and Bill Tomlinson, at University of California, Irvine

    "Launched in 2005, Amazon reports that the system has now more than 200,000 workers registered, and there are about 50,000-100,000 HITs to work on at any given time. MTurk has becoming increasingly popular as a tool for research, being used for performing user studies, image labeling, natural language processing, and relevance evaluation. However, little research has considered the almost entirely anonymous users who are actually performing this work — only a randomly generated ID number represents workers to requesters — and most such previous work has been informal"

    "we created an exploratory survey to gather simple demographic data about the worker population. We also asked users about their Turking habits, such as how often and what kinds of HITs they complete. ... we created an exploratory survey to gather simple demographic data about the worker population. We also asked users about their Turking habits, such as how often and what kinds of HITs they complete. ... The survey was made available at 3pm on a Wednesday, and left open for a full seven days."

    "While this literature has suggested the use of either qualification pre-tests and/or explicitly verifiable questions, these were not appropriate for the current study — qualification tests would exclude a portion of the Turker population, and there was no way of verifying user demographics. Indeed, although MTurk HITs default to being answerable only by users with at least a 95% approval rating (meaning that 95% of the worker's submitted HITs have been approved by the requester of the work), we removed all restrictions for accepting the survey — allowing any worker to complete the HIT — in order to reach as broad a user population as possible."

    " The survey was available as a HIT for a full week. During this time, 573 people submitted valid completed surveys (three respondents reported to be under 18, and are not included in the results presented here).
    According to our survey results, 57% of MTurk workers are from the United States, while 32% are from India — the remaining respondents are from countries ranging from Australia to Ukraine.
    Respondents reported an average age of 31 years old (min 18, max 71, median 27), and the majority of respondents (55%) are female.
    More than half (66%) of respondents have a college or advanced degree, and 33% are either full- or part-time students.
    While 38% of respondents are employed full-time, nearly a third (31%) are currently unemployed. The median annual reported income was between $20,000 and $30,000.
    These demographics reveal a significantly international and highly educated population, though one with lower levels of employment and income. Indeed, 18% of Turkers reported sometimes or always relying on MTurk to "make basic ends meet" (Figure 4). While only a minority of workers rely on the pay earned from completing HITs, they still make up a significant percentage of the Turker population. "

    "MTurk users are relatively new to the system, with most (69%) Turking for less than 6 months. The average respondent spends about 8 hours per week performing HITs and earns around $10 during that time (see Figure 3). Turkers earn less than U.S. minimum wage, but are not generally completing HITs in such a way as to constitute a full-time job: Turking seems to be a part-time activity for most users. This effect could be a result of the relatively low levels of pay — Turkers might Turk more if they made more money from it. Furthermore, this level of payment may still be substantial to some — the less than minimum-wage Turk earnings may mean the difference between paying a bill or not."

    "Notably, a majority (52.9%) of users reported complete surveys more often than other types of HITs (anecdotally, surveys do not appear to be more common than other types of HITs). ... comparing each factor from those who mostly complete surveys and those who mostly perform other tasks finds that this difference is not significant across many of the factors examined in this survey. Age, gender, education, and employment all have similar distributions no matter which task type respondents prefer. But there was a strongly significant difference in other categories: Turkers who answer surveys work less in MTurk, earn less in MTurk, have higher incomes, rely on MTurk less, and are more likely to be from the US. ... Yet because our survey paid more than the reported average earnings ... we likely attracted a wider sample population."

    "The requirements of an Internet connection and English language skills restrict the potential range of MTurk workers. The results somewhat matched those in [Panos Ipeirotis's March 2008 blog post], as we also find that "the profile of the typical Turker is not of a person that completes tasks for a living in a developing country." However, we find a much more international population than in [Panos Ipeirotis's March 2008 blog post], with a greater number of users from India. This may indicate that MTurk has gained substantially more international members in the 18 months since this previous survey was performed."

    "Compared to the entire population, MTurk workers from the U.S. are younger (median age 30 vs. 36.6), much more highly educated (63% vs. 25% with college degrees), and include a significantly greater number of female members (69% vs. 51%). Turkers also tend towards lower levels of annual income."

    "Turkers from India, on the other hand, are much more often male (69%), even younger (median age 25), and even more highly educated (74% have a college degree or higher). Although they have slightly lower unemployment levels than the sample as a whole (26% vs. 31%), they are almost twice as likely to report themselves as relying on the income from MTurk (29% vs. 18%)."

    [charts converted to text:]

    "Figure 1. Nationality and gender of MTurk workers."

    Nationality:
    U.S.: 57%
    India: 32%
    Canada: 3%
    Philippines: 1%
    U.K.: 1%
    Pakistan: 0.5%
    Romania: 0.5%
    Other: 5%

    Gender:
    Female: 55%
    Male: 45%

    "Figure 2. Demographics of MTurk workers."

    Age:
    18-24 yrs: 40%
    25-30 yrs: 22%
    31-40 yrs: 19%
    41-50 yrs: 11%
    51-60 yrs: 8%
    60+ yrs: 1%

    Education:
    High School: 12%
    Some College: 21%
    Associates: 8%
    Bachelor's: 42%
    Advanced: 16%

    Student:
    [Non-Student: 67%]
    Part Time: 12%
    Full Time: 21%

    Employment:
    Unemployed: 31%
    Part Time: 31%
    Full Time: 38%

    Income [per year]:
    <$10k/yr: 27%
    $10k-$20k/yr: 15%
    $20k-$30k/yr: 12%
    $30k-$40k/yr: 12%
    $40k-$50k/yr: 9%
    $50k-$60k/yr: 5%
    $60k-$70k/yr: 4%
    $70k+/yr: 14%

    "Figure 3. Reported system usage statistics"

    Time Using System:
    Just started: 13%
    <1 month: 23%
    1-6 months: 34%
    6-12 months: 15%
    1-2 years: 13%
    2+ years: 3%

    Activity Per Week:
    <1 hr/week: 10%
    1-5 hrs/week: 46%
    5-15 hrs/week: 25%
    15-30 hrs/week: 11%
    30+ hrs/week: 7%

    Earnings Per Week:
    <$0.25/week: 6%
    $0.25-$1/week: 20%
    $1-$5/week: 38%
    $5-$25/week: 27%
    $25-$50/week: 7%
    $50+/week: 3%

    "Figure 4. Reported reliance on money earned on MTurk."

    The money on MTurk is:
    [Level 1] irrelevant to me: 9%
    [Level 2] nice, but doesn't materially change my circumstances: 41%
    [Level 3] a way for me to pay for nice extras: 32%
    [Level 4] sometimes necessary to make basic ends meet: 10%
    [Level 5] always necessary to make basic ends meet: 8%

    "we hope that this survey helps to give some insight into the humans who enable this "artificial artificial intelligence." A significant portion of these workers rely on the pennies they earn performing human computation, some treating the system as a full-time (if low-paying) job. Being aware of the circumstances of these anonymous workers is important if ... researchers ... are to act responsibly toward them as we conduct our research. MTurk may present itself as a form of AI, but behind it are real and potentially vulnerable people."
     
    Last edited: Feb 9, 2015
    • Like Like x 1
  16. clickhappier ★★Ⰼ₳ՖŦξᚱ⌚ Contributor

    Joined:
    Jul 1, 2014
    Messages:
    5,600
    Likes Received:
    9,527
    December 13, 2009 - Greg Little at Massachusetts Institute of Technology wrote a blog post: "How many turkers are there?"

    "it would be nice to know how many individual turkers are out there. We don’t have the answer, but we have run lots of experiments, and we can aggregate all of our historical data ... based on 4,449 HITs, with a total of 28,168 assignments. Most of these were posted over a 75-day period. ... We had work done by 1,496 individual turkers. ... This is a classic Power Law distribution. We even see something close to the 80-20 rule: 80% of assignments are completed by the top 22% of turkers. The last 25% of turkers completed 1 assignment each. So, it looks like there may be relatively few active turkers out there doing most of the work."
     
    • Like Like x 1
  17. clickhappier ★★Ⰼ₳ՖŦξᚱ⌚ Contributor

    Joined:
    Jul 1, 2014
    Messages:
    5,600
    Likes Received:
    9,527
    January 5, 2010 - Raphael Mudge of Feedback Army wrote a blog post "Be a Good Turk Boss – Mechanical Turk Tips", that included this observation from his experience of posting website-testing HITs, for ~2 years at the time:

    "Just like E-Bay has a seller rating– Turkers have a task approval rating. You can set a minimum approval rating for your tasks. 95% or above is near impossible for workers to keep. I tried 85% for a while but eventually lowered it to 70% to open the field to more workers. I have not noticed a change in quality from doing this."


    This opinion is echoed by Michael Buhrmester at University of Texas in his "Amazon Mechanical Turk Guide for Social Scientists" webpage, last updated in January 2012:

    "If you do not wish to place a location requirement but want to avoid potentially sub-par work, the approval rating function is your best shot. The approval rating is calculated for each worker and is the percentage of approved submissions divided by the worker’s total submissions. So if a worker has 3 approved submissions but 1 rejected submission, he/she would have a rating of 75%. MTurk recommends a 95% approval rating. I’m not sure how they decided on that number – perhaps it’s p-value inspired? I’ve personally moved the approval rating around between 50-99 and at least in my experience, the higher rating requirement seems to slow down the flow of incoming submissions without affecting data quality, but I’ve done no formal test of this."

    Note: Until a worker has at least 100 HITs approved, MTurk apparently acts as if they have a 100% approval rating for qualification purposes, to give them a chance to recover from an early rejection more easily.


    On the other hand, Eyal Peer at Bar-Ilan University, Israel, wrote a blog post on Dec 11, 2013: "Reputation as a Sufficient Condition for High Data Quality on MTurk", about a paywalled paper by the same title (co-authored with Joachim Vosgerau and Alessandro Acquisti):

    "I gave Turkers the opportunity to cheat by over-reporting lucky coin tosses, and found very low levels of cheating ... [but] as I usually do, sampled only “high reputation” Turkers (those who had more than 95% of previous HITs approved). I was more surprised to see that when I removed this restriction and allowed any Turker to take part in my study, cheating went up considerably."

    "my colleagues and me have been mostly relying on attention-check questions (henceforth, ACQs) to ensure high quality data. ... Using ACQs have inherent disadvantages ... perhaps most important, even when failing ACQs can be considered a reliable signal that the participant has not paid attention in the rest of the survey, a sampling bias might be created if those who fail ACQs are excluded post-hoc from the sample. On the other hand, not using ACQs and sampling only high reputation workers can also restrict the sample’s size, reduce response rate and might also create a response bias."

    "we found that high reputation workers did provide higher quality data on all measures: they failed common ACQs very rarely, their questionnaires’ reliability was high and they replicated known effects ... whether or not these workers received ACQs. Namely, high reputation workers who did not receive any ACQ showed similarly high quality data compared to those who did receive (and pass) our ACQs. Low reputation workers, on the other hand, failed ACQs more often and their data quality was much lower. However, ACQs did make a difference in that group. Those who passed the ACQs provided higher data quality, sometimes very similar to the quality of data obtained from high reputation workers. So, it seemed that ACQs were not necessary for high reputation workers. Moreover, sampling only high reputation workers did not reduce response rate (in fact, it increased it) and we couldn’t find any evidence for a sampling bias, as their demographics were similar to those that had low reputation."

    "However, we used pretty common ACQs ... We thus ran another study with novel ACQs, and found similar results: high reputation workers produced high quality data with or without ACQs, even when the ACQs were not familiar to them. We also found that, among the high reputation workers, those who have been more productive on MTurk (completed more HITs in the past) produced slightly higher data quality than less productive workers. But even the less productive high-reputation workers produced very high data quality, with or without ACQs."
     
    • Like Like x 1
  18. clickhappier ★★Ⰼ₳ՖŦξᚱ⌚ Contributor

    Joined:
    Jul 1, 2014
    Messages:
    5,600
    Likes Received:
    9,527
    February 27, 2010 - John Horton of CrowdFlower wrote a blog post: "Why People Participate on Mechanical Turk, Now as a Mosaic Plot"

    " "Who are these people?" and "Why do they participate?" are two perennial questions about AMT. ... NYU Prof. Panos Ipeirotis asked a bunch of workers their reasons [in 2008] and tabulated the responses here [summarized, and here with full quoted responses]."

    "For a joint project with Dave Rand and Richard Zeckhauser [at Harvard University], we asked ~400 AMT workers both (a) where they are from and (b) the primary reason they participate on AMT [Have fun, Learn new skills, or Make money]. Because economic opportunities differ by country, we might expect that motivation and behavior should also differ by country. The cross tabulation plot is below (reasons are in the "rows", countries in the "columns" -- the size of each rectangle is proportional to the number of responses in that cell):"

    "Two things to note: 1) Money is a big motivation for everyone. 2) Money aside, people from India are there to learn; people from the US are there to have fun. Although the India/US differences are consistent with the different-countries/different-motivations hypothesis, the most relevant fact is the unconditional importance of money."

    [the original chart:]
    [​IMG]

    [chart approximated as text, calculated from pixel measurements of the areas:]
    Code:
    	           fun   skills  money   share of total
    	U.S.:    ~13.2%   ~2.8%  ~84.0%      ~40.3%
    	India:    ~3.9%  ~12.9%  ~83.2%      ~43.7%
    	Europe:  ~24.7%   ~8.8%  ~66.5%       ~8.2%
    	Other:   ~17.3%   ~8.8%  ~73.9%       ~7.8%
    	
    [Note that workers in countries other than the U.S. and India can only use their earnings to buy things on Amazon.com.]

    [my colorized version of the chart, labeled with the approximated percents:]
    [​IMG]
     
    • Like Like x 1
  19. clickhappier ★★Ⰼ₳ՖŦξᚱ⌚ Contributor

    Joined:
    Jul 1, 2014
    Messages:
    5,600
    Likes Received:
    9,527
    March 10, 2010 - academic paper: "Demographics of Mechanical Turk" (pdf, xlsx data)
    by Panos Ipeirotis at New York University

    This is an update of the surveys reported in his March 19, 2008 blog post "Mechanical Turk: The Demographics", and his September 12, 2008 blog post "How Much Turking Pays?", among others.

    "Amazon.com, until recently, was paying in cash only [to] workers that had a bank account in the US. All other workers could still participate on Amazon Mechanical Turk but the only way to be compensated was to get paid using an Amazon.com gift card. This policy discouraged workers from other countries, and the majority of the workers were from the United States (typically 70%-80% of the participants ...). Extensive surveys, conducted by multiple parties, confirmed that finding. The main findings of earlier surveys indicated that Mechanical Turk workers are relatively representative of the population of US Internet users ... Recently, however, Amazon.com allowed workers in India to receive their payment in Indian rupees ... This changed significantly the population dynamics on Mechanical Turk" [Amazon actually started offering Indian workers payment by check in rupees in ~Feb 2007 (officially publicized in May 2007), but apparently it took a while to grow in popularity in India.]

    "For this reason, we conducted a new survey, collecting demographics of 1,000 Mechanical Turk users. The survey was conducted over a period of three weeks, in February 2010 ... In total, we got participants from 66 countries. The results indicated that still the majority of users are from the United States, but a significant majority is now coming from India. Specifically, we have the following breakdown across countries:"
    United States: 46.8%
    India: 34.0%
    Miscellaneous: 19.2%

    [Full list of countries, generated from provided xlsx file; there are 64 countries, plus '(blank)', represented; the top 9 (five tied for 10th) are displayed, the other 55 are in a spoiler.]
    United States: 468, 46.8%
    India: 340, 34.0%
    Canada: 23, 2.3%
    Romania: 22, 2.2%
    Philippines: 20, 2.0%
    Spain: 7, 0.7%
    Italy: 6, 0.6%
    Netherlands: 6, 0.6%
    United Kingdom: 6, 0.6%
    (blank): 5, 0.5%
    Argentina: 4, 0.4%
    Australia: 4, 0.4%
    China: 4, 0.4%
    Croatia: 4, 0.4%
    Israel: 4, 0.4%
    Bangladesh: 3, 0.3%
    Germany: 3, 0.3%
    Malaysia: 3, 0.3%
    Pakistan: 3, 0.3%
    Serbia: 3, 0.3%
    United Arab Emirates: 3, 0.3%
    Venezuela: 3, 0.3%
    Finland: 2, 0.2%
    France: 2, 0.2%
    Greece: 2, 0.2%
    Ireland: 2, 0.2%
    Japan: 2, 0.2%
    Lithuania: 2, 0.2%
    Morocco: 2, 0.2%
    New Zealand: 2, 0.2%
    Poland: 2, 0.2%
    Russia: 2, 0.2%
    Turkey: 2, 0.2%
    Ukraine: 2, 0.2%
    Uruguay: 2, 0.2%
    Aruba: 1, 0.1%
    Austria: 1, 0.1%
    Bahrain: 1, 0.1%
    Barbados: 1, 0.1%
    Belgium: 1, 0.1%
    Brazil: 1, 0.1%
    Bulgaria: 1, 0.1%
    Cayman Islands: 1, 0.1%
    Czech Republic: 1, 0.1%
    Dem. Rep. of Congo: 1, 0.1%
    Guatemala: 1, 0.1%
    Hungary: 1, 0.1%
    Iceland: 1, 0.1%
    Indonesia: 1, 0.1%
    Jamaica: 1, 0.1%
    Lebanon: 1, 0.1%
    Macau: 1, 0.1%
    Mexico: 1, 0.1%
    Nepal: 1, 0.1%
    Peru: 1, 0.1%
    Portugal: 1, 0.1%
    Macedonia: 1, 0.1%
    Saint Lucia: 1, 0.1%
    Singapore: 1, 0.1%
    Slovenia: 1, 0.1%
    South Africa: 1, 0.1%
    Sweden: 1, 0.1%
    Switzerland: 1, 0.1%
    Taiwan: 1, 0.1%
    U.S. Virgin Islands: 1, 0.1%
    [All subsequent data was presented as charts without data value labels; I have recalculated the values using the provided xlsx file, and included the overall US+India average values.]

    "Across US-based workers, there are significantly more females than males, while the situation is reversed for Indian workers. ... Most participants in the US use Mechanical Turk as a supplementary source of income, and often Mechanical Turk is used by stay-at-home parents, unemployed and underemployed workers, and so on. Since females are more likely to fit into these categories, there is a corresponding increase in representation. On the contrary, more Indian workers treat Mechanical Turk as a primary (or at least significant) source of income, and we see more males working on Mechanical Turk."

    Gender
    Code:
    	        India   US    Overall
    	Female  30.0%  65.6%   50.6%
    	Male    70.0%  34.4%   49.4%
    	
    "there is definitely an overrepresentation of younger workers, compared to the general population of Internet users. While this holds both for the US and for India, we see an even higher skew towards younger workers among Indians."

    Year of Birth
    Code:
    	           India   US    Overall
    	1925-1929   0.0%   0.2%    0.1%
    	1930-1934   0.0%   0.0%    0.0%
    	1935-1939   0.0%   0.0%    0.0%
    	1940-1944   0.0%   1.3%    0.7%
    	1945-1949   0.6%   3.0%    2.0%
    	1950-1954   0.0%   4.3%    2.5%
    	1955-1959   1.8%   7.9%    5.3%
    	1960-1964   2.6%   7.3%    5.3%
    	1965-1969   2.6%   9.4%    6.6%
    	1970-1974   7.9%  11.3%    9.9%
    	1975-1979  13.8%  13.3%   13.5%
    	1980-1984  27.4%  18.2%   22.1%
    	1985-1989  38.5%  18.4%   26.9%
    	1990-1994   4.7%   5.4%    5.1%
    	
    "In general, the (self-declared) educational level of the workers is higher than the general US and Indian population."

    Education Level
    Code:
    	                   India   US    Overall
    	(blank)             1.2%   0.4%    0.7%
    	Some High School    1.2%   1.5%    1.4%
    	High School Grad    8.5%   9.6%    9.2%
    	Some college        9.4%  25.6%   18.8%
    	Associates degree   1.5%   9.0%    5.8%
    	Bachelors degree   52.9%  35.0%   42.6%
    	Masters degree     25.0%  14.3%   18.8%
    	Doctorate degree    0.3%   4.5%    2.7%
    	
    "In the US, the shape of the distribution roughly matches the income distribution in the general US population. However, it is noticeable that the income level of US workers on Mechanical Turk is shifted towards lower income levels. For example, while 45% of the US Internet population has income below $60K/yr, the corresponding percentage across US-based Mechanical Turk workers is 66.7%. (This finding is consistent with the earlier surveys ...) The picture is drastically different across US-based and Indian workers. Workers based in India have significantly lower incomes, as expected, and more than 55% of the workers declared an income of less than $10,000/year."

    Household Income [per year]
    Code:
    	             India   US    Overall
    	(blank)       0.0%   0.4%    0.2%
    	   <$10k     55.3%   8.5%   28.2%
    	 $10k-$15k   17.4%   5.6%   10.5%
    	 $15k-$25k   10.6%  10.5%   10.5%
    	 $25k-$40k    8.8%  20.3%   15.5%
    	 $40k-$60k    3.5%  21.6%   14.0%
    	 $60k-$75k    2.6%  11.3%    7.7%
    	 $75k-$100k   1.5%  11.5%    7.3%
    	$100k-$150k   0.3%   6.2%    3.7%
    	$150k-$200k   0.0%   2.1%    1.2%
    	$200k-$250k   0.0%   0.6%    0.4%
    	$250k-$300k   0.0%   0.0%    0.0%
    	  >$300k      0.0%   1.3%    0.7%
    	
    "In terms of marital status and household size, the answers tend to match the age demographic of the workers reported earlier. The majority of the workers, both in India and in the US, do not have children, and a significant fraction of them are single. An interesting contrast is the household size, which seems more to reflect cultural norms than anything specific to Mechanical Turk: While more Indian workers are single and without children, they seem to stay in houses with larger number of household members, compared to US workers: Indian workers either stay with their family, or they tend to have a comparatively larger number of roommates, compared to US workers."

    Marital Status
    Code:
    	              India   US    Overall
    	single        56.8%  36.5%   45.0%
    	cohabitating   0.0%   8.3%    4.8%
    	engaged        2.1%   3.2%    2.7%
    	married       40.9%  42.9%   42.1%
    	separated      0.0%   1.9%    1.1%
    	divorced       0.3%   6.8%    4.1%
    	widowed        0.0%   0.2%    0.1%
    	
    Number of Children [separated from marital status for clarity/simplicity; the vast majority of those with children selected 'married']
    Code:
    	                 India   US    Overall
    	(blank)           0.3%   0.0%    0.1%
    	No [none]        68.2%  60.0%   63.5%
    	Yes, 1 child     17.6%  12.4%   14.6%
    	Yes, 2 children  11.8%  16.9%   14.7%
    	Yes, 3 children   2.1%   6.8%    4.8%
    	Yes, 4+ children  0.0%   3.8%    2.2%
    	
    Household Size
    Code:
    	        India   US    Overall
    	(blank)	 1.2%   0.9%    1.0%
    	1        2.1%  20.1%   12.5%
    	2       12.1%  33.5%   24.5%
    	3       23.8%  18.4%   20.7%
    	4       38.5%  15.4%   25.1%
    	5+      22.4%  11.8%   16.2%
    	
    Race [wasn't displayed in original report]
    Code:
    	        India   US    Overall
    	Asian   90.6%   8.5%   43.1%
    	Black    2.1%   3.4%    2.8%
    	White    1.8%  81.2%   47.8%
    	Other    5.3%   6.8%    6.2%
    	(blank)  0.3%   0.0%    0.1%
    	
    "We also asked a set of questions for evaluating the level of engagement of Mechanical Turk workers on the marketplace. Since we did not detect significant deviations across countries, we will be reporting the results in aggregate form, without separating by country of origin of the worker. [When recalculating, I went ahead and separated by country.] In general most workers spend a day or less per week working on Mechanical Turk, and tend to complete 20-100 HITs per week. Correspondingly, this generates a relatively low income stream for Mechanical Turk work, which is often less than $20 per week. Of course, there are a few workers that devote a significant amount of time and effort, completing thousands of HITs, and generating a respectable income of more than $1000/month. For these workers, Mechanical Turk tends to be the primary source of income, of course."

    MTurk Tenure [wasn't displayed in original report]
    Code:
    	             India   US    Overall
    	(blank)       0.3%   0.6%    0.5%
    	 0-3 months  55.6%  39.1%   46.0%
    	 3-6 months  19.1%  16.7%   17.7%
    	6-12 months  14.7%  19.0%   17.2%
    	 1-2 years    8.2%  18.4%   14.1%
    	 2-4 years    2.1%   6.2%    4.5%
    	
    Time spent on Mechanical Turk per week
    Code:
    	                India   US    Overall
    	(blank)          0.3%   0.6%    0.5%
    	 <1 hour/wk      0.9%   6.2%    4.0%
    	 1-2 hours/wk    8.5%  20.3%   15.3%
    	 2-4 hours/wk   17.4%  22.6%   20.4%
    	 4-8 hours/wk   24.7%  24.8%   24.8%
    	 8-20 hours/wk  27.9%  16.9%   21.5%
    	20-40 hours/wk  16.2%   6.2%   10.4%
    	 >40 hours/wk    4.1%   2.4%    3.1%
    	
    Number of HITs completed per week
    Code:
    	                  India   US    Overall
    	(blank)            0.3%   0.4%    0.4%
    	   <1 HIT/wk       0.0%   0.9%    0.5%
    	   1-5 HITs/wk     3.8%   4.7%    4.3%
    	   5-10 HITs/wk    7.4%   9.4%    8.5%
    	  10-20 HITs/wk   14.1%  13.2%   13.6%
    	  20-50 HITs/wk   22.1%  15.8%   18.4%
    	  50-100 HITs/wk  18.2%  18.2%   18.2%
    	 100-200 HITs/wk  12.9%  11.8%   12.3%
    	 200-500 HITs/wk  11.8%  14.3%   13.2%
    	 500-1000 HITs/wk  5.3%   5.8%    5.6%
    	1000-5000 HITs/wk  3.8%   4.9%    4.5%
    	  >5000 HITs/wk    0.3%   0.6%    0.5%
    	
    Weekly Income from Mechanical Turk
    Code:
    	             India   US    Overall
    	(blank)       0.9%   0.9%    0.9%
    	  <$1/wk     10.6%   9.2%    9.8%
    	  $1-$5/wk   38.8%  32.9%   35.4%
    	  $5-$10/wk  20.0%  22.4%   21.4%
    	 $10-$20/wk  15.9%  16.5%   16.2%
    	 $20-$50/wk   9.7%  11.3%   10.6%
    	 $50-$100/wk  2.6%   4.5%    3.7%
    	$100-$200/wk  1.2%   1.5%    1.4%
    	$200-$500/wk  0.3%   0.6%    0.5%
    	  >$500/wk    0.0%   0.2%    0.1%
    	
    " To understand better why people participate on Mechanical Turk, we asked for both qualitative (i.e., free text) and a set of structured questions. The main structured question that we asked was the following:
    Why do you complete tasks in Mechanical Turk? Please check any of the following that applies:
    • [fruitful] Fruitful way to spend free time and get some cash (e.g., instead of watching TV)
    • [primary income] For "primary" income purposes (e.g., gas, bills, groceries, credit cards)
    • [secondary inc] For "secondary" income purposes, pocket change (for hobbies, gadgets, going out)
    • [kill time] To kill time
    • [entertainment] I find the tasks to be fun
    • [unemployed/pt] I am currently unemployed, or have only a part time job
    The answers were quite different across Indian and US workers. Very few Indian workers participate on MTurk for "killing time", and significantly more Indians treat MTurk as a primary source of income. (Not surprising given the average income level of an Indian worker vs the income level of the US workers.) "

    Motivation [combination of what was originally presented as 6 charts]
    Code:
    	                India   US    Overall
    	kill time        5.0%  32.7%   21.0%
    	entertainment   20.0%  40.6%   31.9%
    	fruitful        59.1%  69.4%   65.1%
    	secondary inc   37.1%  61.5%   51.2%
    	primary income  27.4%  14.1%   19.7%
    	unemployed/pt   28.2%  30.8%   27.9%
    	

    Panos Ipeirotis also presented the same information in a March 9, 2010 blog post: "The New Demographics of Mechanical Turk"
     
    Last edited: Jan 18, 2015
    • Like Like x 1
  20. clickhappier ★★Ⰼ₳ՖŦξᚱ⌚ Contributor

    Joined:
    Jul 1, 2014
    Messages:
    5,600
    Likes Received:
    9,527
    My (clickhappier) addendum to Panos Ipeirotis's March 2010 "Demographics of Mechanical Turk", with details of the top 10% 'Super Turkers'

    I calculated some additional cross-tabulated data from his xlsx file for approximately the top 10% most productive of the active workers (referred to in Jesse Chandler et al's March 2014 study as the 'Super Turkers'), according to reported average HITs per week (500-1000, 1000-5000, and >5000 HITs/wk, totaling 97 workers out of the 1000 surveyed). I agree with Karen Fort et al's 2011 paper that the currently active accounts are likely about 10% of the total registered, making the top 10% of the active accounts be about the top 1% of accounts ever registered, for a '90-9-1' overall relationship.


    Country for Workers Performing 500+ HITs per week
    United States: 53, 54.6%
    India: 32, 33.0%
    Romania: 2, 2.1%
    Bulgaria: 1, 1.0%
    Canada: 1, 1.0%
    France: 1, 1.0%
    Germany: 1, 1.0%
    Netherlands: 1, 1.0%
    Turkey: 1, 1.0%
    U.S. Virgin Islands: 1, 1.0%
    United Arab Emirates: 1, 1.0%
    United Kingdom: 1, 1.0%
    (blank): 1, 1.0%

    Gender for Workers Performing 500+ HITs per week
    Female: 49, 50.5%
    Male: 48, 49.5%

    Year of Birth for Workers Performing 500+ HITs per week
    1945-1949: 2, 2.1%
    1950-1954: 1, 1.0%
    1955-1959: 8, 8.2%
    1960-1964: 8, 8.2%
    1965-1969: 10, 10.3%
    1970-1974: 7, 7.2%
    1975-1979: 15, 15.5%
    1980-1984: 19, 19.6%
    1985-1989: 22, 22.7%
    1990-1994: 5, 5.2%

    Education Level for Workers Performing 500+ HITs per week
    Some High School: 4, 4.1%
    High School: 8, 8.2%
    Some college: 20, 20.6%
    Associates degree: 9, 9.3%
    Bachelors degree: 39, 40.2%
    Masters degree: 16, 16.5%
    Doctorate degree: 1, 1.0%

    Household Income per year for Workers Performing 500+ HITs per week
    <$10k: 22, 22.7%
    $10k-$15k: 12, 12.4%
    $15k-$25k: 13, 13.4%
    $25k-$40k: 11, 11.3%
    $40k-$60k: 16, 16.5%
    $60k-$75k: 7, 7.2%
    $75k-$100k: 8, 8.2%
    $100k-$150k: 5, 5.2%
    $150k-$200k: 3, 3.1%

    Marital Status for Workers Performing 500+ HITs per week
    single: 44, 45.4%
    cohabitating: 8, 8.2%
    engaged: 1, 1.0%
    married: 41, 42.3%
    separated: 0, 0.0%
    divorced: 3, 3.1%
    widowed: 0, 0.0%

    Number of Children for Workers Performing 500+ HITs per week
    No [none]: 62, 63.9%
    Yes, 1 child: 12, 12.4%
    Yes, 2 children: 13, 13.4%
    Yes, 3 children: 9, 9.3%
    Yes, 4+ children: 1, 1.0%

    Household Size for Workers Performing 500+ HITs per week
    (blank): 1, 1.0%
    1: 12, 12.4%
    2: 21, 21.6%
    3: 20, 20.6%
    4: 17, 17.5%
    5+: 26, 26.8%

    (Note: the 3, 4, & 5+ person household sizes are each about half US, half India; and comparing to the children numbers shows a lot of these must be roommates or extended family.)

    Race for Workers Performing 500+ HITs per week
    Asian: 32, 33.0% (26 in India)
    Black: 1, 1.0%
    White: 57, 58.8%
    Other: 6, 6.2%
    (blank): 1, 1.0%

    MTurk Tenure for Workers Performing 500+ HITs per week
    0-3 months: 29, 29.9%
    3-6 months: 21, 21.6%
    6-12 months: 14, 14.4%
    1-2 years: 23, 23.7%
    2-4 years: 10, 10.3%

    MTurk Weekly Income for Workers Performing 500+ HITs per week
    $1-$5/wk: 2, 2.1%
    $5-$10/wk: 4, 4.1%
    $10-$20/wk: 23, 23.7%
    $20-$50/wk: 36, 37.1%
    $50-$100/wk: 18, 18.6%
    $100-$200/wk: 11, 11.3%
    $200-$500/wk: 3, 3.1%

    MTurk Weekly Time for Workers Performing 500+ HITs per week
    1-2 hours/wk: 2, 2.1%
    2-4 hours/wk: 4, 4.1%
    4-8 hours/wk: 12, 12.4%
    8-20 hours/wk: 33, 34.0%
    20-40 hours/wk: 32, 33.0%
    >40 hours/wk: 14, 14.4%

    Motivation for Workers Performing 500+ HITs per week
    kill time: 19, 19.6%
    entertainment: 26, 26.8%
    fruitful: 57, 58.8%
    secondary inc: 45, 46.4%
    primary income: 30, 30.9%
    unemployed/pt: 35, 36.1%
     
    Last edited: Jan 18, 2015
    • Like Like x 2

Share This Page