Guideline for academic requesters on mturk

Discussion in 'AMT Feedback' started by niloufar, Jul 25, 2014.

  1. niloufar New Member Member

    Joined:
    Jul 24, 2014
    Messages:
    12
    Likes Received:
    72
    Hi,

    I'm Niloufar Salehi a phd student at Stanford. Over the last year I've been studying the various issues that crowdworkers face and areas that we can improve. I've recently had some interesting conversations with turkers who are thinking about putting together a guide for researchers on MTurk. Stuff like what ethical and payment concerns they should have. I want to know if people on this forum are interested in collaborating with us. We're starting a brainstorm so main questions are, what would you add to such a guideline? Do you have any ideas on how to ensure researchers respect your guidelines?

    This is a good example of the kind of positive effects that research requesters can have on the community: http://mturkforum.com/showthread.php?15802-Can-t-Find-Many-HITs-7-15-Temptatious-Tuesday!/page149
    I believe a majority of researchers just don't have a clear sense of what they should or shouldn't do, and such a guideline coming directly from the worker community would be very effective.

    Looking forward to hearing you thoughts! You can also PM me here, or email me at: [email protected]
    Thanks!
     
    • Like Like x 10
  2. clickhappier ★★Ⰼ₳ՖŦξᚱ⌚ Contributor

    Joined:
    Jul 1, 2014
    Messages:
    5,600
    Likes Received:
    9,527
    Requester Guidelines Thoughts, part 1 of 2

    Requester Guidelines Thoughts, part 1 of 2


    Here are the thoughts I've put together about things academic requesters on Mechanical Turk should do / know (and generally would apply to non-academic requesters too, if they care). I'm intentionally not 'naming names' when examples of good and bad behavior are described, to not distract from the generalized intent. It is continued in two posts due to the character limit per post.

    • Learn more about the demographics of MTurk workers. Some requesters seem to assume it's okay to pay low rates because they tend to think of the turker workforce as comprised of something along the lines of people in very-low-cost-of-living countries, bored housewives, and well-employed people who don't need the money casually doing a couple of surveys per week or month for fun. Many turkers do fit those descriptions, but studies indicate the vast majority, perhaps 80%, of the HITs completed on MTurk are performed by turkers who are in the top 10% or so of productivity among active turkers, completing hundreds or thousands of HITs per week; and most of those put forth that much effort because the money is very important to them to make ends meet, whether they have other significant sources of income or not. I think the way Amazon refers to the pay as a 'reward' may subtly encourage the assumption that MTurk work isn't an important source of income which is treated like actual work for a significant portion of turkers.

    • Pay fair rates. Many academic requesters' published papers (even well-intentioned ones), and many of the few university IRB (Institutional Review Board) websites that have any specific guidelines about MTurk, have stated matter-of-factly that they paid or recommend paying rates equivalent to $2-$3 or less per hour ($0.03-$0.05 or less per minute) on MTurk, because it's slightly better than or similar to supposed average rates (perpetuating the status quo), and often further try to justify it by mentioning things such as that they think the workers all either don't really need or care about the money or are in low-income countries where this seems like a lot of money. There is a large US university that I've found routinely posts survey HITs for *less than $0.01* per expected minute. There has been a lot of debate in turker forums about what a fair, or even acceptable, rate of pay is or should be. So far, the most commonly mentioned rate among workers as a suggested minimum has been $6/hr ($0.10/min); largely not because this is what workers really prefer and are satisfied with, but as an easy-to-remember figure and a relatively realistic target in light of the preponderance of both academic and non-academic HITs paying much less so far. I don't think there's one answer appropriate for all situations, but here are some points to consider when trying to decide what a fair and ethical rate of pay on your HITs would be for US-based workers:
      • Honest US-based turkers will generally be paying taxes on their MTurk earnings as self-employment.
      • There is a lot of unpaid overhead time involved in turking, including: looking for the next suitable HITs to do, taking uncompensated qualification tests to hopefully qualify for certain HITs, checking reviews for unfamiliar requesters to decide whether to work on their HITs, writing reviews of requesters, communicating with other workers on forums, dealing with some of Amazon's security measures such as periodic Captchas and forced logouts that interrupt workflow (for the workers who haven't yet been semi-randomly awarded Amazon's 'Masters' qualifications that grant freedom from Captchas), dealing with occasional malfunctions of the worker's ISP/browser/computer, communicating with requesters (or in about half the cases, apparently futilely sending messages into a void) about questions/problems/suggestions, keeping track of the work they've done and the payments and bonuses they have or haven't received so far, checking their records of work they've done to see if it's safe to take a survey that threatens rejections if you take it more than once, and more. All break time (even going to the restroom) is also uncompensated.
      • The more specialized knowledge/skills/characteristics, and/or the more stringent the qualifications (such as higher number of HITs approved, higher approval rate, scores on requesters' custom quals, and/or Masters) that your HITs will expect or require, the higher the pay rate for it generally should be if you want to continue to be fair (a fair *minimum* pay rate logically would only be considered as fair for HITs with minimum requirements, just like more-qualified/experienced workers in the traditional workforce generally expect to receive higher pay than less-qualified/experienced workers).
      • Although self-employment work is not legally obligated to comply with minimum wage laws, they are commonly used as a benchmark in evaluating what pay would be fair and ethical.
      • The current national minimum wage has been $7.25/hr (~$0.12/min) since July 2009, due to the final of three gradual tiers of increases that were passed in May 2007.
      • There is currently a movement trying to raise the national minimum wage to $10.10/hr (~$0.17/min), but a bill that would've done that by late 2016 is stalled in Congress due to the political situation. The President was still able to set $10.10/hr as a minimum wage for employees of companies contracting on federal government projects, which will take effect for contracts that are new or renegotiated after Jan 1, 2015.
      • An increasing number of states/territories, and even some cities, have stepped in to raise their own minimum wages higher than the national one. The current highest state/territory minimum wages as of Aug 2014 are $9.00/hr ($0.15/min) in California (increasing to $10.00/hr (~$0.17/min) on Jan 1, 2016), $9.32/hr (~$0.16/min) in Washington state, and $9.50/hr (~$0.16/min) in DC effective July 1, 2014 (increasing to $10.50 (~$0.18/min) on July 1, 2015, to $11.50 (~$0.19/min) on July 1, 2016, and annual inflation-indexed increases thereafter).
      • If increases in the national minimum wage had kept pace with basic inflation of consumer prices since 1968, it should be $10.86 (~$0.18/min) as of 2013.
      • If increases in the national minimum wage had kept pace with nationwide productivity growth in all industries since the 1940s, it should be $16.54 (~$0.28/min) as of 2012.
      • If only considering 'non-farm' productivity growth (i.e. excluding agricultural workers from the calculation), the minimum wage should be $21.75 (~$0.36/min) as of 2012.
      • "If minimum-wage workers received only half of the productivity gains over the period, the federal minimum would be $15.34 [~$0.26/min]. Even if the minimum wage only grew at one-fourth the rate of productivity, in 2012 it would be set at $12.25 [~$0.20/min]."
      • The per-subject costs for other non-MTurk ways researchers can recruit survey participants reportedly tend to be much higher than you would be paying MTurk workers even at much more fair and ethical pay rates than is currently prevalent. Researchers who say they 'don't have the funding' to pay better rates on MTurk should consider that the alternatives are often to pay even more for a participant pool that may be less diverse and in some cases less attentive than MTurk workers (unless you're allowed to recruit local undergrad students for 'course credit' at no cost to you, and don't mind the likely limitations of this participant pool), or not have a study at all. Even in the case of unfunded student projects, please try to consider that the total difference between fair and unfair pay will usually be less than you might think, a drop in the bucket compared to the other costs you've committed to in pursuing your education; even just the textbooks.
      • The availability of non-US workers on MTurk has apparently been gradually decreasing since Amazon stopped accepting registrations of new international worker accounts in late 2012. And with the exception of India (the only country besides the US that has ever been able to receive monetary payment rather than Amazon.com store credit), there were never a lot of workers from any other particular country. So even if you don't specifically require workers to be in the US to accept your HITs, a large and re-growing proportion of them will be in the US.
      • If an amount of pay you expected to be a fair rate turns out not to be because you accidentally underestimated how long your survey would take for reasonably-efficient workers to complete, consider adjusting for this situation by sending bonuses to the workers to make up the difference (could base the bonus amount on the mean or median time to complete, in case a few workers are unusually inefficient). A requester recently did this unexpectedly for workers who took one of their surveys, basing their target pay rate on Washington state's $9.32/hr minimum wage.
      • "Turking is work, even if it is for science, and academic researchers shouldn't assume that people are happy to do it for fun. They should pay and respect people's time." - Dr. Lilly Irani (of Turkopticon), Department of Communication, University of California at San Diego
    • Clearly identify yourself. This ideally should include: the full name/s of the researcher/s responsible for the HIT's project; the university/organization/s they're affiliated with and its state/country; their department name, lab, project group, etc; and any direct contact information you're willing to provide. The more places that more of this information is clearly provided, the better; requester display name, HIT description, HIT content text visible in preview, and survey consent/intro page (in order of increasing amount of information that would be appropriate there). Requesters providing this information up-front benefits both workers and requesters. Workers generally are more willing to take a chance on a requester they're not familiar with (particularly one who hasn't yet been reviewed by any workers on Turkopticon) if they know it is an academic requester, because it is a sign of legitimacy, and because the university 'chain of command' and IRB oversight are one of the few means of recourse workers have if something goes wrong on MTurk. Amazon takes a very hands-off approach to issues workers may have with unfair requesters. I personally use various ways to try to figure out this information for all academic or academic-seeming HITs I do which don't provide it themselves, and am usually (but not always) able to find most or all of it, but this takes time and effort that could be better spent on other things. There was recently an occasion where a fellow worker asked my opinion of a large batch of HITs, that had been posted by a new requester with no Turkopticon reviews and whose only visible identification was their first-name-only requester display name, trying to decide if it was too risky to do more than a few; when I was able to identify the requester's full name and affiliation with a major university, they felt more confident to do a larger quantity of those HITs. And in another recent situation, due to a lack of obvious indications of its identity/legitimacy, an academic research project trying to improve spam-filtering caused concerns for some turkers that it may have been posted by spammers trying to use MTurk to improve their own spam to bypass filters; concerned workers avoided doing the HIT and posted negative reviews and discussion comments.

    • Always use a consent/intro page or paragraphs. It seems many universities currently exempt online surveys from many or all IRB requirements, or at least exempt online studies from certain departments and/or which don't cover certain sensitive topics. Even if your university doesn't force you to, it's always a good idea to use a consent/intro page at the beginning of a survey, and/or paragraphs in the HIT content text for non-survey tasks, to: clearly identify yourself (as covered above); clearly state the pay to be expected (and make sure this statement of pay matches what the HIT is currently posted for; I've seen a number of consent pages state a higher pay than the HIT's actual current MTurk pay, and the vast majority of the time the researchers have no interest in making up the difference with a bonus), and how soon approval can be expected; clearly state any possible bonuses and/or follow-up studies you may qualify for, and how soon their issuance can be expected; state the number of minutes you expect it to take a worker to complete the study; state any reasons for which you plan to automatically reject submissions; and state a title for the study and however much description of it you reasonably can without compromising it.

    • Provide reasonable time estimates and limits. Clearly state up-front a fair expectation of how long it will take for someone who's not already familiar with it to thoroughly read and answer everything in your survey or task. Err on the side of overestimation, to avoid disappointment/frustration if it takes someone more than a little longer than the estimated time, a situation which can encourage some workers to rush through the rest of it to reduce the decline in their effective pay rate this would be causing. Other workers will return the survey in protest, losing out on all the compensation, if it's taking much longer than the stated time estimate. Displaying an accurate progress bar as workers move through the survey helps them know when they're nearing the end.
      - Set the 'Time Allotted' limit for your HITs to an amount of time much longer than the expected amount of time needed to complete the survey or task. Workers like to have leeway in case your time expectation was underestimated, and to have time available if needed to deal with interruptions that occasionally come up, like ISP/browser malfunctions, restroom breaks, phone calls, visitors or family members needing attention, and such.

    Continued...
     
    • Like Like x 6
  3. clickhappier ★★Ⰼ₳ՖŦξᚱ⌚ Contributor

    Joined:
    Jul 1, 2014
    Messages:
    5,600
    Likes Received:
    9,527
    Requester Guidelines Thoughts, part 2 of 2

    ...continued.


    • Be clear about bonuses. If a bonus is offered, as mentioned above, state as clearly as possible what the potential amount will be (or range of possible amounts and expected/past median or mean) and how to earn it, and how soon workers should expect it to be paid. Pay in as timely a manner as possible. When requesters send out bonuses, the only information workers receive about the bonus is an email from MTurk containing the requester's display name (does not include the unique requester ID#), a 'HIT' ID that is meaningless to workers (representing the worker's unique HIT assignment, not the HIT group), and whatever comment message the requester chooses to provide. Many provide no comment at all (resulting in a message that says "No comment provided by Requester."), or a minimally-informative comment, leaving the worker to try to figure out what the bonus was from. On a number of occasions, I've seen workers have to ask other workers if anyone remembers what a bonus they received might have been from, and how it was determined. Workers are always glad to receive bonuses at all, but ideally your comment should clearly state the title of the HIT (and the topic of the study if this wasn't stated in the HIT title; some just say generic things like "Take a quick survey"), state the date the worker completed the relevant HIT/s or the range of dates the HIT/s was available, and briefly re-explain how the bonus was earned/calculated.
      - If doing a random bonus lottery/drawing/sweepstakes, be aware that some workers are skeptical about these, since it would be easy for a requester to never award one to anyone, and the workers would never know. This concern can't be entirely averted, but it helps if you clearly state the number of participants that you plan to recruit in this pool, the number and dollar amount of bonuses you will be awarding, and as specifically as possible when you plan to be awarding the bonuses (and stick to it). I haven't yet seen a requester do this, but I also had the idea that requesters could consider sending a small bonus (even as little as $0.01, but more would be nice of course) to everyone else when the big bonus/es are awarded, as a both a small consolation prize and a notification that the lottery has concluded, so workers know that at least the requester didn't just forget about it.

    • Compensate for qualifier/screener surveys. Some requesters want to determine if workers fit specfic criteria/demographics for their survey, without revealing those criteria in advance to potentially bias the answers. Some requesters handle this by expecting workers to take a qualifier/screener survey HIT that pays $0.00, or telling them to return the main survey HIT unpaid if they don't match the initial screener's criteria. Instead consider handling it like this: Post a qualifier survey for a small but appropriate fee for the time needed to complete it. Pay that fee to everyone who completes the qualifier survey. For people who fit the criteria you're looking for, either immediately redirect them to the full survey and pay them a bonus appropriate for the additional time needed to complete the full survey (both the amount of the bonus and the time the full survey will take should be clearly stated up-front); or else assign a custom qualification to the workers who fit the criteria, and tell them to take the full survey in another HIT that requires that custom qualification.

    • Avoid completion code malfunctions. Make sure your survey will actually provide the promised completion code to workers who complete it; this is a problem turkers encounter quite frequently. When the code is provided, clearly state it on a separate line by itself rather than buried in the midst of a paragraph of more text, and ideally in a different color, larger font size, and/or bold formatting. Besides using a static code or generating a random code, another option is to provide a box for workers to type in their own completion code they make up, and tell them to type in the code they chose in the HIT to submit it. If you use randomly-generated codes, make sure they are being accurately recorded in your database; there have been several situations where requesters wrongly rejected large amounts of workers for 'incorrect completion codes' due to a mistake like that.

    • Avoid duplicates/retakes in fair ways. Please don't block workers through MTurk just to prevent retakes! Being blocked by requesters can put a worker's MTurk account at risk of being suspended (banned from all future work on MTurk), based on some rather murky factors that are not presented clearly to workers or requesters. Blocking should generally only be a last resort against an occasional worker who submits such terrible work that they're clearly not trying, but even that situation can be remedied without blocking in some cases by the use of custom quals that you can either assign, revoke, or change the scores on for repeatedly-unsatisfactory workers (and custom quals can be used similarly for preventing accidental survey retakes by decent workers, too).
      - If you only ever post your survey in one HIT group, and just increase the amount of assignments available in it as needed, you can simply configure MTurk to only allow each worker to accept the HIT once. But if your survey will be posted more than once (preferably only do this when the rounds will be several months or at least weeks apart, so workers don't have to keep figuring out if they've done it before or not, over and over again, as it keeps popping back up), and if you don't want retakes, say so up-front - and use one of the several free online retake-prevention methods/services or self-hosted open-source HIT management tools to ensure this. Use the same requester account, HIT title, and HIT description when reposting, if at all possible; and say in the description and preview text when it was previously posted (e.g. "If you took this study in March 2014 or June 2014, please don't try to retake it."). And instead of reposting the same survey in new HIT groups repeatedly just to keep causing your HIT to pop back up at the top of the 'HIT Creation Date (newest first)' list, consider that if you simply pay a fair rate for a well-structured survey, word will be spread for you quickly on worker discussion forums to bring your HIT to the attention of more workers.

    • Avoid other causes of unfair rejections. Make sure your instructions are written very clearly and comprehensively, particularly for batch HIT groups; workers often run into 'edge cases' the requester didn't consider/cover in the instructions, and have to either take a risk and guess how to handle it, return the HIT with no compensation, or contact the requester about it and hope the requester responds before the HIT expires (which usually doesn't happen anywhere near fast enough for that, if at all).
      - If using Attention Check questions (ACs), make sure the 'correct' answers are accurate and not vague/ambiguous; and try not to reject based on missing just one, as there are multiple potential downsides to doing so. Another option to consider: I recently had a HIT where the requester paid a certain base pay amount for everyone who completed the survey, and then promised a bonus for each of the two ACs you got right, which in this case doubled the total pay if both bonuses were earned. And if you set your qual requirements high enough, you might not need to rely on ACs at all to get good data.
      - You can choose not to use some workers' data, without rejecting their work on MTurk. If you want to use 'majority rules' (plurality) to evaluate the data you receive from batches, it's okay to use that for internal analysis if that's what you want, but be very hesitant to actually reject workers' HITs based on 'majority rules' results; many of the better workers try to avoid 'majority rules' HITs, since they will often catch something that other less attentive/experienced/knowledgeable workers might miss, but be rejected for being in the minority.
      - If you do reject workers unfairly, know how to undo it. You only have 30 days from the time the HITs were submitted to reverse their rejections. The workers would prefer to have it done as soon as possible, though, not getting anywhere near that 30-day limit. Some requesters have reportedly taken so long to read and respond to a worker's message about a rejection that the 30-day limit ran out before they got around to trying to address it. Trying to make up for a rejection by issuing a bonus, but without reversing the rejection, leaves workers with a mark counting against them on their 'permanent record' at MTurk that may take them below a qualification threshold necessary for certain other HITs.

    • Don't violate workers' trust and the MTurk Terms of Service. Don't require workers to provide personally identifying information to complete your HITs; common problems include asking for email addresses (requesters can use MTurk to send messages to workers without having the workers' email addresses), exact birthdates (year alone, or month and year, should be sufficient), or real names. Don't require workers to register on sites that require this kind of information to complete your HITs. If you have a project that requires workers to register on a special site the requester set up just for the HIT, let workers use their MTurk worker ID# or a username of their choice as the unique login identifier, instead of unnecessarily expecting an email address be provided for this purpose. Many workers also object to HITs that require the use of Facebook accounts, which are intended to be quite personally identifiable. There has been at least one incident where a requester carelessly exposed hundreds of turkers' email addresses that the requester had collected.
      - Don't require workers to download software programs or apps to complete your HITs. This can be a major security risk for workers, particularly if the program comes from an unofficial source set up just for the HIT. It became known a few months ago that an academic researcher had performed a study on MTurk intended to see how low of pay levels would still convince workers to download and install a program that pretends to be malware, so many workers who are aware of this study are now even more hesitant to go along with download-requiring HITs even from seemingly legitimate requesters. Many workers are also specifically bothered by and try to avoid survey HITs that require the Java program Inquisit, which although generally regarded as legitimate, is designed to be intrusive and prevent workers from using their computers as effectively as they are used to working (for example, from what I've heard, Inquisit makes workers unable to copy and paste their worker ID into the survey when asked, and unable to copy and paste the completion code back into the HIT at the end of the survey; and if workers are trying to monitor the HITs being posted by a requester they're particularly interested in getting some of, and a notification from the monitor occurs during an Inquisit survey, the worker will be unable to briefly switch to that window/tab to accept the HIT they'd been waiting for, causing them to miss out on more income). If you insist on using Inquisit anyway, make it very clear up-front in as many places as possible (several or all of the: title, keywords, description, preview text, and consent/intro page) that Inquisit will be required. Workers are particularly irate if they waste their time getting halfway through a survey only to suddenly be surprised by Inquisit attempting to download and run, and then have to return the HIT if they aren't willing to run Inquisit.
      - If you don't follow the Terms of Service, particularly in the aforementioned ways that pose potential threats directly to workers, some workers will give your requester account negative Turkopticon reviews with flags for ToS violations, and report your HITs to Amazon.

    • Communicate with workers promptly and politely. Understand that when a worker goes out of their way to take the unpaid time and effort to send you a message through MTurk, and reveal their MTurk-associated name and email address to you in the process, it's usually going to be for a good/important reason. Check the email account associated with your MTurk requester account frequently. Respond to messages from workers as quickly as possible, preferably in less than 24 hours. I've seen a major university's IRB guidelines for MTurk use suggest that responding within 7 days is their idea of good promptness; I think most workers would find that to be unacceptably slow in most situations. Even 24 hours usually wouldn't be anywhere near fast enough to get clarification on a partially-finished HIT before it's long since expired. Don't ignore messages and be one of what seems to be about half of requesters currently who don't respond to most workers' messages at all. And when you do respond, don't be one of the requesters that workers have complained about being unnecessarily harsh / insulting / condescending / rude to the workers. Also note that the worker's MTurk worker ID# will be automatically included in all messages you receive through MTurk, labeled as 'Customer ID:', so you shouldn't need to ask the worker for that again to address what they've contacted you about.

    • Approve work as soon as possible. One of the configuration options when requesters create a HIT group is the 'Auto-Approve' (AA) time, which is the amount of time from the point a worker submits a completed HIT, to the point at which the MTurk system will automatically move the HIT into 'Approved' status for the requester, if the requester hasn't manually approved or rejected it prior to that point. This setting apparently defaults to 30 days, which is the maximum allowed. That 30 days is generally regarded by workers as a very long time to potentially have to wait, and generally makes your HITs much less desirable to workers who know how to check the AA time setting, if they don't find out from other workers that you have a history of approving much earlier than the AA time. Set your AA time as short as reasonably possible for the time you'll need for any checking of the work; 30 days is very seldom necessary or appropriate; no more than 7 days should generally be more than sufficient, and it's better if it's less than that. Many requesters approve work in less than 3 days, some in less than 24 hours.
      - Some requesters, and even some workers, try to compare MTurk approval times to the time between paychecks at a traditional job, to say that they think workers shouldn't complain about waiting for payment. But with a traditional job, you know you'll get paid for the time you've reported working, even if you don't get the pay until days/weeks after you did that work, and you'll know for sure how many days/weeks that wait should be. Even if they fire you in the meantime, they're still legally obligated to pay what you've earned. With MTurk approval times, you're actually waiting to see *if* you'll get paid at all for the work you've done, and if so, will it be at the end of the auto-approve time (which the worker may not know) or at some point sooner.
     
    • Like Like x 7
  4. Tribune Tribune of the Plebs Contributor

    Joined:
    Nov 25, 2013
    Messages:
    10,748
    Likes Received:
    32,202
    On TO-discuss, I responded to a colleague's defense of Inquisit:

    Sometimes, I find a narrow framework more expeditious than the "big picture."
    Often, an error in a little thing has organic connection to larger questions.
    I find this heuristic to be a great time-saver.
    Yes, it is good to have a view of the forest .. but only because the forest comprises many happy little trees.

    Dr. Aaron Sojourner's colleagues compromised Turkopticon's database without authorization.
    ...[T]he Minnesota researchers committed criminal acts (as defined by their home state and most other jurisdictions).

    My conclusion must be that the project was unethical, no matter what University of Minnesota's IRB says. :p

    Similarly, we have other academic researchers who claim to subscribe to a particular protocol.
    As part of that protocol, they register an account with Amazon and subscribe to a 'contract of adhesion' called the Terms of Service (TOS).

    If the very next thing they do is require a download of Inquisit, I don't need to ask whether they are "ethical" researchers.
    They have already answered that question in the negative, by violating their agreed protocol -- no matter how clueless their IRBs.

    If they have a problem, they should write Amazon. That's what I do. ;)
     
    • Like Like x 4
  5. Jan Veteran Member Member

    Joined:
    May 8, 2014
    Messages:
    2,892
    Likes Received:
    4,633
    I think there needs to be a paragraph about how insecure the mturk system is for workers. I think most of what I put in below is up in that really excellent piece above :). But I wanted to write something up in slightly different language (not sure if it's helpful). I'll also come back and edit it (I'm always missing typos and trying to clean things up). If I've gotten anything wrong, lmk

    But here's my 1 cents:

    Requesters need to understand rejections and 'hardblocks.'

    This is important background information for someone (anyone) setting up a HIT. Why is a rejection important to understand? For one thing, workers have received rejections from requesters who simply do not wish to pay. Although most requesters are fair minded and responsible about setting up a HIT, many others are not. They simply reject a worker because they never intended to pay. Workers have even been rejected because the requester set up the hit incorrectly.

    It's important to understand that if we are rejected unfairly, there is no viable appeal. We can email the requester and ask that they reconsider the rejection, but that's all that we can do. To work as a crowdsource worker is to expose yourself to a system in which you have very little power.

    Rejections carry considerable weight with workers and we try very hard to avoid them. If we get too many rejections, our approval rating becomes too low to accept most work assignments. This means that our income is threatened. If we were hoping to achieve a higher ranking in the mturk system (a so-called 'master worker' status), a rejection could make that harder or even impossible.

    Workers can also be hardblocked for no reason. They can even be hardblocked by 'accident,' from a requestor who doesn't realize what they are doing. Several times, academics have sent out hardblocks to dozens of workers simply because they did'nt want them to retake their survey. What they didn't realize, was that that action jeopardized the workers Amazon Account.

    When a worker is 'hardblocked,' they receive a note from Amazon. If this is their first 'hardblock,' the note says that if they receive another one their account could be in jeopardy. If they already have a 'hardblock' on file (or two), then their account could be terminated. If an account is terminated, the decision is final. Even if it is a mistake.

    Requesters have a right to good data. They are paying for workers to be responsive and give their best effort. But they need to know that the conditions of an mturk worker are that of a contract employee. And that contract can be terminated at any moment. We all (as workers), live with this knowledge and work as cautiously as possible. Making a decision to reject work is necessary: we all acknowledge that. But as an 'employer,' it's your obligation to understand the system and they way it effects individual workers. You can not leave it to Amazon to sort it out: they will not. If you reject a piece of work or hardblock an employee, then it is a decision that you make and are responsible for. Sometimes, it is a necessary action. But you need to be responsible for the decision and understand that it can have serious consequences.
     
    • Like Like x 7
  6. niloufar New Member Member

    Joined:
    Jul 24, 2014
    Messages:
    12
    Likes Received:
    72
    • Like Like x 1
  7. Jan Veteran Member Member

    Joined:
    May 8, 2014
    Messages:
    2,892
    Likes Received:
    4,633
    With respect to this paragraph:

    "Do not block workers to avoid duplicate subjects.

    Blocks are for bad faith workers and can result in workers being suspended by Amazon. This simple mistake can cost livelihoods. Learn how to avoid duplicate subjects fairly"

    I don't know if Amazon says that an account is suspended. If they use this language, then it's misleading. They terminate accounts (a suspension sounds like something reparable - or temporary, to me).

    jan
     
    • Like Like x 1
  8. Jan Veteran Member Member

    Joined:
    May 8, 2014
    Messages:
    2,892
    Likes Received:
    4,633
    Also, here:

    Ensure conditions for rejecting work are clear and fair.

    IDK, but I want something in there to remember that we all make mistakes. I think sometimes requesters forget that when they look at something.

    You know, I get it that a requester doesn't want to get dozens of emails after they post a HIT because we did something stupid (missed a code, forgot something, ect). But if there is a problem, I close a window prematurely and don't get a code or something happen inadvertently - well, shoot........if you can confirm the work then 'have a heart.'

    We really all do make mistakes.

    It's the nature of working on a computer - and being human.
     
    • Like Like x 2
  9. clickhappier ★★Ⰼ₳ՖŦξᚱ⌚ Contributor

    Joined:
    Jul 1, 2014
    Messages:
    5,600
    Likes Received:
    9,527
    For reference as to the current state of affairs in academic requester policies as of Aug 2014 (most date back several years), with select quotes that caught my eye (some things we hope will improve, some things that we hope will catch on more):


    Examples of university Institutional Review Board (IRB) policies and related documents regarding MTurk:

    City University of New York (CUNY), New York, USA - CUNY Human Research Protection Program (HRPP) "CUNY HRPP Guidance: Internet or Mobile Technology Based Human Subject Research" (pdf, archived pdf)
    "Some Internet venues, such as Amazon Mechanical Turk, store identifying information about their users. When using such a venue for subject recruitment, researchers are strongly encouraged to use a different venue, such as Survey Monkey and Qualtrics, for data collection purposes. This type of strategy allows for separation of subject identiers from other subject data, and adds another layer of protection from breach of subject condentiality."

    Cornell University, New York, USA - "Use of Social Networking Sites or Mobile Devices for Human Participant Research: Guidelines for IRB application and review (Policy #20)" (pdf, archived pdf)
    "c. Deception research: ... Special considerations include the following: Individuals must be made aware that they are the subjects of an experiment. Totally blind deception experiments are not acceptable."

    "f. Use of Amazon Mechanical Turk as recruitment venue for surveys and other studies: ... The compensation for the tasks accomplished is typically very small, usually less than $1. ... Address whether or not the compensation is contingent upon certain conditions. Ensure that the complexity of the task and the amount of time expected for completion is reasonable and communicated clearly in the consent process."

    Massachusetts Institute of Technology, Massachusetts, USA - Committee on the Use of Humans as Experimental Subjects (COUHES) "COUHES Policy for Using Amazon's Mechanical Turk" (webpage, archived webpage)
    " In order to ensure that rejection requirements are fair, Requesters must review their data carefully, and especially review any rejection procedure that is leading to rejections of >20% of HITS. Note that experimenters should be especially careful about rejecting HITs that require a lot of work or time (e.g. more than 5 minutes).
    All rejections are made by the Requester after manual inspection of each Provider’s responses. COUHES requests that if an MIT-based Requester does not review a HIT within one week after completion, the HIT should be automatically accepted the Provider should be compensated. Also, Requesters should not use data from rejected (i.e. uncompensated) HITs in any publications. "

    "Payment for Participation: Providers will be paid at a rate that is comparable to that of other tasks offered on the Mechanical Turk website and based on the time that Requesters estimate the task will take. Payment is per HIT, and does not depend on the time spent by the Provider, so pay per minute is estimated by the Requester, based on the average time the HIT takes to complete. For instance many tasks currently hosted on the Mechanical Turk pay approximately $0.02 - $0.04 for tasks that take around a minute to complete, and approximately $3.00 - $8.00 for tasks that take 45 – 90 minutes to complete. Payment offered will be adjusted based on how enjoyable or difficult Requesters judge the task to be."

    "COUHES requests that the Requester monitors their email account on a weekly basis, so that emails are responded to within seven days’ time."
    Massachusetts Institute of Technology, Massachusetts, USA - "Saxe Lab Policy for Using Amazon’s Mechanical Turk" (docx, archived docx, cache view html)
    "6. Every reason that a participant might be rejected should be clea[r]ly listed, as participants who are rejected do not get paid and have their rating lowered. Participants are very upset when they feel that they have been unfairly rejected, and this damages the reputation of our lab, making other workers less likely to participate in our surveys."

    "7. Payment should be based on your estimate of how long the task will take. Saxe lab studies generally pay around $2 to $10 per hour."

    North Carolina State University, North Carolina, USA - "Things to Consider when Working with Human Subjects and Surveys" (pdf, archived pdf)
    "If using the MTurk population for online surveys, remember that MTurk worker IDs count as identifying information so you need to come up with a plan for that"

    Pomona College, California, USA - "Taking the Red Tape Out of the IRB" (pdf, archived pdf)
    "xi. Payment for participation: What do you intend to pay participants? Typically we recommend $10/hour of participation though different rates apply if you are using MTurk." [translation: lower rates]

    Reed College, Oregon, USA - "Institutional Review Board Frequently Asked Questions" (webpage, archived webpage)
    "What about online survey providers (e.g., SurveyMonkey, SurveyGizmo, MTurk, etc.)? ... For MTurk: The use of this mechanism requires special review by the IRB as this provider collects identifying information on participants and what surveys are completed. The IRB expects that investigators will NOT use the survey collector on MTurk, but may use MTurk for recruitment and provision of incentives with links to surveys managed elsewhere."

    University of Arizona - Department of Linguistics "Guidelines and Tips on How to Get Permission to do Research with Human Subjects" (pdf, archived pdf)
    "Cases where you DO need human subjects approval: ... Data collection for research purposes through Amazon Turk or other crowdsourcing methods. (We are just starting to figure out human subjects formats for this.)"

    University of Chicago - Social & Behavioral Sciences Institutional Review Board "Guidance on Internet Research" (webpage, archived webpage; docx, archived docx, cache view html)
    "... The IRB will expect that appropriate language be included in mTurk consent scripts to alert mTurk participants to this [public profile pages] vulnerability and explain how participant confidentiality will be protected. ..."

    University of Texas at Austin, Texas, USA - "IRB Guidelines and Suggestions for Using Mechanical Turk (MTurk) for Social/Behavioral Research Projects" (pdf, archived pdf)
    "Suggestions to consider for Informed Consent Procedures and/or Study Documents: ... If there will be study compensation, add a statement in the consent document that MTurk worker IDs will only be collected for the purposes of distributing compensation and will not be associated with survey responses (if applicable)."

    University of Waterloo, Ontario, Canada - "Human Participant Research Guidelines: Use of Crowdsourcing Services" pdf, archived pdf)
    "Researchers who are not citizens of the USA are not able to post a study directly to Amazon Mechanical Turk therefore many work through an intermediary service"

    " Although remuneration listed in a HIT can be as low as $0.01, Amazon Mechanical Turk recommends “requesters” look at HITs similar to their HIT to determine the appropriate remuneration. Some researchers state “the rule of thumb for payment is around half-a-penny per question” (Schnoebelen and Kuperman, in press). In the spirit of fairness, University of Waterloo researchers should consider providing individuals who participate in research via Amazon Mechanical Turk equivalent monetary remuneration to other HITs listed and set the remuneration appropriately. Researchers should provide remuneration that is comparable to other studies (or HITs) of similar length and difficulty. ...
    As of July 2011, typical Amazon Mechanical Turk remuneration is $.50 for a 15 to 20 minute study or $.25 for a study less than 15 minutes. ... It is important to keep in mind that an amount too low may attract too few participants while an amount too high (in comparison to similar HITs) may attract participants who are wishing to complete HITs strictly for the remuneration. "

    Xavier University, Ohio, USA - "Can I use MTurk (www.mturk.com) for online data collection?" (webpage, archived webpage; pdf, archived pdf)
    "The site was not designed with research such as yours in mind, in all likelihood. Rather, it was created as a means by which “micro-tasks” could be presented to a large and willing audience of potential employees. That it has become increasingly common as a “research gateway” represents an evolution of the tool, and one that IRBs are keeping a close eye on."

    "The IRB does not explicitly regulate the amount of payment that can or should be offered to your participants via mturk. In deciding on an appropriate amount to offer for your task, please keep in mind both the length of time required and the relative rigor of the task, as these are factors that we will be considering in reviewing any mturk-related submission. A two-minute task that offers a reward of $5, for example, clearly constitutes a situation of excessive inducement and would be viewed by our IRB as potentially manipulative. On the other hand, a sixty-minute task that pays only $.02 would constitute exploitation of your participants, and also be viewed as problematic. Please provide a rationale for why the level of payment you have chosen for your study is appropriate, based on what you are requiring of participants in terms of both time and effort."

    US Census Bureau - not an official policy, but was presented at their 2013 Federal Computer-Assisted Survey Information Collection (FedCASIC) Workshop: "Applying Crowdsourcing Methods in Social Science Research" by Michael Keating, of government research contractor RTI International (pdf, archived pdf; abstract webpage, archived abstract webpage)
    " Crowdsourcing should not be a race to the bottom.
    Pay people appropriately!
    Quick way to figure out payment levels:
    - On average, how long does it take to complete your task?
    - How much were you planning to pay per HIT?
    - Calculate the equivalent hourly wage.
    We are not in the business of virtual sweatshops. "


    Continued...
     
    • Like Like x 2
  10. clickhappier ★★Ⰼ₳ՖŦξᚱ⌚ Contributor

    Joined:
    Jul 1, 2014
    Messages:
    5,600
    Likes Received:
    9,527
    ...continued.


    Examples of documents researchers have submitted to IRBs for MTurk-based research:

    Claremont Graduate University, California, USA - has an MTurk-only version of the IRB "Application for Research Project Review" form (docx, archived docx)

    Georgetown University, Washington DC, USA - by Andrew Long, former manager of Behavioral Research Lab at the McDonough School of Business - sample MTurk recruitment protocol (pdf, archived pdf) (described in his 'MTurk training' document pdf, archived pdf)
    "Subjects will be paid using a $2/hr payment scheme, thus for a 15 minute study a participant will be paid $0.50. Within seven days of completing the study, the subject’s participation will be approved on MTurk, which automatically submits a payment to the participant through the Amazon Payments system."

    Johns Hopkins University, Maryland, USA - by Chris Callison-Burch - IRB "Application for Exemption" for an MTurk project (Google Docs view, pdf) and resulting approval letter (Google Docs view, pdf) (from a workshop discussion group)
    "We will pay participants small sums of money to complete our task, ranging from $0.01 to $1. All participants can choose for themselves Whether the Compenation is fair, and opt not to do it if they deem the compensation to be too low. Amazon's Mechanical Turk has many other researchers and Companies offering tasks, so we will offer compensation that is similar to what others Offer."

    Rochester University, New York, USA - "Amazon Mechanical Turk Information Sheet" sample/template (doc, archived doc)

    University of California at Irvine, California, USA - by William C. Thompson - "Protocol Narrative for Exempt Research" for an MTurk-based study he performed (pdf, archived pdf)
    "Participants in Mechanical Turk (Turkers) will be offered a small financial incentive (less than 50 cents) to participate in an online study. ... Participation will be limited to those with IP addresses in the United States. ... It will take subjects approximately 15-20 minutes to read the stimulus materials and respond to questions."

    University of Florida, Florida, USA - minutes from an Institutional Review Board Meeting, as an example of an IRB discussion of an MTurk project (pdf, archived pdf)

    University of North Carolina at Chapel Hill, North Carolina, USA - by Jacob Kramer-Duffield, et al - unsuccessful IRB exemption request form (pdf, archived pdf), eventually-successful full IRB form (pdf, archived pdf), follow-up memo responding to IRB (pdf, archived pdf), and attachment to memo (pdf, archived pdf) (from a presentation by him and Terrell Russell, "Mechanical Turk and IRB" - pdf, archived pdf)
    "There is no risk in these activities - users will answer a short series of questions and their personally identifiable information will not be accessible to the researchers."

    IRB asked: "Please give more information as to why it would not be appropriate for participants to read a short consent statement before they access the questionnaire, and only continue to the questionnaire if they agree."
    Researchers responded: "Participation in this task is entirely voluntary and is in itself implied consent, among a group of users of Amazon's Mechanical Turk who have already registered for the service and are governed by its terms and conditions as described above."

    "Participants will be paid via the Amazon Mechanical Turk system at a rate of $.02 per "Human Intelligence Task" (HIT), i.e., answering of one of the question-tasks."

    [IRB had to ask them to submit 'Part B. Questions for Studies that Involve Direct Interaction with Human Subjects', which the researchers had omitted because they viewed online surveys as not being 'direct interaction' (a number of sources do consider online surveys to be interaction).]



    Examples of other university IRB policies and related documents of interest, that don't specifically mention MTurk:

    Cardiff University, United Kingdom - article by carol Haigh and Neil Jones, published by the Association of Research Ethics Committees, used as a guide by the Research Ethics Committee of the Cardiff School of Journalism, Media, & Cultural Studies: "Techno-research and cyber-ethics: challenges for ethics committees" (doc, archived doc, cache view html)

    Colorado State University, Colorado, USA - IRB Guidance Documents on Internet Research: General Guidelines (pdf, archived pdf), Online Survey Guidelines (pdf, archived pdf), Private Sites Guidelines (pdf, archived pdf), Public Sites Guidelines (pdf, archived pdf), and Virtual World Guidlines (pdf, archived pdf)
    "The fundamentals of informed consent for face-to-face data collection must also be observed for internet & online survey research. Prior to the start of the survey, an adequate informed consent document should be displayed to the participants."

    "A forced response that requires participants to answer all the questions before submitting the survey or before moving to the next question is not allowable and is seen as a violation of the research principles of voluntary participation. Use an N/A response or “Prefer not to Respond” as an option, if necessary."

    "If individuals intentionally post or otherwise provide information on the Internet, such information should be considered public unless existing law and the privacy policies and/or terms of service of the entity/entities receiving or hosting the information indicate that the information should be considered private. To the extent that terms of service or explicit prohibitions would preclude the use of data on the Internet for research purposes, the determination that such data should be considered private is clear."

    Texas A&M University, Texas, USA - "Office of Research Compliance: Research Involving Internet Research" (pdf, archived pdf)

    University of Brighton, United Kingdom - "Ethical issues for consideration when conducting internet research" from the Health and Social Science, Science and Engineering Research Ethics and Governance Committee; also includes a copy of "Ethical Decision-Making and Internet Research" from the Association of Internet Researchers (pdf, archived pdf)

    University of New Hampshire, New Hampshire, USA - "Institutional Review Board for the Protection of Human Subjects in Research: Guidelines for Conducting Web-Based Survey Reseearch" (pdf, archived pdf)

    University of Wisconsin at Milwaukee, Wisconsin, USA - "IRB Guidance for Online Research Activities" (doc, archived doc, cache view html)
    - also, a presentation by Michael Zimmer PhD, a professor at UWM, for the US Department of Energy's Human Subjects Protection Program - "Internet Research Ethics: Issues and Challenges" (pptx, archived pptx, cache view html)

    University of Kentucky, Kentucky, USA - presentation by Jeffrey M. Cohen PhD, of HRP Consulting Group (Human Research Protections), used as a guide by the University of Kentucky's Office of Research Integrity: "IRB Challenge: Research on the Internet" (pdf, archived pdf)
    "Invalid research can have no benefit. – inappropriate when there is risk to subjects"

    American Association for Public Opinion Research (AAPOR) - "IRB FAQs for Survey Researchers" (webpage, archived webpage)

    US Department of Health and Human Services (HHS) - "Considerations and Recommendations Concerning Internet Research and Human Subjects Research Regulations, with Revisions" (pdf, archived pdf) (incidental mention of mturk)
    - also, a presentation by Laura Odwazny, a Senior Attorney for HHS - "Internet Research, Social Media, and the IRB" (pdf, archived pdf)

    'Code of Ethics Collections' from the 'Center for the Study of Ethics in the Professions' at Illinois Institute of Technology
     
    • Like Like x 2
  11. clickhappier ★★Ⰼ₳ՖŦξᚱ⌚ Contributor

    Joined:
    Jul 1, 2014
    Messages:
    5,600
    Likes Received:
    9,527
    Amazon does call it 'suspensions' according to everything I've heard. Theoretically they can be undone (here's an example), but it seems to be very rare. Hopefully this modification clarifies that part:

    "Blocks should only be used for bad-faith workers, as they can result in workers being suspended by Amazon. Suspensions of this type are equivalent to a permanent ban in most cases; this simple mistake can cost livelihoods."

    I think further comments on the current contents of the wiki would be better placed in Niloufar's new thread, Call for comments, edits, and support: Turker-authored guidelines for research on AMT
     
    • Like Like x 3

Share This Page