Experiences in Crowd Sourcing

Mechanical Turk

Daniel Arbix

I have signed up to Mechanical Turk as a Brazilian citizen (not all HIT opportunities were available, then). I have tried to explore different tasks to check the diversity the platform offers for workers. There are some amusing tasks to be performed, but most – against the website’s Participation Terms –, are boring schemes to distort internet advertisement payments or to gather active e-mails (for spam or worse, I presume). There are also numerous HITs which demand fake reviews of products and websites, or which require “turks” to show support to social-networking profiles or events, and even to write posts in blogs making compliments to the blog owner’s clever analyses. A short summary of my experiences follows.

1st try: bad page design, no $ I saw a task as available, was able to follow the instructions and actually perform it on a third party website, but then I realized that the event that would trigger a confirmation number required for my payment had already expired, so of course no payment was made to my account. This was the first contact I had with HITs designed to redirect traffic to earn advertisement revenues (it was described as “a test for page load time, very easy, for USD 2.00”).

2nd try: the second HIT I tried had the following misleading instructions:

• "This HIT is an easy to complete 'sign up' assignment. It shouldn't take you longer than 2 minutes to complete. Many thanks for your time! Simply go to: http://www.awin1.com/awclick.php?mid=633&id=93282 Sign up. Then send a print screen of your confirmation email to: mark.studentearnings@gmail.com and write your username and approximate time of sign up in the box below"

The site redirected me to http://www.offersclick.co.uk/offers/SiteRender.aspx?SiteID=501&ThemeID=21&q14259=AFW&q26274=93282 Again, it seems like arbitrage of internet advertisement revenues… It was impossible to complete the task within the assigned time – the form-filling advertisements website took too much to load –, so I returned the HIT (and received no $, in spite of losing a lot of time)

3rd try: the HIT had the following description:

• "Complete free online quote form. MUST BE 18 OR OLDER AND LIVE IN USA TO COMPLETE THIS HIT Visit Site - Enter Name and Email - Takes less than a minute. IMPORTANT: Eligible only to those who have not yet signed up to this offer. Please don't use disposable e-mails as well. Let's keep this site honest. INSTRUCTION: 1) To get started, visit this website: http://www.aislezone.com/mturk-offer02.php 2) Select make, model and enter zip code. Click "get quotes" 3) Complete the form and click on "get free dealer quotes" REQUIRED PROOF: On activation confirmation copy/paste or type full text to the box below starting with : "Sent!..." "

The time-description was fair, and I received payment after two days. I also (stupidly) provided my real Stanford e-mail, with the result of getting my mailbox now filled by car dealers messages. Again, the HIT seems like arbitrage of internet advertisement revenues.

4th try: the HIT had the following description:

• "Data Collection-Google Results Reward: $1.25 per HIT Insert address, perform google searches and indicate first result"

This HIT was an honest one, requiring a verification of search results (Google), all related to the same company. Time description was fair, and my $ arrived a day later.

5th try: the HIT had the following description:

• "Rate faces of candidates running for office. Usually takes less than 5 minutes. You can find the survey here: http://www.surveygizmo.com/s/207922/faces3 At the end of the survey, you'll find a code. To get paid, please enter the code below:"

This was a fun, quick HIT. It had a fair time description, and payment was received after a day. I was, however, a terrible subject for the MIT social psychology survey, since most pictures to be rated are of Brazilian politicians…

Jason Harrow

It all started out so promisingly. Before I even started, I was excited because "Mechanical Turk" is such a great name. What did it mean? What was mechanical about it? And who or what is the “Turk” in the transaction? Then, in the back of my mind: isn’t that somehow vaguely racist?

Wikipedia would know what a Mechanical Turk is, of course. So I searched for it, and I immediately found my answer. The Mechanical Turk was a chess playing “machine” that, “from 1770 until its destruction by fire in 1854 . . . was exhibited by various owners as an automaton, though it was explained in the early 1820s as an elaborate hoax.” Something thought to be a form of artificial intelligence later explained to be just a smart guy in a box? How cool is that? Could there be a more perfect name for this service? I made a mental note that if I am ever in the position to pick a name for an Internet company or service, I will find out who chose "Mechanical Turk" and consult him or her. I couldn’t have been more excited to get started.

I logged in with my Amazon ID and created my Amazon payment account. Unlike Daniel (above), I am a US citizen; also unlike Daniel, I chose to go first for a little assignment that paid $.02 per HIT. My task was to go to the website of a given educational institution, type in when the spring semester 2010 started, and give the URL of the Academic Calendar page confirming this. I did one successfully and submitted it. It took about 90 seconds. Easiest job of my life. Bring on a few more.

The next one took longer and bore no fruit, though: the little Midwestern barbershop college that was assigned to me didn’t display when their spring semester started (side thought: do barbershop colleges even have spring semesters?). That was frustrating; all that effort, and no shiny pennies at the end of the rainbow. And why in the world does this guy – Aaron Smyth, whoever that is – even want to include barbershop colleges in his survey? What could he possibly be doing with that data? I pressed on and did a few more start dates before trying to find something else. It got boring pretty quickly, and I realized that 2 cents wasn’t really worth it beyond the initial thrill of it all.

Like Daniel, I then encountered what later became obvious as a scam: someone who said they’d pay me money for signing up, but the time given was too fast and there was no way to actually verify that I completed the task in the way they said I needed to. This made me angry. How in the world does Amazon – one of my favorite companies on the planet, by the way – let them get away with this? I reported the bastards, but I don’t know what happened to that complaint. A few days later, Mechanical Turk still seems littered with these scams. That’s disappointing.

I turned to a final HIT, what seemed like just an individual who wanted me to go to his website and post any kind of comment, just for the sake of getting more hits. Seemed a little shady, but I gave it a try and duly posted a comment. Days later, I got the bad news: rejected! I wouldn’t be getting the shiny nickel I was promised. Hey Amazon: how is that allowed?

I was once so excited by the great name and the prospect of making easy money, but I'm now deeply disappointed with both the requesters and with Amazon. Why did all the posters seem so scammy? Why aren't there any tasks that seem interesting and worthwhile? And how does Amazon let the requesters get away with this? I felt somehow betrayed by an Internet company that I feel oddly loyal to; it was as if HBO launched a new channel that turned out to be entirely infomercials. Get your act together, Mr. Bezos!

I logged into my account a few days later. The final tally was 8 HITs accepted; 4 submitted; 1 returned; 3 abandoned; 3 approved; 1 rejected; and, in the end, 6 measly American cents earned. I wish that the original chess-playing Mechanical Turk hadn’t been destroyed in 1854. That Mechanical Turk might have been a sight to see. This one isn’t.

Lien Ceulemans

After all the Christmas presents I had to buy for my family, I thought that a good way to earn some money (while doing my homework) was to volunteer as a mechanical turk. A Google search brought me to the Amazon mechanical turk website (https://www.mturk.com/mturk/welcome).

Before starting, I first had to create a payment account and accept the 6 pages long “Amazon Mechanical Turk Participation Agreement”, thereby exclusively granting my intellectual property rights to the “Requester”. Furthermore, this agreement informed me that, as a mechanical turk, I would perform a “work for hire” “and would “not be entitled […] to any vacation pay, sick leave, insurance programs, including group health insurance or retirement benefits; or […] worker's compensation benefits in the event of injury.”

Amazon describes its platform as “a marketplace for work that requires human intelligence” There is indeed no sign of a worldwide crisis, as 306,354 “Human Intelligence Tasks” or “HIT’s” were available.

I started with an easy, quite boring assignment that would pay me 0.05 dollar for each correct hit: “Find the Website Address for a Restaurant and its Menu”. Since at that time, my “value” was still rated at 100, I met the qualification requirement. While the website of “Merlo La Trattoria” in Chicago is quite easy to find, I ran into difficulties after a few hits. The address of the “Brazil steak” house was only indicated in Chinese characters in the assignment, so I got completely lost at Google. It was time for another challenge…

For my next HIT, I had to “write a short answer about cars” of 50 to 60 words. Since I do not know anything about cars, this seemed the perfect HIT to test the reliability of the mechanical turk platform. To my best knowledge, I answered the question “How do I clean a mass air flow sensor on my Mercury?” as follows: “This is very easy to clean. You just take some water and soap that smells delicious and pour the water into the mass air flow sensor. If it is still not clean enough, repeat this process for a couple of times. You will see that your mass air flow sensor will look brand new again.” Although the HIT explanation mentioned that no qualification was required, after performing the hit, my answer was “submitted to conjecture corporation and [would] be approved or rejected shortly.” A few hours later, I received a “Qualification Request Rejection Notification” email ‏for Amazon Mechanical Turk. Despite I failed the test question, I did receive the 0.03 dollar reward for participating in this HIT.

After this, I decided to find a HIT which was more related to my field of knowledge. For different law schools, I needed to find the final fall application deadline for 2010 and the URL of website where I found the information. There were still 53 HITS available with a reward of 0.05 dollar each. Although the HIT information indicated that each HIT would take me around 10 minutes, I managed to complete a single HIT in about 4 minutes. This would thus result in an hourly wage of 0.75 dollar. Taking into account both the operating costs (e.g. electricity, internet, etc.) and the opportunity costs involved, I wonder who - besides “Difficult Problems in Cyberlaw” students - is willing to be a mechanical turk.

Although I got very bored, I tried to perform another HIT that required me to find the date, parties and subject of so-called “top secret letters of the US war department”. While this sounds as a very exciting and interesting assignment, in reality, trying to analyze unreadable pdf’s was not that fun. I decided to give it a last try, but had a hard time to find any attractive assignment. In the end, I participated in a research of the UNC Charlotte Human Computer Interaction Lab that does research in policy making decisions for social applications. After giving my ‘Informed Consent for Social Application Decision Making”, I had to indicate whether (i) I would or would not accept a certain Facebook application and (ii) share information of my Facebook friends with this application. Although this was not really interesting, this HIT gave me at least a good feeling as I contributed to university research. This was perfect to end my life as a mechanical turk.

Today (i.e. 5 days after submitted more than 20 hits), my Amazon Payment Account indicates that the approval of my submitted HIT’s is still pending. This means that so far, I only earned 0.03 dollars for 2 hours of labor.

LiveOps

Sheel Tyle

I am in the process of signing up as an inexperienced bilingual (Spanish & English - I'm conversational, not fluent, in Spanish, but let's see how much I'm tested on it) LiveOps call center agent. There are videos on the website from independent agents, testimonials on the sidebar ($15-22/hour), and press releases from various periodicals that try and convince.

There are five steps that I must follow in order to be 'submitted for review':

Validate your email address

Verify your understanding of the general requirements

Provide basic information on your background

Assess your comprehension and computer skill

Audition your voice

Under 'basic information', here are some of the questions:

Have you ever contracted your business with LiveOps in the past?, Do you have prior call center experience?, Are you currently licensed to sell both health and life insurance products?, Are you currently licensed to sell both property and casualty insurance products?, Do you have prior experience in sales?, Do you have prior experience taking calls from Radio offers?, Do you have any experience in outbound telemarketing?, Do You Speak, Read and Write Spanish - FLUENTLY?, Do You Speak, Read and Write French - FLUENTLY?

I said "No" to every one except "Yes" to speaking Spanish.

Then, under 'comprehension and computer skill', I had to answer questions like:

Please read this script:"We dont want you to miss out on this great offer, so what I can do for you today is offer you 1 HotBrush for 3 easy payments of $29.99 plus $14.75 to cover processing or you can take advantage of our special offer, the SpeedyHeat model for only $6.67 additional per payment. The SpeedyHeat Model contains a computer chip which lets your HotBrush heat up faster and hold a more even temperature just like the most expensive professional quality hot tools. So, would you like to order the HotBrush or the SpeedyHeat Model?"

In this script, is the customer choosing between two different products or are they deciding whether to add a product (for two products total)? Deciding whether to add a product Choosing between two different products

What is the keyboard shortcut to move between windows you already have open on your computer? Hold down the "Alt" button and press the "Tab" button Drag the window offscreen Click the appropriate button in the task bar Close the program you are currently working in, and open a new one Minimize the window you are currently working in

Finally, the voice test. LiveOps asked me to call in to a 1-800 number and, when prompted, read two passages: one in English and one in Spanish.

Submitted. Once I hear back, I'll update whether I was accepted =)

Ramesh Nagarajan

I've completed the agent qualification process to become a LiveOps representative. Sheel did a good job of explaining the basics of the qualification process, so I'll share two observations I had about the process.

First, there's a fair amount of legal language LiveOps uses, with the intention of disclaiming any liability or even the existence of much of a relationship between it and its contractors. One must agree to the following: "I understand that LiveOps will investigate all of my information provided during the Agent Qualification process, and that Certification of my home business to contract with LiveOps will be contingent upon successful completion of a criminal and credit background check." They aren't currently accepting Massachusetts residents as LiveOps contractors -- perhaps Massachusetts's laws are too worker-friendly -- so I hope that means I get to avoid the "criminal and credit background check." Also, I didn't have to provide a Social Security Number, which makes me wonder if there is a real background check. Going back to the phrase I agreed to, I found it interesting that LiveOps representatives are setting up "home business[es] to contract" with LiveOps. There seem to be at least two advantages that LiveOps has over traditional telemarketing and call center companies -- first, it could save money on infrastructure by having employees work out of their own homes and use their own computers and phones, and second, it could save money by not having actual employees. I wonder which is more important, and if the second is a necessary part of the company's business strategy.

On a lighter note, the questions for "comprehension and computer skill" were quite entertaining. I was asked if booting a computer meant to turn on the sound, throw it out, turn it on, or add extra drives, and I had to decipher the meaning of a call script. I think there's a good chance I got one of the questions wrong. It asked what a customer paid today for a product that had "an upfront trial payment" of $15, three monthly payments of $40, and a shipping and handling charge of $10. I answered $15, but when I think about it, maybe it should be $25.