Government // Enterprise Architecture
News
8/24/2012
04:29 PM
Connect Directly
Google+
LinkedIn
Twitter
RSS
E-Mail
50%
50%

Crowdsourced Work Made Cheaper With Qurk

MIT researchers find that tasks given to people because they're too tough for machines can still be improved through the involvement of computers.

10 Crowdsourcing Success Stories
Slideshow: 10 Crowdsourcing Success Stories
(click image for larger view and for slideshow)
Crowdsourcing, the practice of delegating tasks to large numbers of people because they're too difficult for machines, can be made more efficient by bringing computers back into the loop.

MIT researchers are working on a database system called Qurk that can identify tasks ill-suited to computers and automate the assignment and presentation of those tasks--human intelligence tasks, or HITs--to human workers.

Amazon Mechanical Turk, which launched as a beta service in late 2005, is perhaps the best known crowdsourcing service for labor. It has become a platform for businesses like transcription service CastingWords and has spawned services like CrowdFlower and Samasource that complement it or compete.

Initial work on the Qurk system was described in two academic papers last year and a follow-up paper, "Human Powered Sorts and Joins," will be presented soon at the 38th International Conference on Very Large Databases, which takes place in Istanbul, Turkey, from August 27-31.

[ How can you speed up apps? Read Facebook Takes Mobile App Native For Speed. ]

The paper describes how crowdsourced labor tasks can be integrated into Qurk, "a declarative workflow engine," with a particular focus on "sorts" and "joins," two common database operations. Qurk helps people build crowd-powered data processing workflows using a language similar to Apache Pig.

It turns out that while humans may be better at certain tasks than computers, like judging the age of a person in a photograph, they're not very efficient at generating HITs--descriptions that identify the work to be done--and their performance can be enhanced through a better user interface, as well as through reuse of workflow implementations.

"Human workers periodically introduce mistakes, require compensation or incentives, and take longer than traditional silicon-based operators," the paper notes. "Currently, workflow designers perform ad-hoc parameter tuning when deciding how many assignments of each HIT to post in order to increase answer confidence, how much to pay per task, and how to combine several human-powered operators (e.g., multiple filters) together into one HIT. These parameters are amenable to cost-based optimization, and introduce an exciting new landscape for query optimization and execution research."

In other words, human intelligence tasks can be identified and presented more effectively by software than by flawed, self-interested, carbon-based operators who linger over lunch.

As proof of concept, the researchers conducted an experiment using Amazon Mechanical Turk that involved joining two sets of images--identifying whether they were the same person--to be grouped together or distinct. Done the traditional way, the cost of this HIT came to $67. With Qurk, the cost was reduced to $3.

The paper does not address the impact such efficiency may have on the income of Mechanical Turk workers.

Comment  | 
Print  | 
More Insights
Register for InformationWeek Newsletters
White Papers
Current Issue
InformationWeek Tech Digest, Nov. 10, 2014
Just 30% of respondents to our new survey say their companies are very or extremely effective at identifying critical data and analyzing it to make decisions, down from 42% in 2013. What gives?
Video
Slideshows
Twitter Feed
InformationWeek Radio
Archived InformationWeek Radio
Join us for a roundup of the top stories on InformationWeek.com for the week of November 16, 2014.
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.