Crowd-sourcing is a recent framework in which human intelligence tasks are outsourced to a crowd of unknown people as an open request for services. Requesters use crowd-sourcing for a wide variety of jobs like dictation-transcription, content screening, linguistic tasks, user-studies, etc. These requesters often use complex workflows to subdivide a large task into bite-sized pieces (including the management of these tasks), each of which is independently crowd-sourced. These workflows are paramount to the success of crowd-sourcing, still, there has been little attention paid to methods for dynamically optimizing the throughput of a workflow. Controlling and optimizing such a workflow is an excellent application for AI research for two reasons. First, it is challenging in that the agent has to understand the dynamics of an uncertain, real-time environment and reason about distinct choices for a decision. More importantly, the domain has significant economic value -- progress can potentially impact hundreds of thousands of people and spur economic development in a fast growing sector.
This project is investigating complex workflows using a decision-theoretic framework that optimizes for a quality/price trade-off, with aims of (1) building statistical models of worker behavior derived from a large corpus of online behavior, (2) defining a declarative representation language to describe a wide range of workflows, and (3) developing an automated scheme that optimizes a general workflow resulting in an automated controller for making informed decisions at various stages of the process and for monitoring worker accuracies and computing corrections based on them. In the longer term, perhaps beyond the scope of this project, is (4) development of an interface optimizer that automatically learns the best user interface for a task based on user behavior increasing throughput of the workflow, and (5) integrating these ideas in an open-source, software toolkit to directly benefit the various requesters in managing their tasks.
Crowdsourcing, outsourcing of tasks to a crowd of unknown people ("workers") in an open call, is rapidly rising in popularity. It is already being heavily used by numerous employers ("requesters") for solving a wide variety of tasks, such as audio transcription, content screening, and labeling training data for machine learning. However, quality control of such tasks continues to be a key challenge because of the high variability in worker quality. Our NSF grant funded the investigation of decision-theoretic techniques for the problem of optimizing workflows used in crowdsourcing. In particular, we designed AI agents that use Bayesian network learning and inference in combination with Partially-Observable Markov Decision Processes (POMDPs) for obtaining excellent cost-quality tradeoffs. We studied these techniques for five distinct crowdsourcing scenarios: (1) control of voting to answer a binary-choice question, (2) control of an iterative improvement workflow (where a group of unrelated workers creates and improves an artifact, such as an English-language caption for a picture), (3) control of switching between alternate workflows for a task, (4) multi-label classification, and (5) taxonomy generation. In each scenario, we designed a Bayes net model that relates worker competency, task difficulty and worker response quality. We also design a POMDP for each task, whose solution provides the dynamic control policy. We demonstrate the usefulness of our models and agents in live experiments on Amazon Mechanical Turk. We consistently achieve superior quality results than non-adaptive controllers, while incurring equal or less labor costs.
