Combining Human and Machine Intelligence in Large-scale CrowdsourcingEce Kamar Microsoft Research Redmond, WA 98052eckamarmicrosoft.comSeverin HackerCarnegie Mellon University Pittsburgh, PA 15289 shackercs.cmu.eduEric Horvitz Microsoft Research Redmond, WA 98052horvitzmicrosoft.comABSTRACTWe show how machine learning and inference can be har- nessed to leverage the complementary strengths of humans and computational agents to solve crowdsourcing tasks. We construct a set of Bayesian predictive models from data and describe how the models operate within an overall crowd-sourcing architecture that combines the efforts of people and machine vision on the task of classifying celestial bodies de-fined within a citizens science project named Galaxy Zoo. We show how learned probabilistic models can be used to fuse human and machine contributions and to predict the behaviors of workers. We employ multiple inferences in con- cert to guide decisions on hiring and routing workers to tasksso as to maximize the efficiency of large-scale crowdsourcing processes based on expected utility.Categories and Subject DescriptorsI.2 Distributed Artificial Intelligence: Intelligent agentsGeneral TermsDesign, Algorithms, EconomicsKeywordscrowdsourcing, consensus tasks, complementary computing, decision-theoretic reasoning1.INTRODUCTIONEff orts in the nascent field of human computation have explored methods for gaining programmatic access to peo- ple for solving tasks that computers cannot easily perform without human assistance.Human computation projects include work on crowdsourcing, where sets of people jointly contribute to the solution of problems. Crowdsourcing has been applied to solve tasks such as image labeling, product categorization, and handwriting recognition. To date, com- puters have been employed largely in the role of platforms for recruiting and reimbursing human workers; the burden ofSeverin Hacker contributed to this research during an in- ternship at Microsoft Research.Appears in: Proceedings of the 11th International Con- ference on Autonomous Agents and Multiagent Systems (AAMAS 2012), Conitzer, Winikoff, Padgham, andvanderHoek(eds.), 4-8 June 2012, Valencia, Spain. Copyrightc ?2012, International Foundation for Autonomous Agents and Multiagent Systems (www.ifaamas.org). All rights reserved.managing crowdsourcing tasks and making hiring decisions has relied on manual designs and controls. However, interest has been growing in applications of learning and planning to crowdsourcing. We investigate principles and algorithms for construct- ing crowdsourcing systems in which computer agents learn about tasks and about the competencies of workers con-tributing to solving the tasks, and make effective decisions for guiding and fusing multiple contributions.As part of this investigation, we demonstrate how we can leverage the complementary strengths of humans and computer agentsto solve crowdsourcing tasks more efficiently. We describe the operation of key components and overall architecture of a methodology we refer to as CrowdSynth, and demonstrate the operation and value of the methods with data and work- load drawn from a large-scale legacy crowdsourcing system for citizen science. We focus on solving consensus tasks, a large class of crowd- sourcing. With consensus tasks the goal is to identify a hid- den state of the world by collecting multiple assessments from human workers. Examples of consensus tasks include games with a purpose (e.g., image labeling in the ESP game) 13, paid crowdsourcing systems (e.g., product categoriza- tion in Mechanical Turk) 6, and citizen science projects(e.g., efforts to classify birds or celestial objects). Consen-sus efforts can be subtasks of larger tasks. For example, asystem for providing real-time traffic flow and predictions may contact drivers within targeted regions for reports ontraffic conditions 8. We describe a general system that combines machine learn- ing and decision-theoretic planning to guide the allocationof human effort in consensus tasks. Our work derives froma collaboration with the Galaxy Zoo citizen science effort 1, which serves as a rich domain and source of data for evaluating machine learning and planning methods as well as for studying the overall operation of an architecture forcrowdsourcing. The Galaxy Zoo effort was organized to seekhelp from volunteer citizen scientists on the classification of thousands of galaxies that were previously captured in an automated astronomical survey, known as the Sloan Digital Sky Survey (SDSS). The project has sought assessments via the collection of multiple votes from non-experts. Beyond votes, we have access to a database of SDSS image analysis data, containing 453 image features for each galaxy, which were extracted automatically via automated machine vision. We shall describe how successful optimization of the en- gagement of people with Galaxy Zoo tasks hinges on models learned from data that