BiD Seminar Series 2/7: Collaboratively Crowdsourcing Workflows with Turkomatic && Shepherding the Crowd Yields Better Work
Björn Hartmann and Anand Kulkami Collaboratively Crowdsourcing Workflows with Turkomatic && Shepherding the Crowd Yields Better Work Tuesday, February 7 12:00 - 1:00pm Berkeley Institute of Design (BiD) Lab, 354/360 HMMB (http://bid.berkeley.edu/directions) TITLE: Collaboratively Crowdsourcing Workflows with Turkomatic ABSTRACT: Online crowdsourcing is difficult, requiring us to decompose complex tasks into sequences of simple tasks that individual agents can carry out. How can we use the crowd to make crowdsourcing easier? We present Turkomatic, a tool that recruits crowd agents to aid requesters in planning and solving complex jobs. Requesters can view the status of crowd-designed workflows in real time, intervene to change tasks and solutions, and request new solutions to subtasks from the crowd. These features lower the threshold for crowd employers to request complex work. Turkomatic’s collaborative approach enables us to crowdsource new and more complex types of tasks than ever before. TITLE: Shepherding the Crowd Yields Better Work ABSTRACT: Micro-task platforms provide massively parallel, on-demand labor. However, it can be difficult to reliably achieve high-quality work because online workers may behave irresponsibly, misunderstand the task, or lack necessary skills. This paper investigates whether timely, task-specific feedback helps crowd workers learn, persevere, and produce better results. We investigate this question through Shepherd, a feedback system for crowdsourced work. In a between-subjects study with three conditions, crowd workers wrote consumer reviews for six products they own. Participants in the None condition received no immediate feedback, consistent with most current crowdsourcing practices. Participants in the Self-assessment condition judged their own work. Participants in the External assessment condition received expert feedback. Self-assessment alone yielded better overall work than the None condition and helped workers improve over time. External assessment also yielded these benefits. Participants who received external assessment also revised their work more. We conclude by discussing interaction and infrastructure approaches for integrating real-time assessment into online work. BIO: Anand Kulkarni is a PhD candidate in the Department of IEOR at the University of California, Berkeley, and a co-founder of MobileWorks. Björn Hartmann is an Assistant Professor in the Department of Electrical Engineering and Computer Sciences at the University of California, Berkeley.
