IMDEA Networks Institute Publications Repository

The Importance of Worker Reputation Information in Microtask-Based Crowd Work Systems

Tarable, Alberto and Nordio, Alessandro and Leonardi, Emilio and Ajmone Marsan, Marco (2017) The Importance of Worker Reputation Information in Microtask-Based Crowd Work Systems. [Journal Articles]

[img] PDF (The Importance of Worker Reputation Information in Microtask-Based Crowd Work Systems) - Published Version
Download (697Kb)

Abstract

This paper presents the first systematic investigation of the potential performance gains for crowd work systems, deriving from available information at the requester about individual worker reputation. In particular, we first formalize the optimal task assignment problem when workers' reputation estimates are available, as the maximization of a monotone (submodular) function subject to Matroid constraints. Then, being the optimal problem NP-hard, we propose a simple but efficient greedy heuristic task allocation algorithm. We also propose a simple “maximum a-posteriori” decision rule and a decision algorithm based on message passing. Finally, we test and compare different solutions, showing that system performance can greatly benefit from information about workers' reputation. Our main findings are that: i) even largely inaccurate estimates of workers' reputation can be effectively exploited in the task assignment to greatly improve system performance; ii) the performance of the maximum a-posteriori decision rule quickly degrades as worker reputation estimates become inaccurate; iii) when workers' reputation estimates are significantly inaccurate, the best performance can be obtained by combining our proposed task assignment algorithm with the message-passing decision algorithm.

Item Type: Journal Articles
Uncontrolled Keywords: Human-centered computing, human information processing, systems and information.
Subjects: UNSPECIFIED
Divisions: UNSPECIFIED
Depositing User: Rebeca De Miguel
Date Deposited: 17 Mar 2017 14:46
Last Modified: 21 Mar 2017 08:40
URI: http://eprints.networks.imdea.org/id/eprint/1548

Actions (login required)

View Item View Item