SLADE: A Smart Large-Scale Task Decomposer in Crowdsourcing

Crowdsourcing has been shown to be effective in a wide range of applications, and is seeing increasing use. A large-scale crowdsourcing task often consists of thousands or millions of atomic tasks, each of which is usually a simple task such as binary choice or simple voting. To distribute a large-s...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on knowledge and data engineering 2018-08, Vol.30 (8), p.1588-1601
Main Authors: Tong, Yongxin, Chen, Lei, Zhou, Zimu, Jagadish, H. V., Shou, Lidan, Lv, Weifeng
Format: Article
Language:English
Subjects:
Approximation
Approximation algorithms
Computer science
Cost engineering
Crowdsourcing
Decomposition
Electronic mail
Greedy algorithms
Heuristic methods
Mathematical analysis
Queues
Reliability
Reliability engineering
Task analysis
task decomposition
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Crowdsourcing has been shown to be effective in a wide range of applications, and is seeing increasing use. A large-scale crowdsourcing task often consists of thousands or millions of atomic tasks, each of which is usually a simple task such as binary choice or simple voting. To distribute a large-scale crowdsourcing task to limited crowd workers, a common practice is to pack a set of atomic tasks into a task bin and send to a crowd worker in a batch. It is challenging to decompose a large-scale crowdsourcing task and execute batches of atomic tasks, which ensures reliable answers at a minimal total cost. Large batches lead to unreliable answers of atomic tasks, while small batches incur unnecessary cost. In this paper, we investigate a general crowdsourcing task decomposition problem, called the S mart L arge-sc A le task DE composer (SLADE) problem, which aims to decompose a large-scale crowdsourcing task to achieve the desired reliability at a minimal cost. We prove the NP-hardness of the SLADE problem and propose solutions in both homogeneous and heterogeneous scenarios. For the homogeneous SLADE problem, where all the atomic tasks share the same reliability requirement, we propose a greedy heuristic algorithm and an efficient and effective approximation framework using an optimal priority queue (OPQ) structure with provable approximation ratio. For the heterogeneous SLADE problem, where the atomic tasks can have different reliability requirements, we extend the OPQ-based framework leveraging a partition strategy, and also prove its approximation guarantee. Finally, we verify the effectiveness and efficiency of the proposed solutions through extensive experiments on representative crowdsourcing platforms.
ISSN:1041-4347
1558-2191
DOI:10.1109/TKDE.2018.2797962