Files in this item



application/pdfShih-Wen_Huang.pdf (664kB)
(no description provided)PDF


Title:Peer consistency evaluation in human computation
Author(s):Huang, Shih-Wen
Advisor(s):Fu, Wai-Tat
Department / Program:Computer Science
Discipline:Computer Science
Degree Granting Institution:University of Illinois at Urbana-Champaign
human computation
Abstract:Peer consistency evaluation is often used in games with a purpose (GWAP) to evaluate workers using outputs of other workers without using gold standard answers. Despite its popularity, the reliability of peer consistency evaluation has never been systematically tested to show how it can be used as a general evaluation method in human computation systems. We present experimental results that show that human computation systems using peer consistency evaluation can lead to outcomes that are even better than those that evaluate workers using gold standard answers. We also show that even without evaluation, simply telling the workers that their answers will be used as future evaluation standards can significantly enhance the workers' performance. Results have important implication for methods that improve the reliability of human computation systems.
Issue Date:2013-08-22
Rights Information:Copyright 2013 Shih-Wen Huang
Date Available in IDEALS:2013-08-22
Date Deposited:2013-08

This item appears in the following Collection(s)

Item Statistics