2026-09-20T18:55:01Z https://www.ideals.illinois.edu/oai-pmh

oai:www.ideals.illinois.edu:2142/45484 2023-07-11 col_2142_5131 col_2142_10761 com_2142_5130 com_2142_10755 com_2142_234

Fu, Wai-Tat Huang, Shih-Wen 2013-08-22T16:41:36Z 2013-08-22T16:41:36Z 2013-08 2013-08-22T16:41:36Z 2013-08 Peer consistency evaluation is often used in games with a purpose (GWAP) to evaluate workers using outputs of other workers without using gold standard answers. Despite its popularity, the reliability of peer consistency evaluation has never been systematically tested to show how it can be used as a general evaluation method in human computation systems. We present experimental results that show that human computation systems using peer consistency evaluation can lead to outcomes that are even better than those that evaluate workers using gold standard answers. We also show that even without evaluation, simply telling the workers that their answers will be used as future evaluation standards can significantly enhance the workers' performance. Results have important implication for methods that improve the reliability of human computation systems. Item withdrawn by Mark Zulauf (zulauf@illinois.edu) on 2013-07-18T13:08:20Z Item was in collections: University of Illinois Theses & Dissertations (ID: 1) No. of bitstreams: 1 Huang_Shih-Wen.pdf: 663704 bytes, checksum: e56dfac50e3ed80dce8e92b6276a1b2c (MD5) Made available in DSpace on 2013-08-22T16:41:36Z (GMT). No. of bitstreams: 2 Shih-Wen_Huang.pdf: 663704 bytes, checksum: e56dfac50e3ed80dce8e92b6276a1b2c (MD5) license.txt: 4064 bytes, checksum: dc66a071322548a9a3d72e111fefc518 (MD5) http://hdl.handle.net/2142/45484 en Copyright 2013 Shih-Wen Huang crowdsourcing human computation Peer consistency evaluation in human computation text Computer Science 1434 Computer Science 0112 University of Illinois at Urbana-Champaign Thesis M.S. MS:Computer Science -UIUC 10KS0112MS