2026-09-21T10:05:20Z https://www.ideals.illinois.edu/oai-pmh

oai:www.ideals.illinois.edu:2142/115425 2023-07-11 col_2142_5131 col_2142_10761 com_2142_5130 com_2142_10755 com_2142_234

Koyejo, Oluwasanmi 2022-05 application/pdf en text Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2022-11-11 without embargo terms The student, Zachary Robertson, accepted the attached license on 2022-04-21 at 13:58. The student, Zachary Robertson, submitted this Thesis for approval on 2022-04-21 at 14:03. This Thesis was approved for publication on 2022-04-25 at 14:50. DSpace SAF Submission Ingestion Package generated from Vireo submission #17885 on 2022-11-11 at 13:42:49 Probabilistic performance metric elicitation Robertson, Zachary 2022-04-25 Machine Learning Active Learning Metric Selection Metric elicitation is a type of inverse decision problem where the goal is to learn a loss function for classification using expert comparisons between candidate classifiers. However, for many practical tasks, such an expert can be noisy. Here we present a unified approach for learning metrics robust to constant and location-dependent noise models. Our approach takes advantage of the problem's similarity to probabilistic bisection search and uses pairwise comparisons to update a pseudo-belief distribution for the performance metric. Our theoretical results guarantee convergence in practical settings and extend beyond previous results to include multi-expert elicitation. Quantitative comparisons against existing methods for performance metric elicitation and inverse decision theory demonstrate the advantage of our approach. Thesis eng https://hdl.handle.net/2142/115425 Copyright 2022 Zachary Robertson M.S. Thesis Computer Science University of Illinois at Urbana-Champaign Computer Science