Director of Research (if dissertation) or Advisor (if thesis)
Koyejo, Oluwasanmi
Department of Study
Computer Science
Discipline
Computer Science
Degree Granting Institution
University of Illinois at Urbana-Champaign
Degree Name
M.S.
Degree Level
Thesis
Keyword(s)
Machine Learning
Active Learning
Metric Selection
Language
eng
Abstract
Metric elicitation is a type of inverse decision problem where the goal is to learn a loss function for classification using expert comparisons between candidate classifiers. However, for many practical tasks, such an expert can be noisy. Here we present a unified approach for learning metrics robust to constant and location-dependent noise models. Our approach takes advantage of the problem's similarity to probabilistic bisection search and uses pairwise comparisons to update a pseudo-belief distribution for the performance metric. Our theoretical results guarantee convergence in practical settings and extend beyond previous results to include multi-expert elicitation. Quantitative comparisons against existing methods for performance metric elicitation and inverse decision theory demonstrate the advantage of our approach.
Use this login method if you
don't
have an
@illinois.edu
email address.
(Oops, I do have one)
IDEALS migrated to a new platform on June 23, 2022. If you created
your account prior to this date, you will have to reset your password
using the forgot-password link below.
