Classification performance metric elicitation and its applications

Hiranandani, Gaurush

Permalink

https://hdl.handle.net/2142/113902 Copy

Description

Title

Classification performance metric elicitation and its applications

Author(s)

Hiranandani, Gaurush

Issue Date

2021-12-02

Director of Research (if dissertation) or Advisor (if thesis)

Koyejo, Oluwasanmi

Doctoral Committee Chair(s)

Koyejo, Oluwasanmi

Committee Member(s)

• Rayadurgam, Srikant
• Smaragdis, Paris
• Agarwal, Shivani

Department of Study

Computer Science

Discipline

Computer Science

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

Ph.D.

Degree Level

Dissertation

Keyword(s)

• fairness
• optimization
• user study
• multiclass
• binary
• multigroup
• frank-wolfe
• plug-in
• binary-search

Abstract

Given a learning problem with real-world tradeoffs, which cost function should the model be trained to optimize? This is the metric selection problem in machine learning. Despite its practical interest, there is limited formal guidance on how to select metrics for machine learning applications. This thesis outlines metric elicitation as a principled framework for selecting the performance metric that best reflects implicit user preferences. Once specified, the evaluation metric can be used to compare and train models. In this manuscript, we formalize the problem of Metric Elicitation and devise novel strategies for eliciting classification performance metrics using pairwise preference feedback over classifiers. Specifically, we provide novel strategies for eliciting linear and linear-fractional metrics for binary and multiclass classification problems, which are then extended to a framework that elicits group-fair performance metrics in the presence of multiple sensitive groups. All the elicitation strategies that we discuss are robust to both finite sample and feedback noise, thus are useful in practice for real-world applications. Using the tools and the geometric characterizations of the feasible confusion statistics space from the binary, multiclass, and multiclass-multigroup classification setups, we further provide strategies to elicit from a wider range of complex, modern multiclass metrics defined by quadratic functions of predictive rates by exploiting their local linear structure. This strategy can then be easily extended to eliciting metrics of higher order polynomials. From application perspective, we also propose to use the metric elicitation framework in optimizing complex black box metrics that is amenable to deep network training. In particular, the linear elicitation strategies can be used to elicit local-linear approximation of the black-box metrics, which are then exploited by existing iterative optimization routines. Lastly, to bring theory closer to practice, we conduct a preliminary real-user study that shows the efficacy of the metric elicitation framework in recovering the users' preferred performance metric in a binary classification setup.

Graduation Semester

2021-12

Type of Resource

Thesis

Permalink

http://hdl.handle.net/2142/113902

Copyright and License Information

Copyright 2021 Gaurush Hiranandani

Owning Collections