Uncertainty Quantification from Deep Hyperparameter Ensembles

Mallick, Tanwi; Macfarlane, Jane; Balaprakash, Prasanna

Permalink

https://hdl.handle.net/2142/121816 Copy

Description

Title

Uncertainty Quantification from Deep Hyperparameter Ensembles

Author(s)

• Mallick, Tanwi
• Macfarlane, Jane
• Balaprakash, Prasanna

Issue Date

2023

Keyword(s)

• Uncertainty quantification
• Deep ensemble
• Hyperparameter search

Abstract

Deep neural networks have achieved impressive performance on a wide range of machine learning tasks and are gaining popularity in a variety of disciplines. However, a major limitation of these methods is that they provide prediction without estimates of data and model uncertainty. Uncertainty quantification (UQ) is important to know when to trust the model forecasts. UQ enables decision-makers to understand the level of uncertainty associated with each prediction and make informed decisions based on the risks associated with each outcome. UQ is critical for understanding inherent variations of the data and prediction limitations due to lack of training data. Therefore, UQ can help identify areas where additional data or model improvements may be needed to reduce the level of uncertainty. To that end, we develop a scalable deep hyperparameter ensemble approach to quantify data and model uncertainties. Our approach consists of four stages: (1) using a Gaussian-assumption-free simultaneous quantile regression loss for training a neural network to model the data distribution; (2) applying a scalable Bayesian optimization method to tune the hyperparameters of the neural network; (3) fitting a Gaussian copula generative model to capture the joint distributions of the high-performing hyperparameter configurations, and training an ensemble of models by sampling a new set of hyperparameter configurations from the generative model; and (4) decomposing the data (aleatoric) and model (epistemic) uncertainties from the neural network ensembles using variance decomposition method. We demonstrated the effectiveness of our approach on a diffusion convolutional recurrent neural network, a state-of-the-art method for short-term traffic forecasting. We demonstrated the efficacy of our ensemble- based uncertainty quantification method by comparing it with other uncertainty estimation techniques. We show that our generic and scalable approach outperforms the current state-of-the-art Bayesian and several other commonly used frequentist uncertainty estimation techniques. The key advantages of our approach are generality and scalability. Our proposed approach for building model ensembles is based on a scalable hyperparameter search and computationally inexpensive generative modeling. Our UQ approach can be applied to any neural network model by using SQR loss, running hyperparameters search, training a generative model from the hyperparameters search runs, performing model selection, and decomposing the uncertainty estimates. The scalability of the approach is derived from the scalable hyperparameter search and scalable ensemble training.

Type of Resource

text

Language

eng

Handle URL

https://hdl.handle.net/2142/121816

Owning Collections