Appendix C. Other considerations and notable methods

This guide addresses the most practical and effective ways to ascertain reliable measures of uncertainty. It also addresses some of the major pathologies such as out-of-distribution degeneration and deterministic overconfidence. Other recent techniques include deterministic uncertainty quantification (DUQ) (van Amersfoort et al. 2020) and prediction-time batch normalization (Nado et al. 2020).

DUQs are a new kind of deep learning classifier that do not utilize the traditional softmax function. Instead, DUQs provide reliable uncertainty for out-of-distribution data. DUQs output a vector, f(x), which is transformed by a class-specific weight matrix, W_c, for mapping to a feature vector. The distance between this feature vector and learnt centroids (one centroid for each class) represents the corresponding uncertainties. The distance to the closest centroid is deemed the predictive uncertainty. Feature vectors are able to map far from centroids for out-of-distribution data by regularizing the model’s smoothness. The novel regularization method tunes smoothness so that changes of output coincide with changes in input, without changing so much that it compromises generalization. DUQs are a promising new way for modeling uncertainty and provide an alternative to deep ensembles for reliable uncertainty in out-of-distribution settings. For details, see the publications in the References section.

Another method worth noting is prediction-time batch normalization for out-of-distribution robustness (Nado et al. 2020). This technique requires just a few lines of code to implement and claims to improve uncertainty reliability with out-of-distribution data in a way that is complementary to deep ensembles. An interesting caveat to this method is that the quality of uncertainty actually degenerates for pretraining settings, which raises questions for future work.