Bayesian Neural Networks: Evidence, HMC, and Function-Space Bayes

Summary. Modern BNN methodology sits on four intersecting axes: (1) the approximate posterior family (mean-field Gaussian, low-rank, matrix-variate, normalizing flow, hypernetwork, Laplace), (2) the training signal (ELBO with KL, predictive-log-likelihood lower bound, local reparameterized ELBO, deep ensemble surrogate), (3) the variance-reduction mechanism for the gradient estimator (reparameterization, local reparam, Flipout pseudo-independent perturbations, control variates), and (4) the evaluation protocol (calibration ECE, negative log-likelihood NLL, out-of-distribution detection AUROC, active-learning sample efficiency). The methods surveyed here populate this grid in characteristic ways.

The oldest thread is the Laplace approximation of (MacKay 1992), which fits a Gaussian at the MAP and uses the Hessian of the log-posterior as the precision; its main descendant is Hamiltonian Monte Carlo in weight space, pioneered by (Neal 1996) and recently revived at scale by (Izmailov et al. 2021). Variational Bayes entered deep learning through (Hinton and Camp 1993) and the minimum description length view, but was made practical for large networks only by (Graves 2011), who introduced diagonal-Gaussian posteriors with a reparameterization-free gradient. (Blundell et al. 2015) closed the remaining gap by formally using the reparameterization trick of (Kingma and Welling 2014) (concurrently (Rezende et al. 2014)) to obtain unbiased low-variance gradients through a Gaussian posterior, calling the resulting algorithm Bayes by Backprop. (Kingma et al. 2015) then showed that pushing the reparameterization from weight space into pre-activation space (the local trick) reduces gradient variance by a factor proportional to the minibatch size. (Wen et al. 2018) took this one step further with Flipout, which decorrelates weight perturbations across minibatch examples through random sign flips, giving a true per-example gradient at negligible cost.

A parallel development came from (Gal and Ghahramani 2016b), who reinterpreted dropout regularization as variational inference with a spike-and-slab posterior, making every network trained with dropout retroactively Bayesian; the RNN extension is (Gal and Ghahramani 2016a) and the continuous relaxation is (Gal et al. 2017). (Osband 2016) raised the fundamental objection that MC Dropout conflates aleatoric risk (the data distribution $p(y\mid x)$ variance) with epistemic uncertainty (posterior concentration over $\theta$), and that the MC-Dropout predictive variance does not shrink with more data as a true posterior should.

Flow-based posteriors extend the variational family beyond Gaussian. (Rezende and Mohamed 2015) introduced planar and radial flows for latent-variable VI, and (Louizos and Welling 2017) adapted this to weight-space by multiplicative normalizing flows (MNF), where a flow transforms an auxiliary random variable that scales Gaussian weight samples, yielding a richer, multimodal implicit posterior. Complementary threads include deep ensembles (Lakshminarayanan et al. 2017), SWAG (Maddox et al. 2019), rank-one BNNs (Dusenberry et al. 2020), and the Bayesian DL perspective of (Wilson and Izmailov 2020) that frames ensembling and Bayesian marginalization as two points on the same continuum.

State-of-the-art comparison table

The takeaway is an ordering rather than a set of gaps. Deep ensembles and HMC set the calibration ceiling. Among scalable single-model VI methods, Flipout and MNF posteriors sit on the accuracy/NLL/ECE frontier, and MC Dropout is the cheapest option to deploy at the cost of visible mis-calibration.

References