Variational Classification

We present a latent variable model for classification that provides a novel probabilistic interpretation of neural network softmax classifiers. We derive a variational objective to train the model, analogous to the evidence lower bound (ELBO) used to train variational auto-encoders, that generalises the cross-entropy loss used to train classification models. Treating inputs to the softmax layer as samples of a latent variable, our abstracted perspective reveals a potential inconsistency between their anticipated distribution, required for accurate label predictions, and the empirical distribution they follow in practice. We then devise a variational objective to mitigate such inconsistency and encourage a specified latent distribution, instead of the implicit assumption in off-the-shelf softmax classifiers. Overall, we provide new theoretical insight into the inner workings of widely-used softmax classification; and empirical evaluation on image and text classification datasets demonstrates that our proposed remedy, variational classification, maintains classification accuracy while the reshaped latent space improves other desirable classifier properties, such as calibration, adversarial robustness, robustness to distribution shift and sample efficiency useful in low data settings.