Applying Bayesian inference to neural networks often requires approximating the posterior over parameters with simple distributions. The quality of the resulting approximate predictive distribution in function space is poorly understood. We prove that for single hidden layer ReLU networks, there exist simple situations where it is impossible for factorised Gaussian or MC dropout posteriors to give well-calibrated uncertainty estimates. Precisely, they cannot both fit the data confidently and have increased uncertainty in between well-separated clusters of data. This motivates more careful consideration of the consequences of approximate inference in Bayesian neural networks.
翻译:对神经网络应用贝耶斯语推论往往要求以简单分布方式对参数的后方值进行近似比对准。由此得出的功能空间近似预测分布的质量不甚清楚。我们证明,对于单一隐蔽层ReLU 网络,存在一种简单的情况,即无法将高山或MC的辍学后方人为因素进行充分校准的不确定性估计。确切地说,它们无法既符合数据自信,又增加了分离良好的数据组群之间的不确定性。这促使人们更仔细地考虑贝亚斯神经网络中近似推论的后果。