Dropout is a popular regularization technique in neural networks. Yet, the reason for its success is still not fully understood. This paper provides a new interpretation of Dropout from a frame theory perspective. By drawing a connection to recent developments in analog channel coding, we suggest that for a certain family of autoencoders with a linear encoder, the minimizer of an optimization with dropout regularization on the encoder is an equiangular tight frame (ETF). Since this optimization is non-convex, we add another regularization that promotes such structures by minimizing the cross-correlation between filters in the network. We demonstrate its applicability in convolutional and fully connected layers in both feed-forward and recurrent networks. All these results suggest that there is indeed a relationship between dropout and ETF structure of the regularized linear operations.
翻译:失学是神经网络中流行的正规化技术。 然而, 成功的原因仍然没有得到完全理解。 本文从框架理论角度对辍学作了新的解释。 通过将模拟频道编码的最新发展联系起来, 我们建议, 对于某个具有线性编码器的自动编码器家庭来说, 将离校正规化的优化最小化是一个等角紧框架。 由于这种优化是非凝固的, 我们添加了另一个通过将网络过滤器之间的交叉关系最小化来促进这种结构的正规化。 我们展示了它在进源前网络和经常网络的动态和完全连接层中的可适用性。 所有这些结果都表明,在正常线性操作的辍学和非正规化的离校结构之间确实存在某种关系。