Standard practice in training neural networks involves initializing the weights in an independent fashion. The results of recent work suggest that feature "diversity" at initialization plays an important role in training the network. However, other initialization schemes with reduced feature diversity have also been shown to be viable. In this work, we conduct a series of experiments aimed at elucidating the importance of feature diversity at initialization. Experimenting on a shallow network, we show that a complete lack of diversity is harmful to training, but its effect can be counteracted by a relatively small addition of noise. Furthermore, we construct a deep convolutional network with identical features at initialization and almost all of the weights initialized at 0 that can be trained to reach accuracy matching its standard-initialized counterpart.
翻译:培训神经网络的标准实践涉及独立地初始化权重。最近的工作结果表明,初始化时的“多样性”特征在培训网络中起着重要作用。然而,其他特征多样性减少的初始化计划也证明是可行的。在这项工作中,我们进行了一系列实验,旨在阐明初始化时特征多样性的重要性。在浅层网络上进行实验,我们发现,完全缺乏多样性对培训有害,但其影响可以通过相对较少的噪音来抵消。此外,我们建立了一个深度的革命网络,在初始化时具有相同的特征,几乎所有初始化的重量在零度上都可以被训练达到与其标准初始对应的精确度。