In this paper, we study deep fully circulant neural networks, that is deep neural networks in which all weight matrices are circulant ones. We show that these networks outperform the recently introduced deep networks with other types of structured layers. Besides introducing principled techniques for training these models, we provide theoretical guarantees regarding their expressivity. Indeed, we prove that the function space spanned by circulant networks of bounded depth includes the one spanned by dense networks with specific properties on their rank. We conduct a thorough experimental study to compare the performance of deep fully circulant networks with state of the art models based on structured matrices and with dense models. We show that our models achieve better accuracy than their structured alternatives while required 2x fewer weights as the next best approach. Finally we train deep fully circulant networks to build a compact and accurate models on a real world video classification dataset with over 3.8 million training examples.
翻译:在本文中,我们研究了深厚的电动神经网络,即所有重力矩阵都是电动神经网络的深神经网络。我们发现这些网络优于最近引进的与其他类型结构化层的深网络。除了采用原则性技术来培训这些模型外,我们还为这些模型的表达性提供了理论保障。事实上,我们证明,封闭深度的电动网络所覆盖的功能空间包括由具有特定特性的密度网络所覆盖的宽广网络所覆盖的宽广网络。我们进行了彻底的实验研究,以比较深厚的电动网络的性能与基于结构化矩阵和密度模型的先进模型的状态。我们表明,我们的模型比结构化替代品的准确性更高,而下一个最佳方法则需要减少2x重量。最后,我们培训深厚的电动网络,以建立一个具有380万个培训范例的真正的世界视频分类数据集成的压缩和准确模型。