The matrix-based Renyi's \alpha-entropy functional and its multivariate extension were recently developed in terms of the normalized eigenspectrum of a Hermitian matrix of the projected data in a reproducing kernel Hilbert space (RKHS). However, the utility and possible applications of these new estimators are rather new and mostly unknown to practitioners. In this paper, we first show that our estimators enable straightforward measurement of information flow in realistic convolutional neural networks (CNN) without any approximation. Then, we introduce the partial information decomposition (PID) framework and develop three quantities to analyze the synergy and redundancy in convolutional layer representations. Our results validate two fundamental data processing inequalities and reveal some fundamental properties concerning the training of CNN.
翻译:以矩阵为基础的Renyi 的alpha-entoprotiy功能及其多变扩展最近发展成为在复制Hilbert 空间(RKHS)复制的Hermitian 空间(RKHS)中预测数据表的正常电子光谱,但是,这些新估算器的效用和可能应用相当新,而且从业人员大多不知道。在本文中,我们首先显示我们的测算器能够直截了当地测量现实的神经神经网络的信息流动。然后,我们引入部分信息分解(PID)框架,并开发三批数据,以分析革命层代表的协同作用和冗余。我们的结果证实了两个基本的数据处理不平等,并揭示了有关CNN培训的一些基本特征。