Understanding how neural networks learn remains one of the central challenges in machine learning research. From random at the start of training, the weights of a neural network evolve in such a way as to be able to perform a variety of tasks, like classifying images. Here we study the emergence of structure in the weights by applying methods from topological data analysis. We train simple feedforward neural networks on the MNIST dataset and monitor the evolution of the weights. When initialized to zero, the weights follow trajectories that branch off recurrently, thus generating trees that describe the growth of the effective capacity of each layer. When initialized to tiny random values, the weights evolve smoothly along two-dimensional surfaces. We show that natural coordinates on these learning surfaces correspond to important factors of variation.
翻译:了解神经网络如何学习仍然是机器学习研究的中心挑战之一。 从培训开始时随机,神经网络的重量发展到能够执行各种任务的方式,例如图像分类。在这里,我们通过应用地形学数据分析方法研究重量结构的出现。我们在MNIST数据集上培训简单的进料向神经网络,并监测重量的演变。在初始化为零时,重量跟随经常断开的轨迹,从而产生描述每一层有效能力增长的树木。在初始化为微小随机值时,重量沿两维表面顺利演变。我们显示,这些学习表面的自然坐标与重要的变异因素相对应。