We study how neural networks trained by gradient descent extrapolate, i.e., what they learn outside the support of the training distribution. Previous works report mixed empirical results when extrapolating with neural networks: while feedforward neural networks, a.k.a. multilayer perceptrons (MLPs), do not extrapolate well in certain simple tasks, Graph Neural Networks (GNNs) -- structured networks with MLP modules -- have shown some success in more complex tasks. Working towards a theoretical explanation, we identify conditions under which MLPs and GNNs extrapolate well. First, we quantify the observation that ReLU MLPs quickly converge to linear functions along any direction from the origin, which implies that ReLU MLPs do not extrapolate most nonlinear functions. But, they can provably learn a linear target function when the training distribution is sufficiently "diverse". Second, in connection to analyzing the successes and limitations of GNNs, these results suggest a hypothesis for which we provide theoretical and empirical evidence: the success of GNNs in extrapolating algorithmic tasks to new data (e.g., larger graphs or edge weights) relies on encoding task-specific non-linearities in the architecture or features. Our theoretical analysis builds on a connection of over-parameterized networks to the neural tangent kernel. Empirically, our theory holds across different training settings.
翻译:我们研究的是,由梯度下下坡外推外推,即它们从培训分布所支持的外推外推学到了什么,所训练的神经网络是如何在较梯度下外推外推而出,即它们所学到的在培训分布外推之外外推外推外推的。我们研究的是,由梯度下外推外推外推外推,即它们从培训分布分布外外推外推外推外推外推的神经网络所训练的神经网络是如何在更复杂的任务中表现出某种成功的。我们努力从理论上解释,我们找出了MLPs和GNNNNS在支持培训外外外外外推的设置。首先,我们量化了以下观察,即RELU MLPs在向神经神经神经网络( a.k.a.a.a.a.a.a.a.a.a.a.a.a.a.a.a.a.a.a.a.a.a.a.a.a.a.a.-培训网外加内GNNNNIN结构结构、ILLLPLP任务的升级结构结构,即不.ble等等等结构,或等等等等等等结构的理论结构,为新数据。