Constraining linear layers in neural networks to respect symmetry transformations from a group $G$ is a common design principle for invariant networks that has found many applications in machine learning. In this paper, we consider a fundamental question that has received little attention to date: Can these networks approximate any (continuous) invariant function? We tackle the rather general case where $G\leq S_n$ (an arbitrary subgroup of the symmetric group) that acts on $\mathbb{R}^n$ by permuting coordinates. This setting includes several recent popular invariant networks. We present two main results: First, $G$-invariant networks are universal if high-order tensors are allowed. Second, there are groups $G$ for which higher-order tensors are unavoidable for obtaining universality. $G$-invariant networks consisting of only first-order tensors are of special interest due to their practical value. We conclude the paper by proving a necessary condition for the universality of $G$-invariant networks that incorporate only first-order tensors. Lastly, we propose a conjecture stating that this condition is also sufficient.
翻译:在神经网络中对线性层进行约束,以尊重来自一个集团$G$的对称变换,这是在机器学习中发现许多应用的变换网络的共同设计原则。在本文中,我们考虑一个迄今很少引起注意的根本问题:这些网络能否接近任何(连续的)变换功能?我们处理一个相当普遍的情况,即$G\leq S_n美元(一个对称组的任意分组)通过变换坐标对$mathbb{R ⁇ n$采取行动。这一设置包括最近一些受欢迎的变换网络。我们提出两个主要结果:第一,$G$-变换网络如果允许高压,是普遍性的。第二,有几组G$G$是获得普遍性所不可避免的。$G$-变换网络只有一阶的电压才对其实际价值具有特殊兴趣。我们通过证明一个仅包含第一阶调高压器的变换网络的普遍性的必要条件来完成文件的结尾。我们提出一个充分的假设,即这一条件也是充分的。