In this paper, we consider the problem of learning functions over sets, i.e., functions that are invariant to permutations of input set items. Recent approaches of pooling individual element embeddings can necessitate extremely large embedding sizes for challenging functions. We address this challenge by allowing standard neural networks like LSTMs to succinctly capture the function over the set. However, to ensure invariance with respect to permutations of set elements, we propose a novel architecture called SPAN that simultaneously learns the function as well as adversarial or worst-case permutations for each input set. The learning problem reduces to a min-max optimization problem that is solved via a simple alternating block coordinate descent technique. We conduct extensive experiments on a variety of set-learning tasks and demonstrate that SPAN learns nearly permutation-invariant functions while still ensuring accuracy on test data. On a variety of tasks sampled from the domains of statistics, graph functions and linear algebra, we show that our method can significantly outperform state-of-the-art methods such as DeepSets and Janossy Pooling. Finally, we present a case study of how learning set-functions can help extract powerful features for recommendation systems, and show that such a method can be as much as 2% more accurate than carefully hand-tuned features on a real-world recommendation system.
翻译:在本文中,我们考虑的是各组的学习功能问题,即不同功能与输入集项目的变换不一的功能。最近集中单个元素嵌入的方法可能要求为具有挑战性功能的极大嵌入尺寸。我们通过允许LSTMs等标准神经网络简洁地捕捉成套元素的功能来应对这一挑战。然而,为了确保对设定元素的变异性,我们提议了一个叫SPAN的新结构,它同时学习每个输入集的功能以及对抗性或最坏的变异性。学习问题降低到微量最大优化问题,通过简单的交替区块协调世系技术解决。我们在许多设定学习任务上进行了广泛的实验,并表明SPAN在仍然确保测试数据的准确性的同时,学习了近乎变异性功能。关于从统计、图形功能和线性代数领域抽样的各种任务,我们表明我们的方法可以大大超越了DeepSetset和Janossy Globalling等最新方法。最后,我们展示了一种更精确的案例研究特征,作为更精准的案例研究,我们展示了一种更精确的案例研究,可以如何建立更精确的系统。