与群体平等一起培训Leaky ReLU神经网络的不精密增强Lagrangian Algoorithm (An Inexact Augmented Lagrangian Algorithm for Training Leaky ReLU Neural Network with Group Sparsity)

The leaky ReLU network with a group sparse regularization term has been widely used in the recent years. However, training such a network yields a nonsmooth nonconvex optimization problem and there exists a lack of approaches to compute a stationary point deterministically. In this paper, we first resolve the multi-layer composite term in the original optimization problem by introducing auxiliary variables and additional constraints. We show the new model has a nonempty and bounded solution set and its feasible set satisfies the Mangasarian-Fromovitz constraint qualification. Moreover, we show the relationship between the new model and the original problem. Remarkably, we propose an inexact augmented Lagrangian algorithm for solving the new model and show the convergence of the algorithm to a KKT point. Numerical experiments demonstrate that our algorithm is more efficient for training sparse leaky ReLU neural networks than some well-known algorithms.

翻译：使用群体稀疏的正规化术语的泄漏 ReLU 网络近年来被广泛使用。但是, 培训这样的网络会产生一个非光滑的非convex优化问题, 并且缺乏计算固定点的方法。在本文中, 我们首先通过引入辅助变量和额外的限制来解决原始优化问题的多层次复合术语。我们显示新模型有一个非空和封闭的解决方案集, 并且其可行的集成符合Mangasarian- Fromovitz 限制条件。此外, 我们展示了新模型和原始问题之间的关系。值得注意的是, 我们建议用不精确的拉格朗格算法来解决新模型, 并显示算法与 KKT 点的趋同。数字实验表明, 我们的算法比一些众所周知的算法更高效地培训稀疏漏的ReLU 神经网络。