Astopchatic Proximal 圆形步进大小 (A Stochastic Proximal Polyak Step Size)

Recently, the stochastic Polyak step size (SPS) has emerged as a competitive adaptive step size scheme for stochastic gradient descent. Here we develop ProxSPS, a proximal variant of SPS that can handle regularization terms. Developing a proximal variant of SPS is particularly important, since SPS requires a lower bound of the objective function to work well. When the objective function is the sum of a loss and a regularizer, available estimates of a lower bound of the sum can be loose. In contrast, ProxSPS only requires a lower bound for the loss which is often readily available. As a consequence, we show that ProxSPS is easier to tune and more stable in the presence of regularization. Furthermore for image classification tasks, ProxSPS performs as well as AdamW with little to no tuning, and results in a network with smaller weight parameters. We also provide an extensive convergence analysis for ProxSPS that includes the non-smooth, smooth, weakly convex and strongly convex setting.

翻译：最近,微小的聚氨酯步骤尺寸(SPS)已成为一种具有竞争力的适应性步骤尺寸计划,用于随机梯度下降。在这里,我们开发了Prox-SPS(SPS),这是SPS的近似变体,可以处理正规化条件。开发一个最接近的SPS变体特别重要,因为卫生和植物检疫要求目标功能的较低范围才能很好地发挥作用。当目标功能是损失总和和和和调节器时,对较低比例的估计数可以松动。相反,Prox-SPS(Prox-SPS)只要求较低范围的损失通常很容易得到。因此,我们表明Prox-SPS在正规化的情况下更容易调和和更加稳定。此外,对于图像分类任务,Prox-SPS的表现和AdamW(AdamW)几乎没有调整,结果网络的重量参数较小。我们还对Prox-SPS(Prox-SPS)进行了广泛的趋同分析,其中包括非湿、光滑、弱软软的螺旋和强烈的交汇设置。