Since the Lipschitz properties of convolutional neural network (CNN) are widely considered to be related to adversarial robustness, we theoretically characterize the $\ell_1$ norm and $\ell_\infty$ norm of 2D multi-channel convolutional layers and provide efficient methods to compute the exact $\ell_1$ norm and $\ell_\infty$ norm. Based on our theorem, we propose a novel regularization method termed norm decay, which can effectively reduce the norms of CNN layers. Experiments show that norm-regularization methods, including norm decay, weight decay, and singular value clipping, can improve generalization of CNNs. However, we are surprised to find that they can slightly hurt adversarial robustness. Furthermore, we compute the norms of layers in the CNNs trained with three different adversarial training frameworks and find that adversarially robust CNNs have comparable or even larger norms than their non-adversarially robust counterparts. Moreover, we prove that under a mild assumption, adversarially robust classifiers can be achieved with neural networks and an adversarially robust neural network can have arbitrarily large Lipschitz constant. For these reasons, enforcing small norms of CNN layers may be neither effective nor necessary in achieving adversarial robustness. Our code is available at https://github.com/youweiliang/norm_robustness.
翻译:由于人们广泛认为利普西茨神经网络(CNN)的利普西茨特性与对抗性稳健性有关,因此,我们理论上将2D多渠道共振层的规范与2D多渠道共振层的规范划为1美元标准与1美元标准,并提供计算准确的1美元标准与1美元标准的有效方法。根据我们的理论,我们提议一种称为规范衰败的新颖的规范化方法,这可以有效减少CNN层的规范化规范化规范。实验表明,规范化的规范化方法,包括规范腐蚀、重量衰减和单值剪切片,可以改进CNN的通用性。然而,我们惊讶地发现,它们可能略微伤害到对抗性强的规范。此外,我们用三种不同的对抗性培训框架来计算受训练的CNNCM的规范,发现强势的CNN的规范比非对抗性强的规范要相似甚至更大。此外,我们证明,在一种温和的假设下,通过神经网络和稳健的神经网络可以达到必要的强势性标准。