Privacy in AI remains a topic that draws attention from researchers and the general public in recent years. As one way to implement privacy-preserving AI, differentially private learning is a framework that enables AI models to use differential privacy (DP). To achieve DP in the learning process, existing algorithms typically limit the magnitude of gradients with a constant clipping, which requires carefully tuned due to its significant impact on model performance. As a solution to this issue, latest works NSGD and Auto-S innovatively propose to use normalization instead of clipping to avoid hyperparameter tuning. However, normalization-based approaches like NSGD and Auto-S rely on a monotonic weight function, which imposes excessive weight on small gradient samples and introduces extra deviation to the update. In this paper, we propose a Differentially Private Per-Sample Adaptive Clipping (DP-PSAC) algorithm based on a non-monotonic adaptive weight function, which guarantees privacy without the typical hyperparameter tuning process of using a constant clipping while significantly reducing the deviation between the update and true batch-averaged gradient. We provide a rigorous theoretical convergence analysis and show that with convergence rate at the same order, the proposed algorithm achieves a lower non-vanishing bound, which is maintained over training iterations, compared with NSGD/Auto-S. In addition, through extensive experimental evaluation, we show that DP-PSAC outperforms or matches the state-of-the-art methods on multiple main-stream vision and language tasks.
翻译:AI 的隐私仍然是近年来引起研究人员和一般公众注意的一个话题。作为实施隐私保护AI的一个途径,差异私人学习是一个使AI模型能够使用差异隐私的框架。为了在学习过程中实现DP。为了在学习过程中实现DP,现有的算法通常会通过不断剪切来限制梯度的大小,这需要谨慎调整,因为其对于模型性能的重大影响。作为这一问题的解决方案,NSGD和Auto-S最新作品NSGD和Auto-S创新地提议使用常规化而不是剪辑来避免超度调。然而,NSGD和Auto-S等基于正常化的办法依赖于单一重量功能,对小型梯度样本施加过重,并对更新引入额外的偏差。在本文中,我们提议了一种差异性私人PSAPPSAC(DP-PSAC) 算法,它基于对模型性能的适应性能功能,保证隐私不受典型的超标度调整过程的超标度,同时大幅降低更新和真实的批量梯度偏差。我们提供了严格的理论趋同性趋同分析,并显示不比的NSAL-AD-ADADADADADAV的排序。