Differentially private federated learning (FL) entails bounding the sensitivity to each client's update. The customary approach used in practice for bounding sensitivity is to \textit{clip} the client updates, which is just projection onto an $\ell_2$ ball of some radius (called the clipping threshold) centered at the origin. However, clipping introduces bias depending on the clipping threshold and its impact on convergence has not been properly analyzed in the FL literature. In this work, we propose a simpler alternative for bounding sensitivity which is \textit{normalization}, i.e. use only the \textit{unit vector} along the client updates, completely discarding the magnitude information. We call this algorithm \texttt{DP-NormFedAvg} and show that it has the same order-wise convergence rate as \texttt{FedAvg} on smooth quasar-convex functions (an important class of non-convex functions for modeling optimization of deep neural networks) modulo the noise variance term (due to privacy). Further, assuming that the per-sample client losses obey a strong-growth type of condition, we show that with high probability, the sensitivity reduces by a factor of $\mathcal{O}(\frac{1}{m})$, where $m$ is the minimum number of samples within a client, compared to its worst-case value. Using this high probability sensitivity value enables us to reduce the iteration complexity of \texttt{DP-NormFedAvg} by a factor of $\mathcal{O}(\frac{1}{m^2})$, at the expense of an exponentially small degradation in the privacy guarantee. We also corroborate our theory with experiments on neural networks.
翻译:不同私密的友式学习 (FL) 包含对每个客户端更新的敏感度 。 用于约束敏感度的习惯方法是 & textit{ clip} 客户端更新, 它只是投射到某个半径( 称为剪切阈值) 的 $ ell_ 2$球上。 但是, 剪切会根据剪切阈值产生偏差, 它对于趋同的影响没有在 FL 文献中进行正确分析 。 在这项工作中, 我们建议一个更简单的调试敏感度的替代方法, 即 调试客户端更新时只使用\ textit{ 缩略度} 线性向量, 完全放弃数量信息 。 我们称之为算算法 = 0. 2 球球球球球在某个半径( 剪切阈值 ), 但它在光量 QSarsar- convex 函数( 一种重要的非convex 功能, 用于模拟深层内线网络的模型优化), 也就是调和最坏的客户端变音值 。