The increasing demand for privacy and security has driven the advancement of private inference (PI), a cryptographic method enabling inferences directly on encrypted data. However, the computational and storage burdens of non-linear operators (e.g., ReLUs) render it impractical. Despite these limitations, prior ReLU optimization methods consistently relied on classical networks, that are not optimized for PI. Moreover, the selection of baseline networks in these ReLU optimization methods remains enigmatic and fails to provide insights into network attributes contributing to PI efficiency. In this paper, we investigate the desirable network architecture for efficient PI, and {\em key finding} is wider networks are superior at higher ReLU counts, while networks with a greater proportion of least-critical ReLUs excel at lower ReLU counts. Leveraging these findings, we develop a novel network redesign technique (DeepReShape) with a complexity of $\mathcal{O}(1)$, and synthesize specialized architectures(HybReNet). Compared to the state-of-the-art (SNL on CIFAR-100), we achieve a 2.35\% accuracy gain at 180K ReLUs, and for ResNet50 on TinyImageNet our method saves 4.2$\times$ ReLUs at iso-accuracy.
翻译:随着对隐私和安全的需求增加,隐私推断(PI)的发展促进了加密数据上的推断。然而,非线性算子(如ReLU)的计算和存储负担使得其在实践中不可行。尽管存在这些限制,以往的ReLU优化方法一直依赖于传统网络,这些网络并未针对PI进行优化。此外,在这些ReLU优化方法中选择基线网络仍然是神秘的,未能提供有助于PI效率的网络属性的洞见。在本文中,我们研究了用于高效PI的理想网络体系结构,并发现更宽的网络在更高的ReLU计数下优越,而具有更高比例的最不关键的ReLUs的网络在更低的ReLU计数下表现优异。利用这些发现,我们开发了一种复杂度为$\mathcal{O}(1)$的新网络重设计技术(DeepReshape),并综合了专门的体系结构(HybReNet)。与最先进的方法(在CIFAR-100上的SNL)相比,我们在180K ReLUs时取得了2.35\%的精度增益,并且在同等精度下,与ResNet50相比我们的方法节省了4.2倍的ReLUs。