Differentially private synthetic data generation offers a recent solution to release analytically useful data while preserving the privacy of individuals in the data. In order to utilize these algorithms for public policy decisions, policymakers need an accurate understanding of these algorithms' comparative performance. Correspondingly, data practitioners require standard metrics for evaluating the analytic qualities of the synthetic data. In this paper, we present an in-depth evaluation of several differentially private synthetic data algorithms using actual differentially private synthetic data sets created by contestants in the 2018-2019 National Institute of Standards and Technology Public Safety Communications Research (NIST PSCR) Division's ``Differential Privacy Synthetic Data Challenge.'' We offer analyses of these algorithms based on both the accuracy of the data they created and their usability by potential data providers. We frame the methods used in the NIST PSCR data challenge within the broader differentially private synthetic data literature. We implement additional utility metrics, including two of our own, on the differentially private synthetic data and compare mechanism utility on three categories. Our comparative assessment of the differentially private data synthesis methods and the quality metrics shows the relative usefulness, the general strengths and weaknesses, and offers preferred choices of algorithms and metrics. Finally we describe the implications of our evaluation for policymakers seeking to implement differentially private synthetic data algorithms on future data products.
翻译:为了利用这些算法来作出公共政策决定,决策者需要准确理解这些算法的比较性能。相应地,数据从业者需要标准指标来评价合成数据的分析性能。在本文中,我们利用2018-2019年国家标准和技术公共安全通信研究所(NIST PSCR)竞争者建立的实际有差别的私人合成合成数据集,对几种有差别的私人合成数据算法进行了深入评价。为了利用这些算法来保护数据中个人的隐私隐私。为了利用这些算法来作出公共政策决定,决策者们需要准确理解这些算法的比较性业绩。相应地,数据从业者需要标准指标来评价合成数据的分析性质量。我们在更广泛的有差别的私人合成数据文献中,我们采用了另外的效用指标,包括我们自己的两个指标,关于有差别的私人合成数据以及比较机制在三类方面的效用。我们对差异性私人数据综合方法的比较性评估,以及我们为寻求个人数据分析的相对价值,最后,我们用质量的衡量性分析方法,我们用在寻求个人数据分析中的相对价值,我们用到对分析性分析结果的比较性评估。