We introduce a novel federated learning framework, FedD3, which reduces the overall communication volume and with that opens up the concept of federated learning to more application scenarios in network-constrained environments. It achieves this by leveraging local dataset distillation instead of traditional learning approaches (i) to significantly reduce communication volumes and (ii) to limit transfers to one-shot communication, rather than iterative multiway communication. Instead of sharing model updates, as in other federated learning approaches, FedD3 allows the connected clients to distill the local datasets independently, and then aggregates those decentralized distilled datasets (typically in the form a few unrecognizable images, which are normally smaller than a model) across the network only once to form the final model. Our experimental results show that FedD3 significantly outperforms other federated learning frameworks in terms of needed communication volumes, while it provides the additional benefit to be able to balance the trade-off between accuracy and communication cost, depending on usage scenario or target dataset. For instance, for training an AlexNet model on a Non-IID CIFAR-10 dataset with 10 clients, FedD3 can either increase the accuracy by over 71% with a similar communication volume, or save 98% of communication volume, while reaching the same accuracy, comparing to other one-shot federated learning approaches.
翻译:我们引入了一个新的联合学习框架FedD3, 减少了整体通信量, 从而打开了在网络受限制的环境中对更多应用情景进行联合学习的概念; 通过利用本地数据集蒸馏而不是传统学习方法(一) 大幅降低通信量, (二) 限制向一次性通信的传输, 而不是迭代多路通信的传输; FedD3 允许连接客户与其他联邦学习方法一样,共享模式更新,而不是与其他联邦学习方法一样,让连接客户独立地蒸馏本地数据集,然后将分散式的蒸馏数据集(通常形式为少量无法辨认的图像,通常比模型小一次)汇总到网络上,形成最后模式。 我们的实验结果表明,FedD3在所需通信量方面大大优于其他联合式学习框架,而它提供了额外的好处,能够平衡精确度与通信成本之间的交易,这取决于使用情景或目标数据集。 例如,可以将非IID CIFAR- 10 10 数据模型(通常比模型小于模型) 。 我们的实验结果表明, FedD3 能够将非IID- DFD- 10 10 格式的通信方法与其他精确度进行比较, 以10 10 以10 的精确度相比, 10 将其他通信量 将其他通信方法比为10 。