Deep neural networks are typically trained under a supervised learning framework where a model learns a single task using labeled data. Instead of relying solely on labeled data, practitioners can harness unlabeled or related data to improve model performance, which is often more accessible and ubiquitous. Self-supervised pre-training for transfer learning is becoming an increasingly popular technique to improve state-of-the-art results using unlabeled data. It involves first pre-training a model on a large amount of unlabeled data, then adapting the model to target tasks of interest. In this review, we survey self-supervised learning methods and their applications within the sequential transfer learning framework. We provide an overview of the taxonomy for self-supervised learning and transfer learning, and highlight some prominent methods for designing pre-training tasks across different domains. Finally, we discuss recent trends and suggest areas for future investigation.
翻译:深神经网络通常在一个有监督的学习框架内接受培训,模型利用标签数据学习单一任务。实践者可以使用没有标签或相关的数据来改进模型性能,而这种性能往往更容易获得,而且无处不在。自我监督的转让学习前培训正在成为一种日益流行的技术,目的是利用无标签数据改进最新成果。它涉及对大量无标签数据进行初步培训,然后将模型适应目标任务。在这次审查中,我们调查了自监督的学习方法及其在连续转移学习框架内的应用。我们概述了用于自我监督学习和转移学习的分类学,并突出强调了设计不同领域培训前任务的一些突出方法。最后,我们讨论了最近的趋势,并提出了未来调查的领域。