Fine-tuning pre-trained language models (PLMs) has demonstrated its effectiveness on various downstream NLP tasks recently. However, in many low-resource scenarios, the conventional fine-tuning strategies cannot sufficiently capture the important semantic features for downstream tasks. To address this issue, we introduce a novel framework (named "CSS-LM") to improve the fine-tuning phase of PLMs via contrastive semi-supervised learning. Specifically, given a specific task, we retrieve positive and negative instances from large-scale unlabeled corpora according to their domain-level and class-level semantic relatedness to the task. We then perform contrastive semi-supervised learning on both the retrieved unlabeled and original labeled instances to help PLMs capture crucial task-related semantic features. The experimental results show that CSS-LM achieves better results than the conventional fine-tuning strategy on a series of downstream tasks with few-shot settings, and outperforms the latest supervised contrastive fine-tuning strategies. Our datasets and source code will be available to provide more details.
翻译:最近,在各种低资源情景中,常规微调战略无法充分捕捉下游任务的重要语义特征。为了解决这一问题,我们引入了一个新颖的框架(名为“CSS-LM”),通过对比性半监督性学习来改进PLM的微调阶段。具体地说,根据一项具体任务,我们从大型无标签公司中根据它们与任务有关的域级和级级级语义相关关系检索到正面和负面的事例。然后,我们对检索到的无标签和原始标签的语义特征进行对比性的半监督性学习,以帮助PLM获取关键的任务语义特征。实验结果显示,CSS-LM比对一系列低视环境的下游任务常规微调战略取得更好的效果,并超越了最新的监管下的微调战略。我们的数据集和源代码可以提供更多细节。