近年来,预训练模型(例如ELMo、GPT、BERT和XLNet等)的快速发展大幅提升了诸多NLP任务的整体水平,同时也使得很多应用场景进入到实际落地阶段。预训练语言模型本身就是神经网络语言模型,它的特点包括:第一,可以使用大规模无标注纯文本语料进行训练;第二,可以用于各类下游NLP任务,不是针对某项定制的,但以后可用在下游NIP任务上,你不需要为下游任务专门设计一种神经网络,或者提供一种结构,直接在几种给定的固定框架中选择一种进行 fine-tune,就可以从而得到很好的结果。

预训练语言模型 Pre-trained Language Model专知荟萃

综述

  1. 自然语言处理中的表示学习进展:从Transfomer到BERT 复旦大学邱锡鹏

  2. NLP深度学习的各类模型综述

  3. 预训练语言模型综述

  4. nlp语言模型和预训练综述

进阶论文

模型

知识蒸馏和模型压缩

分析

入门学习

  1. 自然语言处理中的语言模型预训练方法(ELMo、GPT和BERT)

  2. 深入理解语言模型 Language Model

  3. NLP中的语言模型(language model)

  4. 理解语言的 Transformer 模型

代码

  1. Transformer-Attention Is All You Need

  2. BERT-Pre-training of Deep Bidirectional Transformers for Language Understanding

  3. GPT2-Language Models are Unsupervised Multitask Learners

  4. ERNIE-Enhanced Language Representation with Informative Entities

  5. XLM-Cross-lingual Language Model Pretraining

  6. MASS-Masked Sequence to Sequence Pre-training for Language Generation

  7. XLNet-Generalized Autoregressive Pretraining for Language Understanding

  8. LAMA-Language Models as Knowledge Bases?

  9. Investigating BERT's Knowledge of Language: Five Analysis Methods with NPIs

  10. LXMERT-Learning Cross-Modality Encoder Representations from Transformers

  11. XLNet-Generalized Autoregressive Pretraining for Language Understanding

  12. MT-DNN-Multi-Task Deep Neural Networks for Natural Language Understanding

领域专家

  1. 清华大学
  2. 哈尔滨工业大学
  3. 微软亚洲研究院自然语言计算组:
  4. 华为诺亚方舟实验室
    • 刘群
  5. 百度

Tutorial

  1. Latent Structure Models for Natural Language Processing
  2. Graph-Based Meaning Representations: Design and Processing
  3. Discourse Analysis and Its Applications
  4. Deep Learning for Natural Language Processing: Theory and Practice [Tutorial]
  5. Recurrent Neural Networks with Word Embeddings
  6. LSTM Networks for Sentiment Analysis
  7. Semantic Representations of Word Senses and Concepts 语义表示 ACL 2016 Tutorial by José Camacho-Collados, Ignacio Iacobacci, Roberto Navigli and Mohammad Taher Pilehvar
  8. ACL 2016 Tutorial: Understanding Short Texts 短文本理解
  9. Practical Neural Networks for NLP  EMNLP 2016
  10. Structured Neural Networks for NLP: From Idea to Code
  11. Understanding Deep Learning Models in NLP
  12. Deep learning for natural language processing, Part 1
  13. TensorFlow Tutorial on Seq2Seq Models
  14. Natural Language Understanding with Distributed Representation Lecture Note by Cho
  15. Michael Collins
  16. Several tutorials by Radim Řehůřek
  17. Natural Language Processing in Action
  18. Semantic Specialization of Distributional Word Vectors
  19. Dive into Deep Learning for Natural Language Processing
  20. Transfer Learning in Natural Language Processing. Sebastian Ruder, Matthew E. Peters, Swabha Swayamdipta, Thomas Wolf. NAACL 2019.
  21. Transformers: State-of-the-art Natural Language Processing. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Jamie Brew. Preprint.
  22. 【2019 北京智源大会】预训练语言模型的研究与应用 刘群/华为诺亚方舟实验室
成为VIP会员查看完整内容
Top