Language models are generally trained on short, truncated input sequences, which limits their ability to use discourse-level information present in long-range context to improve their predictions. Recent efforts to improve the efficiency of self-attention have led to a proliferation of long-range Transformer language models, which can process much longer sequences than models of the past. However, the ways in which such models take advantage of the long-range context remain unclear. In this paper, we perform a fine-grained analysis of two long-range Transformer language models (including the \emph{Routing Transformer}, which achieves state-of-the-art perplexity on the PG-19 long-sequence LM benchmark dataset) that accept input sequences of up to 8K tokens. Our results reveal that providing long-range context (i.e., beyond the previous 2K tokens) to these models only improves their predictions on a small set of tokens (e.g., those that can be copied from the distant context) and does not help at all for sentence-level prediction tasks. Finally, we discover that PG-19 contains a variety of different document types and domains, and that long-range context helps most for literary novels (as opposed to textbooks or magazines).
翻译:语言模型通常在短短的输入序列上接受培训,这限制了他们使用长距离范围内的谈话级信息的能力,从而改进预测。最近提高自我关注效率的努力导致长程变换语言模型的扩散,这些模型可以处理比过去模型长得多的顺序。然而,这些模型利用长程背景的方式仍然不明确。在本文件中,我们对两种长程变换语言模型(包括\emph{Routing tranger})进行精细分析,这些模型在PG-19长序列LM基准数据集上达到最先进的迷惑状态,接受最高达8K标记的输入序列。我们的结果显示,这些模型提供长程背景(即超过以前的2K标志)的方式只能改进对一小套符号的预测(例如,可以从远处复制的变换换),并且不会帮助所有层次预测任务。最后,我们发现P-19最远的版本和最新一代的版本的版本的版本(我们发现P-19和最新一代的版本的版本的版本的版本)和最新一代的版本的版本。最后,我们发现P-19的版本的版本的版本和新版本的版本的版本的版本的文件。