CoLT5: 有条件计算加速长距离Transformer (CoLT5: Faster Long-Range Transformers with Conditional Computation)

Joshua Ainslie,Tao Lei,Michiel de Jong,Santiago Ontañón,Siddhartha Brahma,Yury Zemlyanskiy,David Uthus,Mandy Guo,James Lee-Thorp,Yi Tay,Yun-Hsuan Sung,Sumit Sanghai

Many natural language processing tasks benefit from long inputs, but processing long documents with Transformers is expensive -- not only due to quadratic attention complexity but also from applying feedforward and projection layers to every token. However, not all tokens are equally important, especially for longer documents. We propose CoLT5, a long-input Transformer model that builds on this intuition by employing conditional computation, devoting more resources to important tokens in both feedforward and attention layers. We show that CoLT5 achieves stronger performance than LongT5 with much faster training and inference, achieving SOTA on the long-input SCROLLS benchmark. Moreover, CoLT5 can effectively and tractably make use of extremely long inputs, showing strong gains up to 64k input length.

翻译：许多自然语言处理任务需要处理长文本，但是使用Transformer处理长文本的代价很高——不仅由于二次的自注意力的复杂度，还由于需要对每个标记应用前馈和投影层。然而，并不是所有标记都同样重要，特别是对于更长的文档。我们提出了CoLT5，一种长输入Transformer模型，利用条件计算这种直觉，在前馈和自注意力层中更多地使用重要标记。我们展示了CoLT5比LongT5表现更好，训练和推理速度更快，在长文本SCROLLS基准测试中取得了SOTA。此外，CoLT5可以有效且可缩放地使用极长的输入，展现出了在64k输入长度上的强大优势。