We present EdiT5 - a novel semi-autoregressive text-editing model designed to combine the strengths of non-autoregressive text-editing and autoregressive decoding. EdiT5 is faster during inference than conventional sequence-to-sequence (seq2seq) models, while being capable of modelling flexible input-output transformations. This is achieved by decomposing the generation process into three sub-tasks: (1) tagging to decide on the subset of input tokens to be preserved in the output, (2) re-ordering to define their order in the output text, and (3) insertion to infill the missing tokens that are not present in the input. The tagging and re-ordering steps, which are responsible for generating the largest portion of the output, are non-autoregressive, while the insertion step uses an autoregressive decoder. Depending on the task, EdiT5 on average requires significantly fewer autoregressive steps, demonstrating speedups of up to 25x when compared to seq2seq models. Quality-wise, EdiT5 is initialized with a pre-trained T5 checkpoint yielding comparable performance to T5 in high-resource settings when evaluated on three NLG tasks: Sentence Fusion, Grammatical Error Correction, and Decontextualization while clearly outperforming T5 in low-resource settings.
翻译:我们提出了EdiT5 - 一种新型的半递增文本编辑模型,目的是将非递减文本编辑和自动递减解码的优点结合起来。 EdiT5 在推断过程中比常规序列到序列模型(seq2seq) 更快,同时能够模拟灵活的输入输出转换。这是通过将生成过程分解成三个子任务来实现的。 (1) 标记以决定产出中要保存的输入符号子组, (2) 重新排序以定义输出文本中的顺序, (3) 插入以填充输入中未显示的缺失符号。 标记和重新排序步骤对生成产出的最大部分负责的是非递增序列(seq2saqeq) 模型, 而插入步骤则使用自递增递增导变变曲线。 根据任务, EdiT5 平均需要大幅减少自动递增级步骤, 显示与后置2eqequal 模型相比, 显示到后置的25xxx的加速度, 在TR5 级前的TRI 中, 向前的递增前的TRI 3 级任务, 递增前的递增性任务在前的TRI5