Wav2vec 2.0 (W2V2) has shown impressive performance in automatic speech recognition (ASR). However, the large model size and the non-streaming architecture make it hard to be used under low-resource or streaming scenarios. In this work, we propose a two-stage knowledge distillation method to solve these two problems: the first step is to make the big and non-streaming teacher model smaller, and the second step is to make it streaming. Specially, we adopt the MSE loss for the distillation of hidden layers and the modified LF-MMI loss for the distillation of the prediction layer. Experiments are conducted on Gigaspeech, Librispeech, and an in-house dataset. The results show that the distilled student model (DistillW2V2) we finally get is 8x faster and 12x smaller than the original teacher model. For the 480ms latency setup, the DistillW2V2's relative word error rate (WER) degradation varies from 9% to 23.4% on test sets, which reveals a promising way to extend the W2V2's application scope.
翻译:Wav2vec 2. 0 (W2V2) 在自动语音识别( ASR) 中表现出了令人印象深刻的性能。 然而,由于模型大小大且非流式结构结构不流化,因此在资源或流式情景下很难使用。 在这项工作中,我们提出了一个两阶段的知识蒸馏方法来解决这两个问题:第一步是缩小大型和非流式教师模式,第二步是使其流化。特别是,我们采用MSE损失来蒸馏隐藏层和修改后的LF-MMI损失来蒸馏预测层。在Gigaspeech、Librispeech和内部数据集中进行了实验。结果显示,蒸馏式学生模式(DistillW2V2)最终比最初的教师模型更快8x和12x小。对于480ms Latency 设置, 蒸馏W2V2的相对单词错误率在测试机组中从9%到23.4%不等,这显示了扩大W2V2应用范围的可行方法。</s>