Incremental learning is one paradigm to enable model building and updating at scale with streaming data. For end-to-end automatic speech recognition (ASR) tasks, the absence of human annotated labels along with the need for privacy preserving policies for model building makes it a daunting challenge. Motivated by these challenges, in this paper we use a cloud based framework for production systems to demonstrate insights from privacy preserving incremental learning for automatic speech recognition (ILASR). By privacy preserving, we mean, usage of ephemeral data which are not human annotated. This system is a step forward for production levelASR models for incremental/continual learning that offers near real-time test-bed for experimentation in the cloud for end-to-end ASR, while adhering to privacy-preserving policies. We show that the proposed system can improve the production models significantly(3%) over a new time period of six months even in the absence of human annotated labels with varying levels of weak supervision and large batch sizes in incremental learning. This improvement is 20% over test sets with new words and phrases in the new time period. We demonstrate the effectiveness of model building in a privacy-preserving incremental fashion for ASR while further exploring the utility of having an effective teacher model and use of large batch sizes.
翻译:递增学习是一种范例,可以使建模和升级规模与流数据相适应。对于终端到终端自动语音识别(ASR)任务而言,缺少人文附加说明的标签,加上需要为建模提供隐私保护政策,这带来了巨大的挑战。受这些挑战的驱使,在本文件中,我们使用基于云的生产系统框架,以展示从隐私保护递增学习到自动语音识别(ILASR)的洞察力。通过隐私保护,我们是指使用非人为附加说明的短时间数据。这个系统是制作ASR增量/连续学习模型的向前迈出的一步,该模型为终端到终端自动语音识别(ASR)的云层实验提供了近实时测试台,同时坚持隐私保护政策。我们表明,拟议的系统可以在6个月的新时期内大大改进生产模式(3%),即使没有具有不同程度的监管和渐进规模的人类附加说明标签。在新的时间段里,这一改进了20 %的测试组,并带有新的词和短语。我们展示了建模的实效,同时进一步探索了空间建设模型的效用,同时进一步探索了大规模使用。