We share the findings of the first shared task on improving robustness of Machine Translation (MT). The task provides a testbed representing challenges facing MT models deployed in the real world, and facilitates new approaches to improve models; robustness to noisy input and domain mismatch. We focus on two language pairs (English-French and English-Japanese), and the submitted systems are evaluated on a blind test set consisting of noisy comments on Reddit and professionally sourced translations. As a new task, we received 23 submissions by 11 participating teams from universities, companies, national labs, etc. All submitted systems achieved large improvements over baselines, with the best improvement having +22.33 BLEU. We evaluated submissions by both human judgment and automatic evaluation (BLEU), which shows high correlations (Pearson's r = 0.94 and 0.95). Furthermore, we conducted a qualitative analysis of the submitted systems using compare-mt, which revealed their salient differences in handling challenges in this task. Such analysis provides additional insights when there is occasional disagreement between human judgment and BLEU, e.g. systems better at producing colloquial expressions received higher score from human judgment.
翻译:我们分享关于提高机器翻译可靠性的第一个共同任务(MT)的结果。任务提供了代表现实世界中部署的MT模型所面临的挑战的测试台,并促进了改进模型的新办法;对噪音输入和域错配的稳健性;我们侧重于两种语文对(英语-法语和英语-日语),对提交的系统进行了由关于Reddit和专业翻译的吵杂评论组成的盲式测试集的评估;作为一项新任务,我们收到了来自大学、公司、国家实验室等11个参与小组的23份材料。所有提交系统都取得了基线方面的重大改进,改进得最佳的是+22.33 BLEU。我们评估了由人类判断和自动评价(BLU)提交的材料,这表明了高度的相关性(Pearson's r= 0.94和0.95)。此外,我们利用对比模型对提交的系统进行了质量分析,其中揭示了它们在处理这项任务挑战方面的显著差异。当人类判断和BLEU(例如更好地制作从人类判断中获得更高分数的学术表达的系统)之间偶尔出现分歧时,这种分析提供了更多见解。