This paper introduces CODA-19, a human-annotated dataset that codes the Background, Purpose, Method, Finding/Contribution, and Other sections of 10,966 English abstracts in the COVID-19 Open Research Dataset. CODA-19 was created by 248 crowd workers from Amazon Mechanical Turk within 10 days, and achieved labeling quality comparable to that of experts. Each abstract was annotated by nine different workers, and the final labels were acquired by majority vote. The inter-annotator agreement (Cohen's kappa) between the crowd and the biomedical expert (0.741) is comparable to inter-expert agreement (0.788). CODA-19's labels have an accuracy of 82.2% when compared to the biomedical expert's labels, while the accuracy between experts was 85.0%. Reliable human annotations help scientists access and integrate the rapidly accelerating coronavirus literature, and also serve as the battery of AI/NLP research, but obtaining expert annotations can be slow. We demonstrated that a non-expert crowd can be rapidly employed at scale to join the fight against COVID-19.
翻译:本文介绍CODA-19,这是一套人类附加说明的数据集,在COVID-19开放研究数据集中将背景、目的、方法、查找/贡献和其他部分的10,966份英文摘要编码为COVID-19开放研究数据集,由来自亚马逊机械土耳其岛的248名群众工人在10天内创建的CODA-19,达到了与专家质量相当的标签;每个摘要都有9名不同工人的注解,最后的标签以多数票获得;人群与生物医学专家之间的顾问间协议(Cohen's kappa)与专家间协议(0.788)相似;CODA-19的标签与生物医学专家标签相比,准确率为82.2%;专家之间的准确度为85.0%;可靠的人类说明有助于科学家获得并综合迅速加速的 Corona病毒文献,同时也作为AI/NLP研究的电池,但获得专家说明的过程可能缓慢。