Computer vision systems require large amounts of manually annotated data to properly learn challenging visual concepts. Crowdsourcing platforms offer an inexpensive method to capture human knowledge and understanding, for a vast number of visual perception tasks. In this survey, we describe the types of annotations computer vision researchers have collected using crowdsourcing, and how they have ensured that this data is of high quality while annotation effort is minimized. We begin by discussing data collection on both classic (e.g., object recognition) and recent (e.g., visual story-telling) vision tasks. We then summarize key design decisions for creating effective data collection interfaces and workflows, and present strategies for intelligently selecting the most important data instances to annotate. Finally, we conclude with some thoughts on the future of crowdsourcing in computer vision.
翻译:计算机视觉系统需要大量人工附加说明的数据,以便正确学习具有挑战性的视觉概念。 众包平台为获取人类知识和理解提供了一种廉价的方法,用于大量视觉认知任务。 在本次调查中,我们描述了计算机视觉研究人员利用众包收集的批注信息的类型,以及他们如何确保这些数据质量高,同时尽量减少批注努力。我们首先讨论关于经典(如物体识别)和近期(如视觉故事描述)视觉任务的数据采集工作。然后我们总结创建有效数据收集界面和工作流程的关键设计决定,并提出明智地选择最重要的数据案例进行批注的战略。 最后,我们最后就计算机视觉众包的未来提出了一些想法。