Although several research works have been reported on audio-visual sound source localization in unconstrained videos, no datasets and metrics have been proposed in the literature to quantitatively evaluate its performance. Defining the ground truth for sound source localization is difficult, because the location where the sound is produced is not limited to the range of the source object, but the vibrations propagate and spread through the surrounding objects. Therefore we propose a new concept, Sounding Object, to reduce the ambiguity of the visual location of sound, making it possible to annotate the location of the wide range of sound sources. With newly proposed metrics for quantitative evaluation, we formulate the problem of Audio-Visual Sounding Object Localization (AVSOL). We also created the evaluation dataset (AVSOL-E dataset) by manually annotating the test set of well-known Audio-Visual Event (AVE) dataset. To tackle this new AVSOL problem, we propose a novel multitask training strategy and architecture called Dual Normalization Multitasking (DNM), which aggregates the Audio-Visual Correspondence (AVC) task and the classification task for video events into a single audio-visual similarity map. By efficiently utilize both supervisions by DNM, our proposed architecture significantly outperforms the baseline methods.
翻译:尽管在不受限制的视频中报告了关于视听声源本地化的若干研究工作,但文献中没有提出数据集和衡量标准,以定量评价其性能。很难为声音源本地化确定地面真相,因为声音生成地点不仅限于源对象的范围,震动在周围物体中传播和传播。因此,我们提出了一个新概念,即“声音对象”,以减少声音视觉位置的模糊性,从而有可能说明各种声音源的位置。根据新的定量评价指标,我们制定了视听声学声音对象本地化问题(AVSOL)。我们还创建了评价数据集(AVSOL-E数据集),方法是手动说明众所周知的音频视频事件(AVE)数据集的测试集。为了解决这个新的AVSOL问题,我们提出了一个新的多任务培训战略和结构,称为“双正常化多任务”(DNM),将视听波斯调对象化(AVACC)任务汇总成,并大幅利用我们的拟议视听基准结构的视听任务,将类似任务和图像结构有效地利用我们的拟议的视听基准结构。