The motivation of our research is to develop a sound-to-image (S2I) translation system for enabling a human receiver to visually infer the occurrence of sound related events. We expect the computer to 'imagine' the scene from the captured sound, generating original images that picture the sound emitting source. Previous studies on similar topics opted for simplified approaches using data with low content diversity and/or strong supervision. Differently, we propose to perform unsupervised S2I translation using thousands of distinct and unknown scenes, with slightly pre-cleaned data, just enough to guarantee aural-visual semantic coherence. To that end, we employ conditional generative adversarial networks (GANs) with a deep densely connected generator. Besides, we implemented a moving-average adversarial loss to address GANs training instability. Though the specified S2I translation problem is quite challenging, we were able to generalize the translator model enough to obtain more than 14%, in average, of interpretable and semantically coherent images translated from unknown sounds. Additionally, we present a solution using informativity classifiers to perform quantitative evaluation of S2I translation.
翻译:我们研究的动机是开发一个声到图像(S2I)翻译系统,使人体接收器能够对声相关事件的发生进行视觉推断。 我们期望计算机能够从所捕听的音响中“想象”场景,生成原始图像,从而描绘出音源。 以往关于类似专题的研究选择了使用内容多样性低和/或监管强的数据的简化方法。 不同地,我们提议使用数千个不同和未知的场景进行不受监督的S2I翻译,并略微清理前的数据,足以保证视听语义的一致性。 为此,我们使用一个具有深度密集连接发电机的有条件的配对网。 此外,我们实施了移动平均对抗性网络,以解决GANs的不稳定性。尽管指定的S2I翻译问题相当棘手,但我们能够将翻译模型普遍化,以获得平均超过14%的、可翻译和语义一致的、由未知声音翻译的图像。 此外,我们提出一种使用信息化分类师对S2I翻译进行定量评估的解决方案。