Speaker diarization is one of the critical components of computational media intelligence as it enables a character-level analysis of story portrayals and media content understanding. Automated audio-based speaker diarization of entertainment media poses challenges due to the diverse acoustic conditions present in media content, be it background music, overlapping speakers, or sound effects. At the same time, speaking faces in the visual modality provide complementary information and not prone to the errors seen in the audio modality. In this paper, we address the problem of speaker diarization in TV shows using the active speaker faces. We perform face clustering on the active speaker faces and show superior speaker diarization performance compared to the state-of-the-art audio-based diarization methods. We additionally report a systematic analysis of the impact of active speaker face detection quality on the diarization performance. We also observe that a moderately well-performing active speaker system could outperform the audio-based diarization systems.
翻译:发言人的diarization是计算媒体情报的关键组成部分之一,因为它有助于对故事描述和媒体内容的理解进行品位分析。娱乐媒体的自动音频扬声器二分化由于媒体内容中存在的声学条件不同而带来了挑战,无论是背景音乐、重复的扬声器,还是声音效应。与此同时,视觉模式中的表情提供了补充信息,而不会发生音频模式中出现的错误。在本文中,我们用积极的扬声器面孔处理在电视节目中扬声器二分化的问题。我们用活跃的扬声器面部进行面部组合,展示与最先进的音频基diarization方法相比的高级扬声器分化性表现。我们还报告对积极扬声器面检测质量对diarization性表现的影响进行系统系统分析。我们还观察到,中度良好活跃的扬声器系统可能超过音频二分化系统。