In this paper we introduce ViSiL, a Video Similarity Learning architecture that considers fine-grained Spatio-Temporal relations between pairs of videos -- such relations are typically lost in previous video retrieval approaches that embed the whole frame or even the whole video into a vector descriptor before the similarity estimation. By contrast, our Convolutional Neural Network (CNN)-based approach is trained to calculate video-to-video similarity from refined frame-to-frame similarity matrices, so as to consider both intra- and inter-frame relations. In the proposed method, pairwise frame similarity is estimated by applying Tensor Dot (TD) followed by Chamfer Similarity (CS) on regional CNN frame features - this avoids feature aggregation before the similarity calculation between frames. Subsequently, the similarity matrix between all video frames is fed to a four-layer CNN, and then summarized using Chamfer Similarity (CS) into a video-to-video similarity score -- this avoids feature aggregation before the similarity calculation between videos and captures the temporal similarity patterns between matching frame sequences. We train the proposed network using a triplet loss scheme and evaluate it on five public benchmark datasets on four different video retrieval problems where we demonstrate large improvements in comparison to the state of the art. The implementation of ViSiL is publicly available.
翻译:在本文中,我们引入了ViSiL这个视频相似性学习架构,它考虑到一对视频之间细微的Spatio-Temporal关系 -- -- 这种关系通常在先前的视频检索方法中消失,在类似估计之前将整个框架或甚至整个视频嵌入矢量描述器。相比之下,我们基于共进神经网络(CNN)的方法经过培训,从完善的框架到框架的相似性矩阵中计算视频与视频的相似性,以便考虑框架内和框架间的关系。在拟议方法中,通过在区域CNN框架特征中应用Tensor Dot(TD)和Chamfer相似性(CS)来估计对相近性框架的相似性(TD)来估计。这避免了在对框架进行相似性计算之前将整个框架甚至整个视频组合成矢量描述。随后,所有视频框架之间的类似性矩阵都被反馈到一个四层CNN,然后将Chamfer 类似性(CS) 用于视频至视频相似性的评分 -- -- 避免在视频之间进行类似性计算之前进行特征汇总,并捕捉取匹配框架序列之间的时间相似性模式。我们用提议的网络在公开检索中用现有五级排序中用三维度测量方法来演示。我们所拟议的三维度数据检索方案。我们用不同的三维基点测试了网络上展示了现有数据。