Cross-modal video-text retrieval, a challenging task in the field of vision and language, aims at retrieving corresponding instance giving sample from either modality. Existing approaches for this task all focus on how to design encoding model through a hard negative ranking loss, leaving two key problems unaddressed during this procedure. First, in the training stage, only a mini-batch of instance pairs is available in each iteration. Therefore, this kind of hard negatives is locally mined inside a mini-batch while ignoring the global negative samples among the dataset. Second, there are many text descriptions for one video and each text only describes certain local features of a video. Previous works for this task did not consider to fuse the multiply texts corresponding to a video during the training. In this paper, to solve the above two problems, we propose a novel memory enhanced embedding learning (MEEL) method for videotext retrieval. To be specific, we construct two kinds of memory banks respectively: cross-modal memory module and text center memory module. The cross-modal memory module is employed to record the instance embeddings of all the datasets for global negative mining. To avoid the fast evolving of the embedding in the memory bank during training, we utilize a momentum encoder to update the features by a moving-averaging strategy. The text center memory module is designed to record the center information of the multiple textual instances corresponding to a video, and aims at bridging these textual instances together. Extensive experimental results on two challenging benchmarks, i.e., MSR-VTT and VATEX, demonstrate the effectiveness of the proposed method.
翻译:跨模版视频文本检索是视觉和语言领域一项具有挑战性的任务,目的是从任一模式的样本中获取相应的对应实例。 用于此任务的现有方法都侧重于如何设计编码模式, 从而通过硬负排位损失来设计编码模式, 从而在此过程中解决两个关键问题。 首先, 在培训阶段, 在每个迭代中, 仅有一小批小型实例配对可使用。 因此, 这种硬负数是本地在小型批中存储, 而忽略数据集中的全球负样本 。 其次, 一个视频有许多文本描述, 每个文本只描述一个视频的某些本地特征 。 之前的任务有效性工作没有考虑在培训期间将与视频对应的多个文本组合起来。 在本文中, 为了解决上述两个问题, 我们提出一个新的记忆强化的嵌入式学习( MEEL) 方法用于视频文本检索。 具体地说, 我们建造两种记忆库: 跨模版存储模块和文本中心存储模块 。 跨模版存储模块用于记录所有对应数据设置的视频地方特征, 视频中两个对应的文本嵌入过程 。 正在快速地将ialder 存储一个用于全球内存中心 。