Exploiting the temporal dependency among video frames or subshots is very important for the task of video summarization. Practically, RNN is good at temporal dependency modeling, and has achieved overwhelming performance in many video-based tasks, such as video captioning and classification. However, RNN is not capable enough to handle the video summarization task, since traditional RNNs, including LSTM, can only deal with short videos, while the videos in the summarization task are usually in longer duration. To address this problem, we propose a hierarchical recurrent neural network for video summarization, called H-RNN in this paper. Specifically, it has two layers, where the first layer is utilized to encode short video subshots cut from the original video, and the final hidden state of each subshot is input to the second layer for calculating its confidence to be a key subshot. Compared to traditional RNNs, H-RNN is more suitable to video summarization, since it can exploit long temporal dependency among frames, meanwhile, the computation operations are significantly lessened. The results on two popular datasets, including the Combined dataset and VTW dataset, have demonstrated that the proposed H-RNN outperforms the state-of-the-arts.
翻译:挖掘视频框架或子集集之间的时间依赖性对于视频总结任务非常重要。 实际上, RNNN 擅长时间依赖模型,并且在许多视频任务(如视频字幕和分类)中取得了压倒性性性表现。 然而, RNN 并不足以处理视频总结任务,因为传统的 RNN 包括 LSTM 在内的传统 RNN 只能处理短视频,而概括任务中的视频通常是较长的时间段。为了解决这一问题,我们建议建立一个等级级的经常性神经网络,用于视频总结。 本文中称为 H- RNN 。 具体地说,它有两个层, 使用第一层来编码从原始视频中剪掉的短视频子集, 而每个子集的最后隐藏状态是输入到第二层, 以计算其作为关键子集的信心。 与传统的 RNNP 相比, H- RNNN 更适合视频总结任务, 因为它可以利用各框架之间的长期时间依赖性, 同时, 计算操作也大大减弱。 在两个流行的数据集上, 包括综合的HNW 和 VT 显示的综合数据集 。