Image paragraph generation is the task of producing a coherent story (usually a paragraph) that describes the visual content of an image. The problem nevertheless is not trivial especially when there are multiple descriptive and diverse gists to be considered for paragraph generation, which often happens in real images. A valid question is how to encapsulate such gists/topics that are worthy of mention from an image, and then describe the image from one topic to another but holistically with a coherent structure. In this paper, we present a new design --- Convolutional Auto-Encoding (CAE) that purely employs convolutional and deconvolutional auto-encoding framework for topic modeling on the region-level features of an image. Furthermore, we propose an architecture, namely CAE plus Long Short-Term Memory (dubbed as CAE-LSTM), that novelly integrates the learnt topics in support of paragraph generation. Technically, CAE-LSTM capitalizes on a two-level LSTM-based paragraph generation framework with attention mechanism. The paragraph-level LSTM captures the inter-sentence dependency in a paragraph, while sentence-level LSTM is to generate one sentence which is conditioned on each learnt topic. Extensive experiments are conducted on Stanford image paragraph dataset, and superior results are reported when comparing to state-of-the-art approaches. More remarkably, CAE-LSTM increases CIDEr performance from 20.93% to 25.15%.
翻译:图像段落生成的任务是制作一个一致的故事(通常是一段),描述图像的视觉内容。然而,问题并不是微不足道的,特别是在为生成段落时需要考虑多个描述性和多样性的多面性格,这往往发生在真实的图像中。一个有效的问题是如何从图像中包含值得一提的、从一个主题到另一个主题,然后用一个连贯的结构整体地描述图像。在本文件中,我们提出了一个新的设计 -- -- 革命自动编码(CAE),它纯粹使用在图像的区域级特征上进行主题建模的革命性和非革命性自动编码框架。此外,我们提出了一个结构,即CAE+长期内存(作为CAE-LSTM的缩放式),它以新颖的方式将所学过的专题纳入支持生成的段落。技术上,CAE-LTM利用基于LSTM的双层段落生成机制。LSTM在段落中记录了相互依赖性关系,而DS-93级LS-CA-CA-C-CA-CL-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-CS-CS-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-S-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-I-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-