Deep latent variable models (LVM) such as variational auto-encoder (VAE) have recently played an important role in text generation. One key factor is the exploitation of smooth latent structures to guide the generation. However, the representation power of VAEs is limited due to two reasons: (1) the Gaussian assumption is often made on the variational posteriors; and meanwhile (2) a notorious "posterior collapse" issue occurs. In this paper, we advocate sample-based representations of variational distributions for natural language, leading to implicit latent features, which can provide flexible representation power compared with Gaussian-based posteriors. We further develop an LVM to directly match the aggregated posterior to the prior. It can be viewed as a natural extension of VAEs with a regularization of maximizing mutual information, mitigating the "posterior collapse" issue. We demonstrate the effectiveness and versatility of our models in various text generation scenarios, including language modeling, unaligned style transfer, and dialog response generation. The source code to reproduce our experimental results is available on GitHub.
翻译:深潜变量模型(LVM),如变式自动读数器(VAE),最近在文本生成中发挥了重要作用。一个关键因素是利用光滑的潜在结构来引导生成。然而,VAE的代表性力量有限,原因有二:(1) 高斯的假设往往是在变式后子星上作出的;(2) 同时,出现了臭名昭著的“前置崩溃”问题。在本文中,我们主张以样本为基础对自然语言的变异分布进行表述,导致隐含的潜在特征,这可以提供与基于高斯的后代星体相比灵活的表达力。我们进一步开发了LVM,以直接将汇总的后代体与前代相匹配。它可以被视为VAE的自然延伸,同时调整相互信息,减轻“未来崩溃”问题。我们展示了我们模型在各种文本生成情景中的有效性和多功能性,包括语言建模、不协调的风格传输和对话响应生成。我们复制实验结果的来源代码可以在GitHub上找到。