Recurrent neural network grammars (RNNG) are generative models of language which jointly model syntax and surface structure by incrementally generating a syntax tree and sentence in a top-down, left-to-right order. Supervised RNNGs achieve strong language modeling and parsing performance, but require an annotated corpus of parse trees. In this work, we experiment with unsupervised learning of RNNGs. Since directly marginalizing over the space of latent trees is intractable, we instead apply amortized variational inference. To maximize the evidence lower bound, we develop an inference network parameterized as a neural CRF constituency parser. On language modeling, unsupervised RNNGs perform as well their supervised counterparts on benchmarks in English and Chinese. On constituency grammar induction, they are competitive with recent neural language models that induce tree structures from words through attention mechanisms.
翻译:经常性神经网络语法(RNNG)是语言的基因模型,这些语言通过递增生成自上而下、左对右顺序的语法树和句子,共同模拟语法和表面结构。受监督的RNNG实现了强大的语言建模和分解功能,但需要一组附加注释的剖析树。在这项工作中,我们试验未经监督的对RNG的学习。由于在潜伏树的空间上直接边缘化是难以控制的,我们反而采用摊分变法。为了尽量扩大证据的下限,我们开发了一个推断网络,作为神经通用报告格式选区分析器参数。在语言建模方面,未受监督的RNNGs在英语和中文基准上的表现以及监督的对应方。在选区语法感上,他们与最近的神经语言模型竞争激烈,这些模型通过注意力机制引导树木结构从文字到注意力机制。