Recently, a new recurrent neural network (RNN) named the Legendre Memory Unit (LMU) was proposed and shown to achieve state-of-the-art performance on several benchmark datasets. Here we leverage the linear time-invariant (LTI) memory component of the LMU to construct a simplified variant that can be parallelized during training (and yet executed as an RNN during inference), thus overcoming a well known limitation of training RNNs on GPUs. We show that this reformulation that aids parallelizing, which can be applied generally to any deep network whose recurrent components are linear, makes training up to 200 times faster. Second, to validate its utility, we compare its performance against the original LMU and a variety of published LSTM and transformer networks on seven benchmarks, ranging from psMNIST to sentiment analysis to machine translation. We demonstrate that our models exhibit superior performance on all datasets, often using fewer parameters. For instance, our LMU sets a new state-of-the-art result on psMNIST, and uses half the parameters while outperforming DistilBERT and LSTM models on IMDB sentiment analysis.
翻译:最近,一个新的经常性神经网络(RNN)(RNN)被提出并展示为在若干基准数据集上达到最新水平的性能。在这里,我们利用LMU的线性时差内存组件(LTI)构建一个简化的变体,在培训期间可以平行(但在推论期间作为RNN),从而克服了在GPU上培训RNN的众所周知的限制。我们表明,这一重塑辅助可普遍适用于任何经常组件为线性的深层网络,使培训速度加快200倍。第二,为了验证其效用,我们将其性能与原始LMU和各种已出版的LSTM和变异网络的7个基准进行比较,从PsMNIST到情绪分析到机器翻译。我们证明我们的模型在所有数据集上表现优异性,往往使用较少的参数。例如,我们的LMU在PMIST上制定了一个新的状态-艺术结果,并使用一半参数,同时在IMDIMS分析上比DTIER和LSTM模型。