Humans can abstract prior knowledge from very little data and use it to boost skill learning. In this paper, we propose routine-augmented policy learning (RAPL), which discovers routines composed of primitive actions from a single demonstration and uses discovered routines to augment policy learning. To discover routines from the demonstration, we first abstract routine candidates by identifying grammar over the demonstrated action trajectory. Then, the best routines measured by length and frequency are selected to form a routine library. We propose to learn policy simultaneously at primitive-level and routine-level with discovered routines, leveraging the temporal structure of routines. Our approach enables imitating expert behavior at multiple temporal scales for imitation learning and promotes reinforcement learning exploration. Extensive experiments on Atari games demonstrate that RAPL improves the state-of-the-art imitation learning method SQIL and reinforcement learning method A2C. Further, we show that discovered routines can generalize to unseen levels and difficulties on the CoinRun benchmark.
翻译:人类可以从非常少的数据中总结先前的知识,并将其用于促进技能学习。 在本文中,我们建议进行常规强化的政策学习(RAPL),通过单一演示发现由原始行动构成的例行活动,并利用发现的例行活动加强政策学习。为了从演示中发现例行活动,我们首先通过辨别示范活动轨迹的语法来抽象的例行活动候选人。然后,选择以长度和频率衡量的最佳例行活动来形成一个例行图书馆。我们建议同时在原始一级和日常一级学习政策,同时发现日常活动,利用日常活动的时间结构。我们的方法使得能够在多个时间尺度上模仿专家行为,以进行模仿学习,并促进强化学习探索。对Atari游戏的广泛实验表明,RAPL改进了最先进的模拟学习方法SQIL和强化学习方法A2C。此外,我们证明发现的日常活动可以概括到看不见的水平和CoinRun基准上的困难。