Phased data augmentation for training PixelCNNs with VQ-VAE-2 and limited data

With development of deep learning, researchers have developed generative models in generating realistic images. One of such generative models, a PixelCNNs model with Vector Quantized Variational AutoEncoder 2 (VQ-VAE-2), can generate more various images than other models. However, a PixelCNNs model with VQ-VAE-2, I call it PC-VQ2, requires sufficiently much training data like other deep learning models. Its practical applications are often limited in domains where collecting sufficient data is not difficult. To solve the problem, researchers have recently proposed more data-efficient methods for training generative models with limited unlabeled data from scratch. However, no such methods in PC-VQ2s have been researched. This study provides the first step in this direction, considering generation of images using PC-VQ2s and limited unlabeled data. In this study, I propose a training strategy for training a PC-VQ2 with limited data from scratch, phased data augmentation. In the strategy, ranges of parameters of data augmentation is narrowed in phases through learning. Quantitative evaluation shows that the phased data augmentation enables the model with limited data to generate images competitive with the one with sufficient data in diversity and outperforming it in fidelity. The evaluation suggests that the proposed method should be useful for training a PC-VQ2 with limited data efficiently to generate various and natural images.