Synthesizing novel views from a single view image is a highly ill-posed problem. We discover an effective solution to reduce the learning ambiguity by expanding the single-view view synthesis problem to a multi-view setting. Specifically, we leverage the reliable and explicit stereo prior to generate a pseudo-stereo viewpoint, which serves as an auxiliary input to construct the 3D space. In this way, the challenging novel view synthesis process is decoupled into two simpler problems of stereo synthesis and 3D reconstruction. In order to synthesize a structurally correct and detail-preserved stereo image, we propose a self-rectified stereo synthesis to amend erroneous regions in an identify-rectify manner. Hard-to-train and incorrect warping samples are first discovered by two strategies, 1) pruning the network to reveal low-confident predictions; and 2) bidirectionally matching between stereo images to allow the discovery of improper mapping. These regions are then inpainted to form the final pseudo-stereo. With the aid of this extra input, a preferable 3D reconstruction can be easily obtained, and our method can work with arbitrary 3D representations. Extensive experiments show that our method outperforms state-of-the-art single-view view synthesis methods and stereo synthesis methods.
翻译:从单一视角图像合成新视角是一个高度不适定的问题。我们发现将单视图视图合成问题扩展到多视图设置可以有效减少学习的歧义。具体地,我们利用可靠和显式的立体视觉先验生成伪立体视图,其作为辅助输入构建三维空间。通过这种方法,具有挑战性的新视图合成过程被分解为两个较简单的问题:立体合成和三维重建。为了合成结构正确且保留细节的立体图像,我们提出了自校正立体合成来以识别-校正的方式修正错误区域。首先,通过两种策略发现难以训练和不正确的变形样本:1)通过修剪网络来显示低置信度预测;2)在立体图像之间进行双向匹配以允许发现不适当的映射。然后,将这些区域进行修补以形成最终的伪立体视图。在这个额外的输入的帮助下,可以轻松地获取更好的三维重建,并且我们的方法可以与任意三维表示一起使用。大量实验证明,我们的方法优于最先进的单视图视图合成方法和立体合成方法。