Image-based geometric modeling and novel view synthesis based on sparse, large-baseline samplings are challenging but important tasks for emerging multimedia applications such as virtual reality and immersive telepresence. Existing methods fail to produce satisfactory results due to the limitation on inferring reliable depth information over such challenging reference conditions. With the popularization of commercial light field (LF) cameras, capturing LF images (LFIs) is as convenient as taking regular photos, and geometry information can be reliably inferred. This inspires us to use a sparse set of LF captures to render high-quality novel views globally. However, fusion of LF captures from multiple angles is challenging due to the scale inconsistency caused by various capture settings. To overcome this challenge, we propose a novel scale-consistent volume rescaling algorithm that robustly aligns the disparity probability volumes (DPV) among different captures for scale-consistent global geometry fusion. Based on the fused DPV projected to the target camera frustum, novel learning-based modules have been proposed (i.e., the attention-guided multi-scale residual fusion module, and the disparity field guided deep re-regularization module) which comprehensively regularize noisy observations from heterogeneous captures for high-quality rendering of novel LFIs. Both quantitative and qualitative experiments over the Stanford Lytro Multi-view LF dataset show that the proposed method outperforms state-of-the-art methods significantly under different experiment settings for disparity inference and LF synthesis.
翻译:以分散、大基线抽样为基础的基于图像的建模模型和新观点合成,对于虚拟现实和隐蔽的远程存在等新兴多媒体应用而言,挑战性但很重要。现有方法无法产生令人满意的结果,因为无法根据这种具有挑战性的参考条件推断可靠的深度信息。随着商业光场(LF)摄像头的普及,捕捉LF图像(LFIs)与定期拍照一样方便,而且可以可靠地推断几何信息。这促使我们使用一组稀少的LF捕获来在全球形成高质量的新观点。然而,由于各种捕获环境造成的规模不一致,LF从多个角度捕获的集成具有挑战性。为了克服这一挑战,我们提出了一个新的与规模一致的量级调整算法,使不同捕捉到的LFI图像(DPV)与定期拍照一样容易,并且可以可靠地推断目标摄取的DPV,因此提出了新的基于学习的模块(i.e.e.,从多个角度收集的LFI-CRI(I)不同层次观察的注意-导导导价的多级的多级的多级标准级模型模型模型,并展示了跨级的模型的模型,以显示高级的跨级的模型的模型。