Applying data-driven approaches to non-rigid 3D reconstruction has been difficult, which we believe can be attributed to the lack of a large-scale training corpus. One recent approach proposes self-supervision based on non-rigid reconstruction. Unfortunately, this method fails for important cases such as highly non-rigid deformations. We first address this problem of lack of data by introducing a novel semi-supervised strategy to obtain dense inter-frame correspondences from a sparse set of annotations. This way, we obtain a large dataset of 400 scenes, over 390,000 RGB-D frames, and 2,537 densely aligned frame pairs; in addition, we provide a test set along with several metrics for evaluation. Based on this corpus, we introduce a data-driven non-rigid feature matching approach, which we integrate into an optimization-based reconstruction pipeline. Here, we propose a new neural network that operates on RGB-D frames, while maintaining robustness under large non-rigid deformations and producing accurate predictions. Our approach significantly outperforms both existing non-rigid reconstruction methods that do not use learned data terms, as well as learning-based approaches that only use self-supervision.
翻译:对非硬化3D重建应用数据驱动方法一直很困难,我们认为,这可以归因于缺乏大规模培训材料。最近的一种方法提出基于非硬化重建的自我监督。不幸的是,这种方法在诸如高度非硬化变形等重要情况下失败。我们首先采用新的半监督战略,从稀疏的一组说明中获取密集的跨框架通信,以解决数据缺乏问题。这样,我们获得了400个大数据集,超过390,000 RGB-D框架,以及2,537个高度一致的框架配对;此外,我们提供了一套测试,并附有若干评价指标。基于这一工具,我们采用了一种以数据驱动的非硬化特征匹配方法,我们将其纳入一个基于优化重建管道。我们在这里提出了一个新的神经网络,在RGB-D框架下运作,同时保持非硬化型变形下的坚固性,并产生准确的预测。我们的方法大大优于现有的非硬性重建方法,但并不使用学习的数据术语,而是作为学习的升级方法。