NVFi: Neural Velocity Fields for 3D Physics Learning from Dynamic Videos

Autor:	Li, Jinxi, Song, Ziyang, Yang, Bo
Rok vydání:	2023
Předmět:	Computer Science - Computer Vision and Pattern Recognition Computer Science - Artificial Intelligence Computer Science - Graphics Computer Science - Machine Learning Computer Science - Robotics
Druh dokumentu:	Working Paper
Popis:	In this paper, we aim to model 3D scene dynamics from multi-view videos. Unlike the majority of existing works which usually focus on the common task of novel view synthesis within the training time period, we propose to simultaneously learn the geometry, appearance, and physical velocity of 3D scenes only from video frames, such that multiple desirable applications can be supported, including future frame extrapolation, unsupervised 3D semantic scene decomposition, and dynamic motion transfer. Our method consists of three major components, 1) the keyframe dynamic radiance field, 2) the interframe velocity field, and 3) a joint keyframe and interframe optimization module which is the core of our framework to effectively train both networks. To validate our method, we further introduce two dynamic 3D datasets: 1) Dynamic Object dataset, and 2) Dynamic Indoor Scene dataset. We conduct extensive experiments on multiple datasets, demonstrating the superior performance of our method over all baselines, particularly in the critical tasks of future frame extrapolation and unsupervised 3D semantic scene decomposition. Comment: NeurIPS 2023. Code and data are available at: https://github.com/vLAR-group/NVFi
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2312.06398 Zobrazit plný text záznamu View this record from Arxiv