3D Reconstruction of Objects in Hands without Real World 3D Supervision

Autor:	Prakash, Aditya, Chang, Matthew, Jin, Matthew, Tu, Ruisen, Gupta, Saurabh
Rok vydání:	2023
Předmět:	Computer Science - Computer Vision and Pattern Recognition Computer Science - Artificial Intelligence Computer Science - Machine Learning
Druh dokumentu:	Working Paper
Popis:	Prior works for reconstructing hand-held objects from a single image train models on images paired with 3D shapes. Such data is challenging to gather in the real world at scale. Consequently, these approaches do not generalize well when presented with novel objects in in-the-wild settings. While 3D supervision is a major bottleneck, there is an abundance of a) in-the-wild raw video data showing hand-object interactions and b) synthetic 3D shape collections. In this paper, we propose modules to leverage 3D supervision from these sources to scale up the learning of models for reconstructing hand-held objects. Specifically, we extract multiview 2D mask supervision from videos and 3D shape priors from shape collections. We use these indirect 3D cues to train occupancy networks that predict the 3D shape of objects from a single RGB image. Our experiments in the challenging object generalization setting on in-the-wild MOW dataset show 11.6% relative improvement over models trained with 3D supervision on existing datasets. Comment: ECCV 2024, Project Webpage: https://ap229997.github.io/projects/wild-hoi/
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2305.03036 Zobrazit plný text záznamu View this record from Arxiv