Photorealistic Object Insertion with Diffusion-Guided Inverse Rendering

Autor:	Liang, Ruofan, Gojcic, Zan, Nimier-David, Merlin, Acuna, David, Vijaykumar, Nandita, Fidler, Sanja, Wang, Zian
Rok vydání:	2024
Předmět:	Computer Science - Computer Vision and Pattern Recognition Computer Science - Artificial Intelligence Computer Science - Graphics
Druh dokumentu:	Working Paper
Popis:	The correct insertion of virtual objects in images of real-world scenes requires a deep understanding of the scene's lighting, geometry and materials, as well as the image formation process. While recent large-scale diffusion models have shown strong generative and inpainting capabilities, we find that current models do not sufficiently "understand" the scene shown in a single picture to generate consistent lighting effects (shadows, bright reflections, etc.) while preserving the identity and details of the composited object. We propose using a personalized large diffusion model as guidance to a physically based inverse rendering process. Our method recovers scene lighting and tone-mapping parameters, allowing the photorealistic composition of arbitrary virtual objects in single frames or videos of indoor or outdoor scenes. Our physically based pipeline further enables automatic materials and tone-mapping refinement. Comment: ECCV 2024, Project page: https://research.nvidia.com/labs/toronto-ai/DiPIR/
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2408.09702 Zobrazit plný text záznamu View this record from Arxiv