CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memory

Autor:	Shafiullah, Nur Muhammad Mahi, Paxton, Chris, Pinto, Lerrel, Chintala, Soumith, Szlam, Arthur
Rok vydání:	2022
Předmět:	Computer Science - Robotics Computer Science - Computer Vision and Pattern Recognition
Druh dokumentu:	Working Paper
Popis:	We propose CLIP-Fields, an implicit scene model that can be used for a variety of tasks, such as segmentation, instance identification, semantic search over space, and view localization. CLIP-Fields learns a mapping from spatial locations to semantic embedding vectors. Importantly, we show that this mapping can be trained with supervision coming only from web-image and web-text trained models such as CLIP, Detic, and Sentence-BERT; and thus uses no direct human supervision. When compared to baselines like Mask-RCNN, our method outperforms on few-shot instance identification or semantic segmentation on the HM3D dataset with only a fraction of the examples. Finally, we show that using CLIP-Fields as a scene memory, robots can perform semantic navigation in real-world environments. Our code and demonstration videos are available here: https://mahis.life/clip-fields Comment: Code, video, and interactive demonstrations available at https://mahis.life/clip-fields. Accepted for publication at Robotics: Science and Systems 2023 in Daegu, Korea
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2210.05663 Zobrazit plný text záznamu View this record from Arxiv