No Train, all Gain: Self-Supervised Gradients Improve Deep Frozen Representations

Autor:	Simoncini, Walter, Gidaris, Spyros, Bursuc, Andrei, Asano, Yuki M.
Rok vydání:	2024
Předmět:	Computer Science - Computer Vision and Pattern Recognition Computer Science - Computation and Language Computer Science - Machine Learning
Druh dokumentu:	Working Paper
Popis:	This paper introduces FUNGI, Features from UNsupervised GradIents, a method to enhance the features of vision encoders by leveraging self-supervised gradients. Our method is simple: given any pretrained model, we first compute gradients from various self-supervised objectives for each input. These are projected to a lower dimension and then concatenated with the model's embedding. The resulting features are evaluated on k-nearest neighbor classification over 11 datasets from vision, 5 from natural language processing, and 2 from audio. Across backbones spanning various sizes and pretraining strategies, FUNGI features provide consistent performance improvements over the embeddings. We also show that using FUNGI features can benefit linear classification and image retrieval, and that they significantly improve the retrieval-based in-context scene understanding abilities of pretrained models, for example improving upon DINO by +17% for semantic segmentation - without any training. Comment: Preprint. Code available at https://github.com/WalterSimoncini/fungivision
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2407.10964 Zobrazit plný text záznamu View this record from Arxiv