Supervised Fine-tuning in turn Improves Visual Foundation Models

Autor:	Jiang, Xiaohu, Ge, Yixiao, Ge, Yuying, Shi, Dachuan, Yuan, Chun, Shan, Ying
Rok vydání:	2024
Předmět:	Computer Science - Computer Vision and Pattern Recognition Computer Science - Artificial Intelligence
Druh dokumentu:	Working Paper
Popis:	Image-text training like CLIP has dominated the pretraining of vision foundation models in recent years. Subsequent efforts have been made to introduce region-level visual learning into CLIP's pretraining but face scalability challenges due to the lack of large-scale region-level datasets. Drawing inspiration from supervised fine-tuning (SFT) in natural language processing such as instruction tuning, we explore the potential of fine-grained SFT in enhancing the generation of vision foundation models after their pretraining. Thus a two-stage method ViSFT (Vision SFT) is proposed to unleash the fine-grained knowledge of vision foundation models. In ViSFT, the vision foundation model is enhanced by performing visual joint learning on some in-domain tasks and then tested on out-of-domain benchmarks. With updating using ViSFT on 8 V100 GPUs in less than 2 days, a vision transformer with over 4.4B parameters shows improvements across various out-of-domain benchmarks including vision and vision-linguistic scenarios. Comment: 23 pages, 3 figures, Project page: https://github.com/TencentARC/ViSFT/tree/main
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2401.10222 Zobrazit plný text záznamu View this record from Arxiv