Text-guided Foundation Model Adaptation for Pathological Image Classification

Autor:	Zhang, Yunkun, Gao, Jin, Zhou, Mu, Wang, Xiaosong, Qiao, Yu, Zhang, Shaoting, Wang, Dequan
Rok vydání:	2023
Předmět:	Computer Science - Computer Vision and Pattern Recognition Computer Science - Artificial Intelligence
Druh dokumentu:	Working Paper
Popis:	The recent surge of foundation models in computer vision and natural language processing opens up perspectives in utilizing multi-modal clinical data to train large models with strong generalizability. Yet pathological image datasets often lack biomedical text annotation and enrichment. Guiding data-efficient image diagnosis from the use of biomedical text knowledge becomes a substantial interest. In this paper, we propose to Connect Image and Text Embeddings (CITE) to enhance pathological image classification. CITE injects text insights gained from language models pre-trained with a broad range of biomedical texts, leading to adapt foundation models towards pathological image understanding. Through extensive experiments on the PatchGastric stomach tumor pathological image dataset, we demonstrate that CITE achieves leading performance compared with various baselines especially when training data is scarce. CITE offers insights into leveraging in-domain text knowledge to reinforce data-efficient pathological image classification. Code is available at https://github.com/Yunkun-Zhang/CITE. Comment: Accepted to MICCAI2023
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2307.14901 Zobrazit plný text záznamu View this record from Arxiv