CAME: Contrastive Automated Model Evaluation

Autor:	Peng, Ru, Duan, Qiuyang, Wang, Haobo, Ma, Jiachen, Jiang, Yanbo, Tu, Yongjun, Jiang, Xiu, Zhao, Junbo
Rok vydání:	2023
Předmět:	Computer Science - Computer Vision and Pattern Recognition Computer Science - Artificial Intelligence Computer Science - Machine Learning
Druh dokumentu:	Working Paper
Popis:	The Automated Model Evaluation (AutoEval) framework entertains the possibility of evaluating a trained machine learning model without resorting to a labeled testing set. Despite the promise and some decent results, the existing AutoEval methods heavily rely on computing distribution shifts between the unlabelled testing set and the training set. We believe this reliance on the training set becomes another obstacle in shipping this technology to real-world ML development. In this work, we propose Contrastive Automatic Model Evaluation (CAME), a novel AutoEval framework that is rid of involving training set in the loop. The core idea of CAME bases on a theoretical analysis which bonds the model performance with a contrastive loss. Further, with extensive empirical validation, we manage to set up a predictable relationship between the two, simply by deducing on the unlabeled/unseen testing set. The resulting framework CAME establishes a new SOTA results for AutoEval by surpassing prior work significantly. Comment: ICCV2023 main conference
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2308.11111 Zobrazit plný text záznamu View this record from Arxiv