Is it the model or the metric -- On robustness measures of deeplearning models

Autor:	Lyu, Zhijin, Jin, Yutong, Das, Sneha
Rok vydání:	2024
Předmět:	Computer Science - Machine Learning Computer Science - Computer Vision and Pattern Recognition
Druh dokumentu:	Working Paper
Popis:	Determining the robustness of deep learning models is an established and ongoing challenge within automated decision-making systems. With the advent and success of techniques that enable advanced deep learning (DL), these models are being used in widespread applications, including high-stake ones like healthcare, education, border-control. Therefore, it is critical to understand the limitations of these models and predict their regions of failures, in order to create the necessary guardrails for their successful and safe deployment. In this work, we revisit robustness, specifically investigating the sufficiency of robust accuracy (RA), within the context of deepfake detection. We present robust ratio (RR) as a complementary metric, that can quantify the changes to the normalized or probability outcomes under input perturbation. We present a comparison of RA and RR and demonstrate that despite similar RA between models, the models show varying RR under different tolerance (perturbation) levels. Comment: Extended abstract at Northern Lights Deep Learning (NLDL) Conference 2025
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2412.09795 Zobrazit plný text záznamu View this record from Arxiv