The Data Representativeness Criterion: Predicting the Performance of Supervised Classification Based on Data Set Similarity
| Autor: | Adriënne M. Mendrik, Rens van de Schoot, Duco Veen, Wouter M. Kouw, Evelien Schat |
|---|---|
| Přispěvatelé: | Signal Processing Systems |
| Jazyk: | angličtina |
| Rok vydání: | 2020 |
| Předmět: |
Central Nervous System
Data Analysis FOS: Computer and information sciences Computer science Computer Vision Computer Vision and Pattern Recognition (cs.CV) Bayesian inference Computer Science - Computer Vision and Pattern Recognition 02 engineering and technology computer.software_genre Nervous System Diagnostic Radiology Pattern Recognition Automated Machine Learning DESIGN Medicine and Health Sciences Image Processing Computer-Assisted 0202 electrical engineering electronic engineering information engineering BRAIN Multidisciplinary Training set Radiology and Imaging Applied Mathematics Simulation and Modeling Magnetic Resonance Imaging Multidisciplinary Sciences Data Acquisition Data Interpretation Statistical Physical Sciences Science & Technology - Other Topics Medicine 020201 artificial intelligence & image processing Supervised Machine Learning Anatomy Transfer of learning Algorithms Data set similarity Research Article Computer and Information Sciences Imaging Techniques Science Research and Analysis Methods Machine learning Proof of Concept Study Representativeness heuristic Methodology (stat.ME) Machine Learning Algorithms Diagnostic Medicine Artificial Intelligence 020204 information systems Humans Statistics - Methodology Science & Technology business.industry Biology and Life Sciences Probability Theory Probability Distribution Transfer learning Data set Artificial intelligence business Classifier (UML) computer Mathematics |
| Zdroj: | PLoS ONE, Vol 15, Iss 8, p e0237009 (2020) PLoS ONE PLoS ONE, 15(8):e0237009. Public Library of Science |
| ISSN: | 1932-6203 |
| Popis: | In a broad range of fields it may be desirable to reuse a supervised classification algorithm and apply it to a new data set. However, generalization of such an algorithm and thus achieving a similar classification performance is only possible when the training data used to build the algorithm is similar to new unseen data one wishes to apply it to. It is often unknown in advance how an algorithm will perform on new unseen data, being a crucial reason for not deploying an algorithm at all. Therefore, tools are needed to measure the similarity of data sets. In this paper, we propose the Data Representativeness Criterion (DRC) to determine how representative a training data set is of a new unseen data set. We present a proof of principle, to see whether the DRC can quantify the similarity of data sets and whether the DRC relates to the performance of a supervised classification algorithm. We compared a number of magnetic resonance imaging (MRI) data sets, ranging from subtle to severe difference is acquisition parameters. Results indicate that, based on the similarity of data sets, the DRC is able to give an indication as to when the performance of a supervised classifier decreases. The strictness of the DRC can be set by the user, depending on what one considers to be an acceptable underperformance. 12 pages, 6 figures |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|