Cross-validation Confidence Intervals for Test Error

Autor:	Bayle, Pierre, Bayle, Alexandre, Janson, Lucas, Mackey, Lester
Rok vydání:	2020
Předmět:	Statistics - Machine Learning Computer Science - Machine Learning Mathematics - Statistics Theory
Druh dokumentu:	Working Paper
Popis:	This work develops central limit theorems for cross-validation and consistent estimators of its asymptotic variance under weak stability conditions on the learning algorithm. Together, these results provide practical, asymptotically-exact confidence intervals for $k$-fold test error and valid, powerful hypothesis tests of whether one learning algorithm has smaller $k$-fold test error than another. These results are also the first of their kind for the popular choice of leave-one-out cross-validation. In our real-data experiments with diverse learning algorithms, the resulting intervals and tests outperform the most popular alternative methods from the literature. Comment: 34th Conference on Neural Information Processing Systems (NeurIPS 2020); 40 pages, 15 figures
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2007.12671 Zobrazit plný text záznamu View this record from Arxiv