Information criteria and cross validation for Bayesian inference in regular and singular cases
| Autor: | Sumio Watanabe |
|---|---|
| Rok vydání: | 2021 |
| Předmět: |
Statistics and Probability
Computer science business.industry Posterior probability Bayesian probability Statistical model Information Criteria 02 engineering and technology Machine learning computer.software_genre Bayesian inference 01 natural sciences Cross-validation Normal distribution 010104 statistics & probability Computational Theory and Mathematics 0202 electrical engineering electronic engineering information engineering 020201 artificial intelligence & image processing Artificial intelligence 0101 mathematics Akaike information criterion business computer |
| Zdroj: | Japanese Journal of Statistics and Data Science. 4:1-19 |
| ISSN: | 2520-8764 2520-8756 |
| DOI: | 10.1007/s42081-021-00121-3 |
| Popis: | In data science, an unknown information source is estimated by a predictive distribution defined from a statistical model and a prior. In an older Bayesian framework, it was explained that the Bayesian predictive distribution should be the best on the assumption that a statistical model is convinced to be correct and a prior is given by a subjective belief in a small world. However, such a restricted treatment of Bayesian inference cannot be applied to highly complicated statistical models and learning machines in a large world. In 1980, a new scientific paradigm of Bayesian inference was proposed by Akaike, in which both a model and a prior are candidate systems and they had better be designed by mathematical procedures so that the predictive distribution is the better approximation of unknown information source. Nowadays, Akaike’s proposal is widely accepted in statistics, data science, and machine learning. In this paper, in order to establish a mathematical foundation for developing a measure of a statistical model and a prior, we show the relation among the generalization loss, the information criteria, and the cross-validation loss, then compare them from three different points of view. First, their performances are compared in singular problems where the posterior distribution is far from any normal distribution. Second, they are studied in the case when a leverage sample point is contained in data. And last, their stochastic properties are clarified when they are used for the prior optimization problem. The mathematical and experimental comparison shows the equivalence and the difference among them, which we expect useful in practical applications. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|
Full text from SpringerLink