Generalizing Information to the Evolution of Rational Belief
| Autor: | Thomas A. Catanach, Jed A. Duersch |
|---|---|
| Rok vydání: | 2020 |
| Předmět: |
FOS: Computer and information sciences
Kullback–Leibler divergence Theoretical computer science Computer science Computer Science - Information Theory General Physics and Astronomy Machine Learning (stat.ML) lcsh:Astrophysics Feature selection Information theory Bayesian inference 01 natural sciences Article information 010104 statistics & probability Statistics - Machine Learning lcsh:QB460-466 0103 physical sciences Entropy (information theory) 0101 mathematics mutual information lcsh:Science 94A15 62A01 62B10 62F15 68Q32 94A17 010306 general physics self information maximal uncertainty bayesian inference proper utility Information Theory (cs.IT) Self-information lindley information kullback–leibler divergence Mutual information lcsh:QC1-999 Cross entropy lcsh:Q entropy lcsh:Physics |
| Zdroj: | Entropy, Vol 22, Iss 1, p 108 (2020) Entropy Volume 22 Issue 1 |
| ISSN: | 1099-4300 |
| DOI: | 10.3390/e22010108 |
| Popis: | Information theory provides a mathematical foundation to measure uncertainty in belief. Belief is represented by a probability distribution that captures our understanding of an outcome&rsquo s plausibility. Information measures based on Shannon&rsquo s concept of entropy include realization information, Kullback&ndash Leibler divergence, Lindley&rsquo s information in experiment, cross entropy, and mutual information. We derive a general theory of information from first principles that accounts for evolving belief and recovers all of these measures. Rather than simply gauging uncertainty, information is understood in this theory to measure change in belief. We may then regard entropy as the information we expect to gain upon realization of a discrete latent random variable. This theory of information is compatible with the Bayesian paradigm in which rational belief is updated as evidence becomes available. Furthermore, this theory admits novel measures of information with well-defined properties, which we explored in both analysis and experiment. This view of information illuminates the study of machine learning by allowing us to quantify information captured by a predictive model and distinguish it from residual information contained in training data. We gain related insights regarding feature selection, anomaly detection, and novel Bayesian approaches. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|
| Nepřihlášeným uživatelům se plný text nezobrazuje | K zobrazení výsledku je třeba se přihlásit. |