Naturally Constrained Online Expectation Maximization
| Autor: | Antoine Manzanera, Daniela Pamplona |
|---|---|
| Přispěvatelé: | Unité d'Informatique et d'Ingénierie des Systèmes (U2IS), École Nationale Supérieure de Techniques Avancées (ENSTA Paris), Manzanera, Antoine |
| Jazyk: | angličtina |
| Rok vydání: | 2021 |
| Předmět: |
Independent and identically distributed random variables
Forgetting Theoretical computer science Computer science business.industry Probabilistic logic 02 engineering and technology [INFO.INFO-LG] Computer Science [cs]/Machine Learning [cs.LG] 010501 environmental sciences 01 natural sciences Constraint (information theory) Rate of convergence [INFO.INFO-LG]Computer Science [cs]/Machine Learning [cs.LG] Convergence (routing) 0202 electrical engineering electronic engineering information engineering 020201 artificial intelligence & image processing Artificial intelligence Sequence learning Transfer of learning business 0105 earth and related environmental sciences |
| Zdroj: | International Conference on Pattern Recognition (ICPR 2020) International Conference on Pattern Recognition (ICPR 2020), Jan 2021, Milan, Italy ICPR |
| Popis: | International audience; With the rise of big data sets, learning algorithms must be adapted to piece-wise mechanisms to tackle large-scale calculations' time and memory costs. Furthermore, for most learning embedded systems, the input data are fed sequentially and contingently: one by one, and possibly class by class. Thus, learning algorithms should not only run online but cope with time-varying, non-independent, and non-balanced training data for the system's entire life. Online Expectation-Maximization is a well-known algorithm for learning probabilistic models in real-time, due to its simplicity and convergence properties. However, these properties are only valid in the case of large, independent and identically distributed samples. In this paper, we propose to constrain the online Expectation-Maximization on the Fisher distance between the parameters. After presenting the algorithm, we make a thorough study of its use in Probabilistic Principal Components Analysis. First, we derive the update rules, and then we analyze the effect of the constraint on major problems of online and sequential learning: convergence, forgetting and interference. Furthermore, we use several algorithmic protocols: iid vs sequential data, and constraint parameters updated step-wise vs class-wise. Our results show that this constraint increases the convergence rate of online Expectation-Maximization, decreases forgetting and slightly introduces positive transfer learning. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|