Overcoming Catastrophic Forgetting with Gaussian Mixture Replay
| Autor: | Benedikt Pfülb, Alexander Gepperth |
|---|---|
| Rok vydání: | 2021 |
| Předmět: |
FOS: Computer and information sciences
Computer Science - Machine Learning Forgetting Artificial neural network Computer science business.industry Gaussian Pattern recognition Density estimation Mixture model Machine Learning (cs.LG) symbols.namesake Outlier Feature (machine learning) symbols Artificial intelligence business Time complexity |
| Zdroj: | IJCNN |
| Popis: | We present Gaussian Mixture Replay (GMR), a rehearsal-based approach for continual learning (CL) based on Gaussian Mixture Models (GMM). CL approaches are intended to tackle the problem of catastrophic forgetting (CF), which occurs for Deep Neural Networks (DNNs) when sequentially training them on successive sub-tasks. GMR mitigates CF by generating samples from previous tasks and merging them with current training data. GMMs serve several purposes here: sample generation, density estimation (e.g., for detecting outliers or recognizing task boundaries) and providing a high-level feature representation for classification. GMR has several conceptual advantages over existing replay-based CL approaches. First of all, GMR achieves sample generation, classification and density estimation in a single network structure with strongly reduced memory requirements. Secondly, it can be trained at constant time complexity w.r.t. the number of sub-tasks, making it particularly suitable for life-long learning. Furthermore, GMR minimizes a differentiable loss function and seems to avoid mode collapse. In addition, task boundaries can be detected by applying GMM density estimation. Lastly, GMR does not require access to sub-tasks lying in the future for hyper-parameter tuning, allowing CL under real-world constraints. We evaluate GMR on multiple image datasets, which are divided into class-disjoint sub-tasks. accepted at IJCNN2021, 9 pages, 12 figures |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|