Learning a Low-Rank Feature Representation: Achieving Better Trade-Off between Stability and Plasticity in Continual Learning

Autor:	Liu, Zhenrong, Li, Yang, Gong, Yi, Wu, Yik-Chung
Rok vydání:	2023
Předmět:	Computer Science - Machine Learning Computer Science - Computer Vision and Pattern Recognition
Druh dokumentu:	Working Paper
Popis:	In continual learning, networks confront a trade-off between stability and plasticity when trained on a sequence of tasks. To bolster plasticity without sacrificing stability, we propose a novel training algorithm called LRFR. This approach optimizes network parameters in the null space of the past tasks' feature representation matrix to guarantee the stability. Concurrently, we judiciously select only a subset of neurons in each layer of the network while training individual tasks to learn the past tasks' feature representation matrix in low-rank. This increases the null space dimension when designing network parameters for subsequent tasks, thereby enhancing the plasticity. Using CIFAR-100 and TinyImageNet as benchmark datasets for continual learning, the proposed approach consistently outperforms state-of-the-art methods. Comment: Accepted for publication in the proceedings of ICASSP 2024
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2312.08740 Zobrazit plný text záznamu View this record from Arxiv