Diagnosing Catastrophe: Large parts of accuracy loss in continual learning can be accounted for by readout misalignment

Autor:	Anthes, Daniel, Thorat, Sushrut, König, Peter, Kietzmann, Tim C.
Rok vydání:	2023
Předmět:	Computer Science - Machine Learning Computer Science - Computer Vision and Pattern Recognition
Druh dokumentu:	Working Paper
Popis:	Unlike primates, training artificial neural networks on changing data distributions leads to a rapid decrease in performance on old tasks. This phenomenon is commonly referred to as catastrophic forgetting. In this paper, we investigate the representational changes that underlie this performance decrease and identify three distinct processes that together account for the phenomenon. The largest component is a misalignment between hidden representations and readout layers. Misalignment occurs due to learning on additional tasks and causes internal representations to shift. Representational geometry is partially conserved under this misalignment and only a small part of the information is irrecoverably lost. All types of representational changes scale with the dimensionality of hidden representations. These insights have implications for deep learning applications that need to be continuously updated, but may also aid aligning ANN models to the rather robust biological vision. Comment: 3 pages, 1 figure; published at the 2023 Conference on Cognitive Computational Neuroscience
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2310.05644 Zobrazit plný text záznamu View this record from Arxiv