Learning Dynamics in Linear VAE: Posterior Collapse Threshold, Superfluous Latent Space Pitfalls, and Speedup with KL Annealing

Autor:	Ichikawa, Yuma, Hukushima, Koji
Rok vydání:	2023
Předmět:	Statistics - Machine Learning Condensed Matter - Disordered Systems and Neural Networks Computer Science - Machine Learning Mathematics - Statistics Theory
Druh dokumentu:	Working Paper
Popis:	Variational autoencoders (VAEs) face a notorious problem wherein the variational posterior often aligns closely with the prior, a phenomenon known as posterior collapse, which hinders the quality of representation learning. To mitigate this problem, an adjustable hyperparameter $\beta$ and a strategy for annealing this parameter, called KL annealing, are proposed. This study presents a theoretical analysis of the learning dynamics in a minimal VAE. It is rigorously proved that the dynamics converge to a deterministic process within the limit of large input dimensions, thereby enabling a detailed dynamical analysis of the generalization error. Furthermore, the analysis shows that the VAE initially learns entangled representations and gradually acquires disentangled representations. A fixed-point analysis of the deterministic process reveals that when $\beta$ exceeds a certain threshold, posterior collapse becomes inevitable regardless of the learning period. Additionally, the superfluous latent variables for the data-generative factors lead to overfitting of the background noise; this adversely affects both generalization and learning convergence. The analysis further unveiled that appropriately tuned KL annealing can accelerate convergence. Comment: 24 pages, 5 figures
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2310.15440 Zobrazit plný text záznamu View this record from Arxiv