Gradient descent in matrix factorization: Understanding large initialization

Autor:	Chen, Hengchao, Chen, Xin, Elmasri, Mohamad, Sun, Qiang
Rok vydání:	2023
Předmět:	Mathematics - Optimization and Control Statistics - Machine Learning
Druh dokumentu:	Working Paper
Popis:	Gradient Descent (GD) has been proven effective in solving various matrix factorization problems. However, its optimization behavior with large initial values remains less understood. To address this gap, this paper presents a novel theoretical framework for examining the convergence trajectory of GD with a large initialization. The framework is grounded in signal-to-noise ratio concepts and inductive arguments. The results uncover an implicit incremental learning phenomenon in GD and offer a deeper understanding of its performance in large initialization scenarios. Comment: Published in the 40th Conference on Uncertainty in Artificial Intelligence (UAI 2024)
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2305.19206 Zobrazit plný text záznamu View this record from Arxiv