A Contour Stochastic Gradient Langevin Dynamics Algorithm for Simulations of Multi-modal Distributions

Autor:	Deng, Wei, Lin, Guang, Liang, Faming
Rok vydání:	2020
Předmět:	Statistics - Machine Learning Computer Science - Machine Learning Statistics - Computation
Druh dokumentu:	Working Paper
Popis:	We propose an adaptively weighted stochastic gradient Langevin dynamics algorithm (SGLD), so-called contour stochastic gradient Langevin dynamics (CSGLD), for Bayesian learning in big data statistics. The proposed algorithm is essentially a \emph{scalable dynamic importance sampler}, which automatically \emph{flattens} the target distribution such that the simulation for a multi-modal distribution can be greatly facilitated. Theoretically, we prove a stability condition and establish the asymptotic convergence of the self-adapting parameter to a {\it unique fixed-point}, regardless of the non-convexity of the original energy function; we also present an error analysis for the weighted averaging estimators. Empirically, the CSGLD algorithm is tested on multiple benchmark datasets including CIFAR10 and CIFAR100. The numerical results indicate its superiority to avoid the local trap problem in training deep neural networks. Comment: Accepted by NeurIPS 2020
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2010.09800 Zobrazit plný text záznamu View this record from Arxiv