A stochastic extra-step quasi-Newton method for nonsmooth nonconvex optimization
Autor: | Tong Zhang, Zaiwen Wen, Andre Milzarek, Minghan Yang |
---|---|
Rok vydání: | 2021 |
Předmět: |
021103 operations research
business.industry General Mathematics Deep learning Numerical analysis MathematicsofComputing_NUMERICALANALYSIS 0211 other engineering and technologies 010103 numerical & computational mathematics 02 engineering and technology Extension (predicate logic) Type (model theory) 01 natural sciences Stationary point Convergence (routing) Applied mathematics Quasi-Newton method Variance reduction Artificial intelligence 0101 mathematics business Software Mathematics |
Zdroj: | Mathematical Programming. 194:257-303 |
ISSN: | 1436-4646 0025-5610 |
Popis: | In this paper, a novel stochastic extra-step quasi-Newton method is developed to solve a class of nonsmooth nonconvex composite optimization problems. We assume that the gradient of the smooth part of the objective function can only be approximated by stochastic oracles. The proposed method combines general stochastic higher order steps derived from an underlying proximal type fixed-point equation with additional stochastic proximal gradient steps to guarantee convergence. Based on suitable bounds on the step sizes, we establish global convergence to stationary points in expectation and an extension of the approach using variance reduction techniques is discussed. Motivated by large-scale and big data applications, we investigate a stochastic coordinate-type quasi-Newton scheme that allows to generate cheap and tractable stochastic higher order directions. Finally, numerical results on large-scale logistic regression and deep learning problems show that our proposed algorithm compares favorably with other state-of-the-art methods. |
Databáze: | OpenAIRE |
Externí odkaz: |