A Stochastic Variance Reduced Nesterov's Accelerated Quasi-Newton Method
| Autor: | Hiroshi Ninomiya, Shahrzad Mahboubi, Hideki Asai, S. Indrapriyadarsini, Sota Yasuda |
|---|---|
| Rok vydání: | 2019 |
| Předmět: |
FOS: Computer and information sciences
Computer Science - Machine Learning 021103 operations research Artificial neural network Computer science 0211 other engineering and technologies Machine Learning (stat.ML) 02 engineering and technology Variance (accounting) 010501 environmental sciences Curvature 01 natural sciences Machine Learning (cs.LG) Term (time) Momentum Noise Statistics - Machine Learning Broyden–Fletcher–Goldfarb–Shanno algorithm Benchmark (computing) Applied mathematics Quasi-Newton method 0105 earth and related environmental sciences |
| Zdroj: | ICMLA |
| DOI: | 10.1109/icmla.2019.00301 |
| Popis: | Recently algorithms incorporating second order curvature information have become popular in training neural networks. The Nesterov's Accelerated Quasi-Newton (NAQ) method has shown to effectively accelerate the BFGS quasi-Newton method by incorporating the momentum term and Nesterov's accelerated gradient vector. A stochastic version of NAQ method was proposed for training of large-scale problems. However, this method incurs high stochastic variance noise. This paper proposes a stochastic variance reduced Nesterov's Accelerated Quasi-Newton method in full (SVR-NAQ) and limited (SVRLNAQ) memory forms. The performance of the proposed method is evaluated in Tensorflow on four benchmark problems - two regression and two classification problems respectively. The results show improved performance compared to conventional methods. Accepted in ICMLA 2019 |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|