Optimization Methods for Large-Scale Machine Learning
| Autor: | Frank E. Curtis, Jorge Nocedal, Léon Bottou |
|---|---|
| Jazyk: | angličtina |
| Rok vydání: | 2016 |
| Předmět: |
FOS: Computer and information sciences
Optimization problem Computer science Machine Learning (stat.ML) Context (language use) 010103 numerical & computational mathematics 02 engineering and technology Machine learning computer.software_genre 01 natural sciences Machine Learning (cs.LG) Theoretical Computer Science Nonlinear programming Statistics - Machine Learning FOS: Mathematics 0202 electrical engineering electronic engineering information engineering 0101 mathematics Mathematics - Optimization and Control Optimization algorithm business.industry Applied Mathematics Scale (chemistry) Computational Mathematics Improved performance Computer Science - Learning Optimization and Control (math.OC) Optimization methods Deep neural networks 020201 artificial intelligence & image processing Artificial intelligence business computer |
| Popis: | This paper provides a review and commentary on the past, present, and future of numerical optimization algorithms in the context of machine learning applications. Through case studies on text classification and the training of deep neural networks, we discuss how optimization problems arise in machine learning and what makes them challenging. A major theme of our study is that large-scale machine learning represents a distinctive setting in which the stochastic gradient (SG) method has traditionally played a central role while conventional gradient-based nonlinear optimization techniques typically falter. Based on this viewpoint, we present a comprehensive theory of a straightforward, yet versatile SG algorithm, discuss its practical behavior, and highlight opportunities for designing algorithms with improved performance. This leads to a discussion about the next generation of optimization methods for large-scale machine learning, including an investigation of two main streams of research on techniques that diminish noise in the stochastic directions and methods that make use of second-order derivative approximations. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|