Adaptive Sampling Strategies for Stochastic Optimization
| Autor: | Richard H. Byrd, Raghu Bollapragada, Jorge Nocedal |
|---|---|
| Jazyk: | angličtina |
| Rok vydání: | 2017 |
| Předmět: |
Sample selection
FOS: Computer and information sciences Mathematical optimization 021103 operations research Adaptive sampling Computation 0211 other engineering and technologies Machine Learning (stat.ML) 02 engineering and technology Theoretical Computer Science Sample size determination Optimization and Control (math.OC) Statistics - Machine Learning 0202 electrical engineering electronic engineering information engineering FOS: Mathematics 020201 artificial intelligence & image processing Stochastic optimization Variance reduction Mathematics - Optimization and Control Software Mathematics |
| Popis: | In this paper, we propose a stochastic optimization method that adaptively controls the sample size used in the computation of gradient approximations. Unlike other variance reduction techniques that either require additional storage or the regular computation of full gradients, the proposed method reduces variance by increasing the sample size as needed. The decision to increase the sample size is governed by an inner product test that ensures that search directions are descent directions with high probability. We show that the inner product test improves upon the well known norm test, and can be used as a basis for an algorithm that is globally convergent on nonconvex functions and enjoys a global linear rate of convergence on strongly convex functions. Numerical experiments on logistic regression problems illustrate the performance of the algorithm. 32 Pages |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|