A Contracting BFGS Update in Quasi-Newton Methods for Unconstrained Optimization
| Autor: | Jiongcheng Li, Hai-Lin Liu |
|---|---|
| Rok vydání: | 2019 |
| Předmět: |
Hessian matrix
Mathematical optimization Line search Computer science 020209 energy 020208 electrical & electronic engineering MathematicsofComputing_NUMERICALANALYSIS 02 engineering and technology Matrix (mathematics) symbols.namesake Quadratic equation Positive-definite function Broyden–Fletcher–Goldfarb–Shanno algorithm 0202 electrical engineering electronic engineering information engineering symbols Symmetric matrix Newton's method Eigenvalues and eigenvectors |
| Zdroj: | CIS |
| DOI: | 10.1109/cis.2019.00021 |
| Popis: | Unconstrained optimization problems, arise in many practical applications. Especially, significant improvement in deep learning training came from the Quasi-Newton methods. They exploit the idea of building up curvature information as the iterations of the training method are progressing. BFGS update in Quasi-Newton methods is the most commonly used update rule for training deep neural networks. The accuracy of computed search direction depends largely on how sensitive the Hessian approximation matrix Bk+1 is to small changes. The larger distribution of eigenvalues of the matrix will cause more sensation. In this paper, we propose a contracting BFGS update (C-BFGS), in order to contract the interval of distribution. The new update retains the Hessian approximation matrix positive definiteness, so that it makes sure the search direction down. For a quadratic positive definite function, the search directions generated by C-BFGS under the exact line search are Gconjugate, and the Quasi-Newton method with C-BFGS update satisfies quadratic termination property. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|