Archetypal landscapes for deep neural networks
| Autor: | Philipp Verpoort, Alpha A. Lee, David J. Wales |
|---|---|
| Přispěvatelé: | Verpoort, Philipp C [0000-0003-1319-5006], Wales, David J [0000-0002-3555-6645], Apollo - University of Cambridge Repository, University of Cambridge [UK] (CAM), ANR-19-P3IA-0002,3IA@cote d'azur,3IA Côte d'Azur(2019) |
| Rok vydání: | 2020 |
| Předmět: |
Mathematical optimization
Computer science [INFO.INFO-CG]Computer Science [cs]/Computational Geometry [cs.CG] 01 natural sciences 0103 physical sciences Limit (mathematics) 010306 general physics ComputingMilieux_MISCELLANEOUS Multidisciplinary 010304 chemical physics Artificial neural network business.industry Deep learning deep learning Statistical mechanics neural networks Maxima and minima Stochastic gradient descent energy landscapes 13. Climate action Physical Sciences Deep neural networks statistical mechanics Minification Artificial intelligence business optimization |
| Zdroj: | Proceedings of the National Academy of Sciences of the United States of America Proceedings of the National Academy of Sciences of the United States of America, National Academy of Sciences, 2020, 117 (36), pp.21857-21864. ⟨10.1073/pnas.1919995117⟩ Proc Natl Acad Sci U S A |
| ISSN: | 1091-6490 0027-8424 |
| Popis: | The predictive capabilities of deep neural networks (DNNs) continue to evolve to increasingly impressive levels. However, it is still unclear how training procedures for DNNs succeed in finding parameters that produce good results for such high-dimensional and nonconvex loss functions. In particular, we wish to understand why simple optimization schemes, such as stochastic gradient descent, do not end up trapped in local minima with high loss values that would not yield useful predictions. We explain the optimizability of DNNs by characterizing the local minima and transition states of the loss-function landscape (LFL) along with their connectivity. We show that the LFL of a DNN in the shallow network or data-abundant limit is funneled, and thus easy to optimize. Crucially, in the opposite low-data/deep limit, although the number of minima increases, the landscape is characterized by many minima with similar loss values separated by low barriers. This organization is different from the hierarchical landscapes of structural glass formers and explains why minimization procedures commonly employed by the machine-learning community can navigate the LFL successfully and reach low-lying solutions. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|