Autor:	Lafond, Jean, Vasilache, Nicolas, Bottou, Léon
Rok vydání:	2017
Předmět:	Computer Science - Learning Statistics - Machine Learning
Druh dokumentu:	Working Paper
Popis:	We define a second-order neural network stochastic gradient training algorithm whose block-diagonal structure effectively amounts to normalizing the unit activations. Investigating why this algorithm lacks in robustness then reveals two interesting insights. The first insight suggests a new way to scale the stepsizes, clarifying popular algorithms such as RMSProp as well as old neural network tricks such as fanin stepsize scaling. The second insight stresses the practical importance of dealing with fast changes of the curvature of the cost.
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/1705.09319 Zobrazit plný text záznamu View this record from Arxiv