Hot Swapping for Online Adaptation of Optimization Hyperparameters

Autor:	Bache, Kevin, DeCoste, Dennis, Smyth, Padhraic
Rok vydání:	2014
Předmět:	Computer Science - Learning 62L20 G.1.6 I.2.6
Druh dokumentu:	Working Paper
Popis:	We describe a general framework for online adaptation of optimization hyperparameters by `hot swapping' their values during learning. We investigate this approach in the context of adaptive learning rate selection using an explore-exploit strategy from the multi-armed bandit literature. Experiments on a benchmark neural network show that the hot swapping approach leads to consistently better solutions compared to well-known alternatives such as AdaDelta and stochastic gradient with exhaustive hyperparameter search. Comment: Submission to ICLR 2015
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/1412.6599 Zobrazit plný text záznamu View this record from Arxiv