Value Function Based Difference-of-Convex Algorithm for Bilevel Hyperparameter Selection Problems

Autor:	Gao, Lucy, Ye, Jane J., Yin, Haian, Zeng, Shangzhi, Zhang, Jin
Rok vydání:	2022
Předmět:	Mathematics - Optimization and Control Computer Science - Machine Learning
Druh dokumentu:	Working Paper
Popis:	Gradient-based optimization methods for hyperparameter tuning guarantee theoretical convergence to stationary solutions when for fixed upper-level variable values, the lower level of the bilevel program is strongly convex (LLSC) and smooth (LLS). This condition is not satisfied for bilevel programs arising from tuning hyperparameters in many machine learning algorithms. In this work, we develop a sequentially convergent Value Function based Difference-of-Convex Algorithm with inexactness (VF-iDCA). We show that this algorithm achieves stationary solutions without LLSC and LLS assumptions for bilevel programs from a broad class of hyperparameter tuning applications. Our extensive experiments confirm our theoretical findings and show that the proposed VF-iDCA yields superior performance when applied to tune hyperparameters. Comment: 19 pages
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2206.05976 Zobrazit plný text záznamu View this record from Arxiv