Ratio Divergence Learning Using Target Energy in Restricted Boltzmann Machines: Beyond Kullback--Leibler Divergence Learning

Autor:	Ishida, Yuichi, Ichikawa, Yuma, Dote, Aki, Miyazawa, Toshiyuki, Hukushima, Koji
Rok vydání:	2024
Předmět:	Statistics - Machine Learning Condensed Matter - Disordered Systems and Neural Networks Computer Science - Machine Learning Mathematics - Statistics Theory Statistics - Methodology
Druh dokumentu:	Working Paper
Popis:	We propose ratio divergence (RD) learning for discrete energy-based models, a method that utilizes both training data and a tractable target energy function. We apply RD learning to restricted Boltzmann machines (RBMs), which are a minimal model that satisfies the universal approximation theorem for discrete distributions. RD learning combines the strength of both forward and reverse Kullback-Leibler divergence (KLD) learning, effectively addressing the "notorious" issues of underfitting with the forward KLD and mode-collapse with the reverse KLD. Since the summation of forward and reverse KLD seems to be sufficient to combine the strength of both approaches, we include this learning method as a direct baseline in numerical experiments to evaluate its effectiveness. Numerical experiments demonstrate that RD learning significantly outperforms other learning methods in terms of energy function fitting, mode-covering, and learning stability across various discrete energy-based models. Moreover, the performance gaps between RD learning and the other learning methods become more pronounced as the dimensions of target models increase. Comment: 14 pages, 19 figures
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2409.07679 Zobrazit plný text záznamu View this record from Arxiv