Optimization Solution Functions as Deterministic Policies for Offline Reinforcement Learning

Autor:	Khattar, Vanshaj, Jin, Ming
Rok vydání:	2024
Předmět:	Computer Science - Machine Learning Electrical Engineering and Systems Science - Systems and Control
Zdroj:	American Control Conference 2024
Druh dokumentu:	Working Paper
Popis:	Offline reinforcement learning (RL) is a promising approach for many control applications but faces challenges such as limited data coverage and value function overestimation. In this paper, we propose an implicit actor-critic (iAC) framework that employs optimization solution functions as a deterministic policy (actor) and a monotone function over the optimal value of optimization as a critic. By encoding optimality in the actor policy, we show that the learned policies are robust to the suboptimality of the learned actor parameters via the exponentially decaying sensitivity (EDS) property. We obtain performance guarantees for the proposed iAC framework and show its benefits over general function approximation schemes. Finally, we validate the proposed framework on two real-world applications and show a significant improvement over state-of-the-art (SOTA) offline RL methods. Comment: American Control Conference 2024
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2408.15368 Zobrazit plný text záznamu View this record from Arxiv