Model-Based Uncertainty in Value Functions

Autor:	Luis, Carlos E., Bottero, Alessandro G., Vinogradska, Julia, Berkenkamp, Felix, Peters, Jan
Rok vydání:	2023
Předmět:	Computer Science - Machine Learning Computer Science - Artificial Intelligence Statistics - Machine Learning
Druh dokumentu:	Working Paper
Popis:	We consider the problem of quantifying uncertainty over expected cumulative rewards in model-based reinforcement learning. In particular, we focus on characterizing the variance over values induced by a distribution over MDPs. Previous work upper bounds the posterior variance over values by solving a so-called uncertainty Bellman equation, but the over-approximation may result in inefficient exploration. We propose a new uncertainty Bellman equation whose solution converges to the true posterior variance over values and explicitly characterizes the gap in previous work. Moreover, our uncertainty quantification technique is easily integrated into common exploration strategies and scales naturally beyond the tabular setting by using standard deep reinforcement learning architectures. Experiments in difficult exploration tasks, both in tabular and continuous control settings, show that our sharper uncertainty estimates improve sample-efficiency. Comment: AISTATS 2023
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2302.12526 Zobrazit plný text záznamu View this record from Arxiv