Offline model-based reinforcement learning with causal structured world models.

Autor: Zhu, Zhengmao, Tian, Honglong, Chen, Xionghui, Zhang, Kun, Yu, Yang
Zdroj: Frontiers of Computer Science; Apr2025, Vol. 19 Issue 4, p1-14, 14p
Abstrakt: Model-based methods have recently been shown promising for offline reinforcement learning (RL), which aims at learning good policies from historical data without interacting with the environment. Previous model-based offline RL methods employ a straightforward prediction method that maps the states and actions directly to the next-step states. However, such a prediction method tends to capture spurious relations caused by the sampling policy preference behind the offline data. It is sensible that the environment model should focus on causal influences, which can facilitate learning an effective policy that can generalize well to unseen states. In this paper, we first provide theoretical results that causal environment models can outperform plain environment models in offline RL by incorporating the causal structure into the generalization error bound. We also propose a practical algorithm, oFfline mOdel-based reinforcement learning with CaUsal Structured World Models (FOCUS), to illustrate the feasibility of learning and leveraging causal structure in offline RL. Experimental results on two benchmarks show that FOCUS reconstructs the underlying causal structure accurately and robustly, and, as a result, outperforms both model-based offline RL algorithms and causal model-based offline RL algorithms. [ABSTRACT FROM AUTHOR]
Databáze: Complementary Index
Externí odkaz: