Model-Based Reinforcement Learning via Meta-Policy Optimization

Autor:	Clavera, Ignasi, Rothfuss, Jonas, Schulman, John, Fujita, Yasuhiro, Asfour, Tamim, Abbeel, Pieter
Rok vydání:	2018
Předmět:	Computer Science - Machine Learning Computer Science - Artificial Intelligence Statistics - Machine Learning
Druh dokumentu:	Working Paper
Popis:	Model-based reinforcement learning approaches carry the promise of being data efficient. However, due to challenges in learning dynamics models that sufficiently match the real-world dynamics, they struggle to achieve the same asymptotic performance as model-free methods. We propose Model-Based Meta-Policy-Optimization (MB-MPO), an approach that foregoes the strong reliance on accurate learned dynamics models. Using an ensemble of learned dynamic models, MB-MPO meta-learns a policy that can quickly adapt to any model in the ensemble with one policy gradient step. This steers the meta-policy towards internalizing consistent dynamics predictions among the ensemble while shifting the burden of behaving optimally w.r.t. the model discrepancies towards the adaptation step. Our experiments show that MB-MPO is more robust to model imperfections than previous model-based approaches. Finally, we demonstrate that our approach is able to match the asymptotic performance of model-free methods while requiring significantly less experience. Comment: First 2 authors contributed equally. Accepted for Conference on Robot Learning (CoRL)
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/1809.05214 Zobrazit plný text záznamu View this record from Arxiv