Rethinking Transformers in Solving POMDPs

Autor:	Lu, Chenhao, Shi, Ruizhe, Liu, Yuyao, Hu, Kaizhe, Du, Simon S., Xu, Huazhe
Rok vydání:	2024
Předmět:	Computer Science - Machine Learning Computer Science - Artificial Intelligence
Druh dokumentu:	Working Paper
Popis:	Sequential decision-making algorithms such as reinforcement learning (RL) in real-world scenarios inevitably face environments with partial observability. This paper scrutinizes the effectiveness of a popular architecture, namely Transformers, in Partially Observable Markov Decision Processes (POMDPs) and reveals its theoretical limitations. We establish that regular languages, which Transformers struggle to model, are reducible to POMDPs. This poses a significant challenge for Transformers in learning POMDP-specific inductive biases, due to their lack of inherent recurrence found in other models like RNNs. This paper casts doubt on the prevalent belief in Transformers as sequence models for RL and proposes to introduce a point-wise recurrent structure. The Deep Linear Recurrent Unit (LRU) emerges as a well-suited alternative for Partially Observable RL, with empirical results highlighting the sub-optimal performance of the Transformer and considerable strength of LRU. Comment: Accepted by ICML 2024; references added; typos fixed
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2405.17358 Zobrazit plný text záznamu View this record from Arxiv