Computational evidence for hierarchically structured reinforcement learning in humans
| Autor: | Maria K. Eckstein, Anne G.E. Collins |
|---|---|
| Rok vydání: | 2020 |
| Předmět: |
computational modeling
Adult Male reinforcement learning Adolescent Computer science Bayesian probability Context (language use) Bioengineering hierarchy task-sets Models Psychological Bayesian inference Basic Behavioral and Social Science Field (computer science) Machine Learning Young Adult Models Clinical Research Behavioral and Social Science Reinforcement learning Psychology Humans Structure (mathematical logic) Multidisciplinary Hierarchy (mathematics) business.industry Flexibility (personality) Bayes Theorem Reinforcement Colloquium on Brain Produces Mind by Modeling structure learning Psychological Female Artificial intelligence business Reinforcement Psychology Learning Curve |
| Zdroj: | Proc Natl Acad Sci U S A Proceedings of the National Academy of Sciences of the United States of America, vol 117, iss 47 |
| ISSN: | 1091-6490 |
| Popis: | Humans have the fascinating ability to achieve goals in a complex and constantly changing world, still surpassing modern machine-learning algorithms in terms of flexibility and learning speed. It is generally accepted that a crucial factor for this ability is the use of abstract, hierarchical representations, which employ structure in the environment to guide learning and decision making. Nevertheless, how we create and use these hierarchical representations is poorly understood. This study presents evidence that human behavior can be characterized as hierarchical reinforcement learning (RL). We designed an experiment to test specific predictions of hierarchical RL using a series of subtasks in the realm of context-based learning and observed several behavioral markers of hierarchical RL, such as asymmetric switch costs between changes in higher-level versus lower-level features, faster learning in higher-valued compared to lower-valued contexts, and preference for higher-valued compared to lower-valued contexts. We replicated these results across three independent samples. We simulated three models—a classic RL, a hierarchical RL, and a hierarchical Bayesian model—and compared their behavior to human results. While the flat RL model captured some aspects of participants’ sensitivity to outcome values, and the hierarchical Bayesian model captured some markers of transfer, only hierarchical RL accounted for all patterns observed in human behavior. This work shows that hierarchical RL, a biologically inspired and computationally simple algorithm, can capture human behavior in complex, hierarchical environments and opens the avenue for future research in this field. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|