Variational Dynamic Programming for Stochastic Optimal Control

Autor: Lambert, Marc, Bach, Francis, Bonnabel, Silvère
Rok vydání: 2024
Předmět:
Zdroj: 2024 Conference on Decision and Control, Dec 2024, Milano, Italy
Druh dokumentu: Working Paper
Popis: We consider the problem of stochastic optimal control, where the state-feedback control policies take the form of a probability distribution and where a penalty on the entropy is added. By viewing the cost function as a Kullback- Leibler (KL) divergence between two joint distributions, we bring the tools from variational inference to bear on our optimal control problem. This allows for deriving a dynamic programming principle, where the value function is defined as a KL divergence again. We then resort to Gaussian distributions to approximate the control policies and apply the theory to control affine nonlinear systems with quadratic costs. This results in closed-form recursive updates, which generalize LQR control and the backward Riccati equation. We illustrate this novel method on the simple problem of stabilizing an inverted pendulum.
Databáze: arXiv
načítá se...