2024-06-11-han24a.md

title	abstract	layout	series	publisher	issn	id	month	tex_title	firstpage	lastpage	page	order	cycles	bibtex_author	author	date	address	container-title	volume	genre	issued	pdf	extras
On the nonsmooth geometry and neural approximation of the optimal value function of infinite-horizon pendulum swing-up	We revisit the inverted pendulum problem with the goal of understanding and computing the true optimal value function. We start with an observation that the true optimal value function must be nonsmooth (i.e., not globally C1) due to symmetry of the problem. We then give a result that can certify the optimality of a candidate piece-wise C1 value function. Further, for a candidate value function obtained via numerical approximation, we provide a bound of suboptimality based on its Hamilton-Jacobi-Bellman (HJB) equation residuals. Inspired by Holzhüter (2004), we then design an algorithm that solves backwards the Pontryagin’s minimum principle (PMP) ODE from terminal conditions provided by the locally optimal LQR value function. This numerical procedure leads to a piece-wise C1 value function whose nonsmooth region contains periodic spiral lines and smooth regions attain HJB residuals about $10^{-4}$, hence certified to be the optimal value function up to minor numerical inaccuracies. This optimal value function checks the power of optimality: (i) it sits above a polynomial lower bound; (ii) its induced controller globally swings up and stabilizes the pendulum, and (iii) attains lower trajectory cost than baseline methods such as energy shaping, model predictive control (MPC), and proximal policy optimization (with MPC attaining almost the same cost). We conclude by distilling the optimal value function into a simple neural network.	inproceedings	Proceedings of Machine Learning Research	PMLR	2640-3498	han24a	0	On the nonsmooth geometry and neural approximation of the optimal value function of infinite-horizon pendulum swing-up	654	666	654-666	654	false	Han, Haoyu and Yang, Heng	given family Haoyu Han given family Heng Yang	2024-06-11		Proceedings of the 6th Annual Learning for Dynamics & Control Conference	242	inproceedings	date-parts 2024 6 11	https://proceedings.mlr.press/v242/han24a/han24a.pdf