Selection dynamics for deep neural networks

Hailiang Liu, Peter Markowich

Published 2019-05-22Version 1

This paper introduces the mathematical formulation of deep residual neural networks as a PDE optimal control problem. We study the wellposedness, the large time solution behavior, and the characterization of the steady states for the forward problem. Several useful time-uniform estimates and stability/instability conditions are presented. We state and prove optimality conditions for the inverse deep learning problem, using the Hamilton-Jacobi-Bellmann equation and the Pontryagin maximum principle. This serves to establish a mathematical foundation for investigating the algorithmic and theoretical connections between optimal control and deep learning.