Kenichi Nakazato
Published 2023-09-14Version 1
Deep neural networks have shown many fruitful applications in this decade. A network can get the generalized function through training with a finite dataset. The degree of generalization is a realization of the proximity scale in the data space. Specifically, the scale is not clear if the dataset is complicated. Here we consider a network for the distribution estimation of the dataset. We show the estimation is unstable and the instability depends on the data density and training duration. We derive the kernel-balanced equation, which gives a short phenomenological description of the solution. The equation tells us the reason for the instability and the mechanism of the scale. The network outputs a local average of the dataset as a prediction and the scale of averaging is determined along the equation. The scale gradually decreases along training and finally results in instability in our case.
Statistical Physics of Deep Neural Networks: Generalization Capability, Beyond the Infinite Width, and Feature Learning
Geometry of energy landscapes and the optimizability of deep neural networks
The jamming transition as a paradigm to understand the loss landscape of deep neural networks
![QR code for arXiv:2309.07367 [cond-mat.dis-nn]](http://oss.arxitics.com/images/favicon.svg)