Predicting Generalization in Deep Learning via Local Measures of Distortion

Abhejit Rajagopal, Vamshi C. Madala, Shivkumar Chandrasekaran, Peder E. Z. Larson

Published 2020-12-13, updated 2020-12-16Version 2

We study generalization in deep learning by appealing to complexity measures originally developed in approximation and information theory. While these concepts are challenged by the high-dimensional and data-defined nature of deep learning, we show that simple vector quantization approaches such as PCA, GMMs, and SVMs capture their spirit when applied layer-wise to deep extracted features giving rise to relatively inexpensive complexity measures that correlate well with generalization performance. We discuss our results in 2020 NeurIPS PGDL challenge.