Combinatorial Designs for Deep Learning

Shoko Chisaki, Ryoh Fuji-Hara, Nobuko Miyamoto

Published 2018-09-22Version 1

Deep learning is a multi-layer neural network. It can be regarded as a chain of complete bipartite graphs. The nodes of the first partite is the input layer and the last is the output layer. The edges of a bipartite graph function as weights which are represented as a matrix. The values of i-th partite are computed by multiplication of the weight matrix and values of (i-1)-th partite. Using mass training and teacher data, the weight parameters are estimated little by little. Overfitting (or Overlearning) refers to a model that models the 'training data' too well. It then becomes difficult for the model to generalize to new data which were not in the training set. The most popular method to avoid overfitting is called dropout. Dropout deletes a random sample of activations (nodes) to zero during the training process. A random sample of nodes cause more irregular frequency of dropout edges. We propose a combinatorial design on dropout nodes from each partite which balances frequency of edges. We analyze and construct such designs in this paper.