{
  "id": "2008.03501",
  "version": "v1",
  "published": "2020-08-08T11:55:24.000Z",
  "updated": "2020-08-08T11:55:24.000Z",
  "title": "Why to \"grow\" and \"harvest\" deep learning models?",
  "authors": [
    "Ilona Kulikovskikh",
    "Tarzan Legović"
  ],
  "categories": [
    "cs.LG",
    "cs.NE",
    "stat.ML"
  ],
  "abstract": "Current expectations from training deep learning models with gradient-based methods include: 1) transparency; 2) high convergence rates; 3) high inductive biases. While the state-of-art methods with adaptive learning rate schedules are fast, they still fail to meet the other two requirements. We suggest reconsidering neural network models in terms of single-species population dynamics where adaptation comes naturally from open-ended processes of \"growth\" and \"harvesting\". We show that the stochastic gradient descent (SGD) with two balanced pre-defined values of per capita growth and harvesting rates outperform the most common adaptive gradient methods in all of the three requirements.",
  "revisions": [
    {
      "version": "v1",
      "updated": "2020-08-08T11:55:24.000Z"
    }
  ],
  "analyses": {
    "keywords": [
      "deep learning models",
      "high convergence rates",
      "stochastic gradient descent",
      "single-species population dynamics",
      "reconsidering neural network models"
    ],
    "note": {
      "typesetting": "TeX",
      "pages": 0,
      "language": "en",
      "license": "arXiv",
      "status": "editable"
    }
  }
}