{
  "id": "1810.11953",
  "version": "v1",
  "published": "2018-10-29T04:50:18.000Z",
  "updated": "2018-10-29T04:50:18.000Z",
  "title": "Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift",
  "authors": [
    "Stephan Rabanser",
    "Stephan Günnemann",
    "Zachary C. Lipton"
  ],
  "comment": "Submitted to the NIPS 2018 Workshop on Security in Machine Learning",
  "categories": [
    "stat.ML",
    "cs.LG"
  ],
  "abstract": "We might hope that when faced with unexpected inputs, well-designed software systems would fire off warnings. Machine learning (ML) systems, however, which depend strongly on properties of their inputs (e.g. the i.i.d. assumption), tend to fail silently. This paper explores the problem of building ML systems that fail loudly, investigating methods for detecting dataset shift and identifying exemplars that most typify the shift. We focus on several datasets and various perturbations to both covariates and label distributions with varying magnitudes and fractions of data affected. Interestingly, we show that while classifier-based methods perform well in high-data settings, they perform poorly in low-data settings. Moreover, across the dataset shifts that we explore, a two-sample-testing-based approach, using pretrained classifiers for dimensionality reduction performs best.",
  "revisions": [
    {
      "version": "v1",
      "updated": "2018-10-29T04:50:18.000Z"
    }
  ],
  "analyses": {
    "keywords": [
      "detecting dataset shift",
      "empirical study",
      "dimensionality reduction performs best",
      "building ml systems",
      "software systems"
    ],
    "note": {
      "typesetting": "TeX",
      "pages": 0,
      "language": "en",
      "license": "arXiv",
      "status": "editable"
    }
  }
}