{
  "id": "math/0211020",
  "version": "v2",
  "published": "2002-11-01T17:21:52.000Z",
  "updated": "2004-11-17T10:26:58.000Z",
  "title": "Entropy and the Law of Small Numbers",
  "authors": [
    "Ioannis Kontoyiannis",
    "Peter Harremoes",
    "Oliver Johnson"
  ],
  "comment": "15 pages. To appear, IEEE Trans Inform Theory",
  "journal": "IEEE Transactions on Information Theory, Vol 51/2, 2005, pages 466-472",
  "doi": "10.1109/TIT.2004.840861",
  "categories": [
    "math.PR"
  ],
  "abstract": "Two new information-theoretic methods are introduced for establishing Poisson approximation inequalities. First, using only elementary information-theoretic techniques it is shown that, when $S_n=\\sum_{i=1}^nX_i$ is the sum of the (possibly dependent) binary random variables $X_1,X_2,...,X_n$, with $E(X_i)=p_i$ and $E(S_n)=\\la$, then \\ben D(P_{S_n}\\|\\Pol)\\leq \\sum_{i=1}^n p_i^2 + \\Big[\\sum_{i=1}^nH(X_i) - H(X_1,X_2,..., X_n)\\Big], \\een where $D(P_{S_n}\\|{Po}(\\la))$ is the relative entropy between the distribution of $S_n$ and the Poisson($\\la$) distribution. The first term in this bound measures the individual smallness of the $X_i$ and the second term measures their dependence. A general method is outlined for obtaining corresponding bounds when approximating the distribution of a sum of general discrete random variables by an infinitely divisible distribution. Second, in the particular case when the $X_i$ are independent, the following sharper bound is established, \\ben D(P_{S_n}\\|\\Pol)\\leq \\frac{1}{\\lambda} \\sum_{i=1}^n \\frac{p_i^3}{1-p_i}, % \\label{eq:abs2} \\een and it is also generalized to the case when the $X_i$ are general integer-valued random variables. Its proof is based on the derivation of a subadditivity property for a new discrete version of the Fisher information, and uses a recent logarithmic Sobolev inequality for the Poisson distribution.",
  "revisions": [
    {
      "version": "v2",
      "updated": "2004-11-17T10:26:58.000Z"
    }
  ],
  "analyses": {
    "keywords": [
      "small numbers",
      "distribution",
      "general discrete random variables",
      "elementary information-theoretic techniques",
      "binary random variables"
    ],
    "tags": [
      "journal article"
    ],
    "note": {
      "typesetting": "TeX",
      "pages": 15,
      "language": "en",
      "license": "arXiv",
      "status": "editable",
      "adsabs": "2002math.....11020K"
    }
  }
}