Statistical comparison of classifiers through Bayesian hierarchical modelling

Giorgio Corani, Alessio Benavoli, Janez Demšar, Francesca Mangili, Marco Zaffalon

Published 2016-09-28Version 1

We propose a new approach for the statistical comparison of algorithms which have been cross-validated on multiple data sets. It is a Bayesian hierarchical method; it draws inferences on single and on multiple datasets taking into account the mean and the variability of the cross-validation results. It is able to detect equivalent classifiers and to claim significances which have a practical impact. On each data sets it estimates more accurately than the existing methods the difference of accuracy between the two classifiers thanks to shrinkage. Such advantages are demonstrated by simulations on synthetic and real data.