When classifier selection meets information theory: A unifying view

Classifier selection aims to reduce the size of an ensemble of classifiers in order to improve its efficiency and classification accuracy. Recently an information-theoretic view was presented for feature selection. It derives a space of possible selection criteria and show that several feature selection criteria in the literature are points within this continuous space. The contribution of this paper is to export this information-theoretic view to solve an open issue in ensemble learning which is classifier selection. We investigated a couple of information-theoretic selection criteria that are used to rank classifiers.

[1]  F. Fleuret Fast Binary Feature Selection with Conditional Mutual Information , 2004, J. Mach. Learn. Res..

[2]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[3]  Wei Tang,et al.  Selective Ensemble of Decision Trees , 2003, RSFDGrC.

[4]  Janez Demsar,et al.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[5]  William J. McGill Multivariate information transmission , 1954, Trans. IRE Prof. Group Inf. Theory.

[6]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[7]  Fuhui Long,et al.  Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy , 2003, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Gavin Brown,et al.  A New Perspective for Information Theoretic Feature Selection , 2009, AISTATS.

[9]  D. J. Newman,et al.  UCI Repository of Machine Learning Database , 1998 .

[10]  Roberto Battiti,et al.  Using mutual information for selecting features in supervised neural net learning , 1994, IEEE Trans. Neural Networks.

[11]  John E. Moody,et al.  Data Visualization and Feature Selection: New Algorithms for Nongaussian Data , 1999, NIPS.

[12]  Thomas G. Dietterich,et al.  Pruning Adaptive Boosting , 1997, ICML.

[13]  Dahua Lin,et al.  Conditional Infomax Learning: An Integrated Framework for Feature Extraction and Fusion , 2006, ECCV.

[14]  Christopher J. Merz,et al.  UCI Repository of Machine Learning Databases , 1996 .