A Novel Pruning Approach for Bagging Ensemble Regression Based on Sparse Representation

This work aims to propose an approach for pruning a bagging ensemble regression (BER) model based on sparse representation, which we call sparse representation pruning (SRP). Firstly, a BER model with a specific number of subensembles should be trained. Then, the BER model is pruned by our sparse representation idea. For this type of regression problems, pruning means to remove the subensembles that do not have a significant effect on prediction of the output. The pruning problem is casted as a sparse representation problem, which will be solved by orthogonal matching pursuit (OMP) algorithm. Experiments show that the pruned BER with only 20% of the initial subensembles has a better generalization compared to a complete BER.

[1]  Geoffrey E. Hinton,et al.  Adaptive Mixtures of Local Experts , 1991, Neural Computation.

[2]  Jun Ma,et al.  Feed-forward neural network training using sparse representation , 2019, Expert Syst. Appl..

[3]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[4]  Gonzalo Martínez-Muñoz,et al.  Pruning in ordered bagging ensembles , 2006, ICML.

[5]  Wei Tang,et al.  Ensembling neural networks: Many could be better than all , 2002, Artif. Intell..

[6]  Abdesselam Bouzerdoum,et al.  A training algorithm for sparse LS-SVM using Compressive Sampling , 2010, 2010 IEEE International Conference on Acoustics, Speech and Signal Processing.

[7]  Wei-Yin Loh,et al.  Classification and regression trees , 2011, WIREs Data Mining Knowl. Discov..

[8]  Daniel Hernández-Lobato,et al.  An Analysis of Ensemble Pruning Techniques Based on Ordered Aggregation , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  Abdesselam Bouzerdoum,et al.  A Neural Network pruning approach based on Compressive Sampling , 2009, 2009 International Joint Conference on Neural Networks.

[10]  Stéphane Mallat,et al.  Matching pursuits with time-frequency dictionaries , 1993, IEEE Trans. Signal Process..

[11]  Michael Elad,et al.  Sparse and Redundant Representations - From Theory to Applications in Signal and Image Processing , 2010 .

[12]  D. J. Newman,et al.  UCI Repository of Machine Learning Database , 1998 .

[13]  P. Bühlmann,et al.  Analyzing Bagging , 2001 .

[14]  Joel A. Tropp,et al.  Signal Recovery From Random Measurements Via Orthogonal Matching Pursuit , 2007, IEEE Transactions on Information Theory.

[15]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[16]  D. Ruppert.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction , 2004 .

[17]  Daniel Hernández-Lobato,et al.  Pruning in Ordered Regression Bagging Ensembles , 2006, The 2006 IEEE International Joint Conference on Neural Network Proceedings.