Selective negative correlation learning approach to incremental learning

Negative correlation learning (NCL) is a successful approach to constructing neural network ensembles. In batch learning mode, NCL outperforms many other ensemble learning approaches. Recently, NCL has also shown to be a potentially powerful approach to incremental learning, while the advantages of NCL have not yet been fully exploited. In this paper, we propose a selective NCL (SNCL) algorithm for incremental learning. Concretely, every time a new training data set is presented, the previously trained neural network ensemble is cloned. Then the cloned ensemble is trained on the new data set. After that, the new ensemble is combined with the previous ensemble and a selection process is applied to prune the whole ensemble to a fixed size. This paper is an extended version of our preliminary paper on SNCL. Compared to the previous work, this paper presents a deeper investigation into SNCL, considering different objective functions for the selection process and comparing SNCL to other NCL-based incremental learning algorithms on two more real world bioinformatics data sets. Experimental results demonstrate the advantage of SNCL. Further, comparisons between SNCL and other existing incremental learning algorithms, such Learn++ and ARTMAP, are also presented.

[1]  Yong Liu.  Negative correlation learning and evolutionary design of neural network ensembles , 1999 .

[2]  Ludmila I. Kuncheva,et al.  Measures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble Accuracy , 2003, Machine Learning.

[3]  Xin Yao,et al.  Ensemble learning via negative correlation , 1999, Neural Networks.

[4]  Vasant Honavar,et al.  Learn++: an incremental learning algorithm for supervised neural networks , 2001, IEEE Trans. Syst. Man Cybern. Part C.

[5]  Ponnuthurai N. Suganthan,et al.  MegaMotifBase: a database of structural motifs in protein families and superfamilies , 2008, Nucleic Acids Res..

[6]  Manfred M. Fischer,et al.  Neural network ensembles and their application to traffic flow prediction in telecommunications networks , 2001, IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222).

[7]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[8]  Ponnuthurai N. Suganthan,et al.  A machine learning approach for the identification of odorant binding proteins from sequence-derived properties , 2007, BMC Bioinformatics.

[9]  Christophe G. Giraud-Carrier,et al.  A Note on the Utility of Incremental Learning , 2000, AI Commun..

[10]  Xin Yao,et al.  Selective negative correlation learning algorithm for incremental learning , 2008, 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence).

[11]  Xin Yao,et al.  Ensemble Learning Using Multi-Objective Evolutionary Algorithms , 2006, J. Math. Model. Algorithms.

[12]  H. Inoue,et al.  Self-organizing neural grove and its applications , 2005, Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005..

[13]  Stephen Grossberg,et al.  Fuzzy ARTMAP: A neural network architecture for incremental supervised learning of analog multidimensional maps , 1992, IEEE Trans. Neural Networks.

[14]  Xin Yao,et al.  Simultaneous training of negatively correlated neural networks in an ensemble , 1999, IEEE Trans. Syst. Man Cybern. Part B.

[15]  Ramanathan Sowdhamini,et al.  GenDiS: Genomic Distribution of protein structural domain Superfamilies , 2004, Nucleic Acids Res..

[16]  Bernhard Sendhoff,et al.  Pareto-Based Multiobjective Machine Learning: An Overview and Case Studies , 2008, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[17]  Ponnuthurai N. Suganthan,et al.  Identification of structurally conserved residues of proteins in absence of structural homologs using neural network ensemble , 2008, Bioinform..

[18]  Xin Yao,et al.  An analysis of diversity measures , 2006, Machine Learning.

[19]  Xin Yao,et al.  Evolutionary ensembles with negative correlation learning , 2000, IEEE Trans. Evol. Comput..

[20]  R. Durbin,et al.  Pfam: A comprehensive database of protein domain families based on seed alignments , 1997, Proteins.

[21]  Gavin Brown,et al.  Diversity in neural network ensembles , 2004 .

[22]  Zbigniew Michalewicz,et al.  Genetic Algorithms + Data Structures = Evolution Programs , 1996, Springer Berlin Heidelberg.

[23]  David D. Denison,et al.  Nonlinear estimation and classification , 2003 .

[24]  Xin Yao,et al.  Making use of population information in evolutionary artificial neural networks , 1998, IEEE Trans. Syst. Man Cybern. Part B.

[25]  Yaochu Jin,et al.  Pareto-based Multi-Objective Machine Learning , 2007, 7th International Conference on Hybrid Intelligent Systems (HIS 2007).

[26]  Nikola K. Kasabov,et al.  A Preliminary Study on Negative Correlation Learning via Correlation-Corrected Data (NCCD) , 2005, Neural Processing Letters.

[27]  Nikola K. Kasabov,et al.  Evolving fuzzy neural networks for supervised/unsupervised online knowledge-based learning , 2001, IEEE Trans. Syst. Man Cybern. Part B.

[28]  Robert E. Schapire,et al.  The Boosting Approach to Machine Learning An Overview , 2003 .

[29]  Colin R. Reeves,et al.  Evolutionary computation: a unified approach , 2007, Genetic Programming and Evolvable Machines.

[30]  Wei Tang,et al.  Ensembling neural networks: Many could be better than all , 2002, Artif. Intell..

[31]  John A. Bullinaria,et al.  Evolving improved incremental learning schemes for neural network systems , 2005, 2005 IEEE Congress on Evolutionary Computation.

[32]  Hussein A. Abbass,et al.  An evolutionary artificial neural networks approach for breast cancer diagnosis , 2002, Artif. Intell. Medicine.

[33]  Stephen Grossberg,et al.  ARTMAP: supervised real-time learning and classification of nonstationary data by a self-organizing neural network , 1991, [1991 Proceedings] IEEE Conference on Neural Networks for Ocean Engineering.

[34]  Xin Yao,et al.  Negative correlation in incremental learning , 2007, Natural Computing.