C2FS: An Algorithm for Feature Selection in Cascade Neural Networks

Wrapper-based feature selection is attractive because wrapper methods are able to optimize the features they select to the specific learning algorithm. Unfortunately, wrapper methods are prohibitively expensive to use with neural nets. We present an internal wrapper feature selection method for Cascade Correlation (C2) nets called C2FS that is 2-3 orders of magnitude faster than external wrapper feature selection. This new internal wrapper feature selection method selects features at the same time hidden units are being added to the growing C2 net architecture. Experiments with five test problems show that C2FS feature selection usually improves accuracy and squared error while dramatically reducing the number of features needed for good performance. Comparison to feature selection via an information theoretic ordering on features (gain ratio) shows that C2FS usually yields better performance and always uses substantially fewer features.

[1]  Yann LeCun,et al.  Optimal Brain Damage , 1989, NIPS.

[2]  Tom Heskes,et al.  Partial Retraining: A New Approach to Input Relevance Determination , 1999, Int. J. Neural Syst..

[3]  S. Griffis EDITOR , 1997, Journal of Navigation.

[4]  Ron Kohavi,et al.  Wrappers for Feature Subset Selection , 1997, Artif. Intell..

[5]  Christian Lebiere,et al.  The Cascade-Correlation Learning Architecture , 1989, NIPS.

[6]  Huan Liu,et al.  Improving backpropagation learning with feature selection , 1996, Applied Intelligence.

[7]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[8]  Rich Caruana,et al.  Ensemble selection from libraries of models , 2004, ICML.

[9]  JoachimsThorsten,et al.  KDD-Cup 2004 , 2004 .

[10]  J. R. Quinlan Induction of decision trees , 2004, Machine Learning.

[11]  Vitaly Schetinin A Learning Algorithm for Evolving Cascade Neural Networks , 2004, Neural Processing Letters.

[12]  Patrick Gallinari,et al.  FEATURE SELECTION WITH NEURAL NETWORKS , 1999 .

[13]  Nils J. Nilsson,et al.  Artificial Intelligence , 1974, IFIP Congress.