Investigation of the CasCor Family of Learning Algorithms

Six learning algorithms are investigated and compared empirically. All of them are based on variants of the candidate training idea of the Cascade Correlation method. The comparison was performed using 42 different datasets from the PROBEN1 benchmark collection. The results indicate: (1) for these problems it is slightly better not to cascade the hidden units; (2) error minimization candidate training is better than covariance maximization for regression problems but may be a little worse for classification problems; (3) for most learning tasks, considering validation set errors during the selection of the best candidate will not lead to improved networks, but for a few tasks it will. Copyright 1997 Elsevier Science Ltd.

[1]  Christian Lebiere,et al.  The Cascade-Correlation Learning Architecture , 1989, NIPS.

[2]  Stephen Jose Hanson,et al.  Meiosis Networks , 1989, NIPS.

[3]  T. Ash,et al.  Dynamic node creation in backpropagation networks , 1989, International 1989 Joint Conference on Neural Networks.

[4]  Martin A. Riedmiller,et al.  A direct adaptive method for faster backpropagation learning: the RPROP algorithm , 1993, IEEE International Conference on Neural Networks.

[5]  Russell Reed,et al.  Pruning algorithms-a survey , 1993, IEEE Trans. Neural Networks.

[6]  Helge Ritter,et al.  Cascade network architectures , 1992, [Proceedings 1992] IJCNN International Joint Conference on Neural Networks.

[7]  E. Littmann.  Generalization Abilities of Cascade Network Architectures , 1992 .

[8]  Uwe Krey,et al.  Fast generating algorithm for a general three-layer perceptron , 1992, Neural Networks.

[9]  Yann LeCun,et al.  Optimal Brain Damage , 1989, NIPS.

[10]  John E. Moody,et al.  Fast Pruning Using Principal Components , 1993, NIPS.

[11]  Julian Morris,et al.  A procedure for determining the topology of multilayer feedforward neural networks , 1994, Neural Networks.

[12]  Marcus Frean,et al.  The Upstart Algorithm: A Method for Constructing and Training Feedforward Neural Networks , 1990, Neural Computation.

[13]  Michael I. Jordan,et al.  Advances in Neural Information Processing Systems 30 , 1995 .

[14]  Bernd Fritzke,et al.  Growing cell structures--A self-organizing network for unsupervised and supervised learning , 1994, Neural Networks.

[15]  Geoffrey E. Hinton,et al.  Simplifying Neural Networks by Soft Weight-Sharing , 1992, Neural Computation.

[16]  J. Nadal,et al.  Learning in feedforward layered networks: the tiling algorithm , 1989 .

[17]  Ferdinand Hergert,et al.  Improving model selection by nonconvergent methods , 1993, Neural Networks.

[18]  John M. Zelle,et al.  Growing layers of perceptrons: introducing the Extentron algorithm , 1992, [Proceedings 1992] IJCNN International Joint Conference on Neural Networks.

[19]  Babak Hassibi,et al.  Second Order Derivatives for Network Pruning: Optimal Brain Surgeon , 1992, NIPS.

[20]  Elie Bienenstock,et al.  Neural Networks and the Bias/Variance Dilemma , 1992, Neural Computation.

[21]  Anders Krogh,et al.  A Simple Weight Decay Can Improve Generalization , 1991, NIPS.

[22]  Helge J. Ritter,et al.  Generalization Abilities of Cascade Network Architecture , 1992, NIPS.

[23]  Dit-Yan Yeung.  A Neural Network Approach to Constructive Induction , 1991, ML.

[24]  Lutz Prechelt,et al.  PROBEN 1 - a set of benchmarks and benchmarking rules for neural network training algorithms , 1994 .