Learning in Evolutive Neural Architectures: An Ill-Posed Problem?

Basically, evolutive architectures are networks able to be modified by adding or pruning neurons or connections. In the paper, by a synthesis a various works, we point out that evolutive architectures involve a lot of tricks because of indeterminacy of solutions and suboptimality, which are characteristic of ill-posed problems. We also emphasize on interest of stopping criteria, essential to control adding as well as pruning procedure and to avoid overfitting. Finally, we suggest another formulation of learning in evolutive architectures based on more realistic “hardware” constraints.

[1]  Pierre Comon,et al.  Supervised Design of Optimal Receivers , 1993 .

[2]  Shang-Liang Chen,et al.  Orthogonal least squares learning algorithm for radial basis function networks , 1991, IEEE Trans. Neural Networks.

[3]  Marie Cottrell,et al.  Neural modeling for time series: A statistical stepwise method for weight elimination , 1995, IEEE Trans. Neural Networks.

[4]  Babak Hassibi,et al.  Second Order Derivatives for Network Pruning: Optimal Brain Surgeon , 1992, NIPS.

[5]  Christian Jutten,et al.  Incremental Learning with a Stopping Criterion - Experimental Results , 1995, IWANN.

[6]  Peter M. Williams,et al.  Bayesian Regularization and Pruning Using a Laplace Prior , 1995, Neural Computation.

[7]  Christian Jutten,et al.  A comparison of two weight pruning methods , 1994, ESANN.

[8]  Yann LeCun,et al.  Optimal Brain Damage , 1989, NIPS.

[9]  Pierre Comon,et al.  Invited paper: Supervised classification: a probabilistic approach , 1995, The European Symposium on Artificial Neural Networks.

[10]  Jirí Benes,et al.  On neural networks , 1990, Kybernetika.

[11]  E. Parzen.  On Estimation of a Probability Density Function and Mode , 1962 .

[12]  B. Silverman.  Density estimation for statistics and data analysis , 1986 .

[13]  Mark Plutowski,et al.  Selecting concise training sets from clean data , 1993, IEEE Trans. Neural Networks.

[14]  Tomaso A. Poggio,et al.  Regularization Theory and Neural Networks Architectures , 1995, Neural Computation.

[15]  H. Akaike.  Statistical predictor identification , 1970 .

[16]  Helge Ritter,et al.  Cascade network architectures , 1992, [Proceedings 1992] IJCNN International Joint Conference on Neural Networks.

[17]  F. Vallet,et al.  Robustness in Multilayer Perceptrons , 1993, Neural Computation.

[18]  Jocelyn Sietsma,et al.  Creating artificial neural networks that generalize , 1991, Neural Networks.

[19]  Russell Reed,et al.  Pruning algorithms-a survey , 1993, IEEE Trans. Neural Networks.

[20]  D. L. Reilly,et al.  A neural model for category learning , 1982, Biological Cybernetics.

[21]  Christian Lebiere,et al.  The Cascade-Correlation Learning Architecture , 1989, NIPS.

[22]  Geoffrey E. Hinton,et al.  Experiments on Learning by Back Propagation. , 1986 .

[23]  Pierre Comon,et al.  Neural Bayesian Classifier , 1993, IWANN.

[24]  Christian Jutten,et al.  Pruning kernel density estimators , 1995, ESANN.

[25]  Jean-Pierre Nadal,et al.  Neural trees: a new tool for classification , 1990 .

[26]  W. Härdle.  Smoothing Techniques: With Implementation in S , 1991 .

[27]  Donald F. Specht,et al.  Probabilistic neural networks , 1990, Neural Networks.

[28]  Christian Jutten,et al.  Invited paper: Pruning methods: a review , 1995, ESANN.