Variance analysis of sensitivity information for pruning multilayer feedforward neural networks

This paper presents an algorithm for pruning feedforward neural network architectures using sensitivity analysis. Sensitivity Analysis is used to quantify the relevance of input and hidden units. A new statistical pruning heuristic is proposed, based on the variance analysis, to decide which units to prune. Results are presented to show that the pruning algorithm correctly prunes irrelevant input and hidden units.

[1]  Jacek M. Zurada,et al.  Perturbation method for deleting redundant inputs of perceptron networks , 1997, Neurocomputing.

[2]  Roberto Battiti,et al.  First- and Second-Order Methods for Learning: Between Steepest Descent and Newton's Method , 1992, Neural Computation.

[3]  Tiong-Hwee Goh.  Semantic extraction using neural network modelling and sensitivity analysis , 1993, Proceedings of 1993 International Conference on Neural Networks (IJCNN-93-Nagoya, Japan).

[4]  Andries P. Engelbrecht,et al.  A sensitivity analysis algorithm for pruning feedforward neural networks , 1996, Proceedings of International Conference on Neural Networks (ICNN'96).

[5]  Andries Petrus Engelbrecht,et al.  Rule Improvement Through Decision Boundary Detection Using Sensitivity Analysis , 1999, IWANN.

[6]  M. Koda.  Neural network learning based on stochastic sensitivity analysis , 1997, IEEE Trans. Syst. Man Cybern. Part B.

[7]  Andries Petrus Engelbrecht,et al.  Determining the Significance of Input Parameters using Sensitivity Analysis , 1995, IWANN.

[8]  Kurt Hornik,et al.  Universal approximation of an unknown mapping and its derivatives using multilayer feedforward networks , 1990, Neural Networks.

[9]  Thomas Czernichow.  Architecture Selection through Statistical Sensitivity Analysis , 1996, ICANN.

[10]  R. E. Uhrig,et al.  Sensitivity analysis and applications to nuclear power plant , 1992, [Proceedings 1992] IJCNN International Joint Conference on Neural Networks.

[11]  Michael C. Mozer,et al.  Skeletonization: A Technique for Trimming the Fat from a Network via Relevance Assessment , 1988, NIPS.

[12]  Yann LeCun,et al.  Improving the convergence of back-propagation learning with second-order methods , 1989 .

[13]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[14]  Andries Petrus Engelbrecht,et al.  Incremental learning using sensitivity analysis , 1999, IJCNN'99. International Joint Conference on Neural Networks. Proceedings (Cat. No.99CH36339).

[15]  Babak Hassibi,et al.  Second Order Derivatives for Network Pruning: Optimal Brain Surgeon , 1992, NIPS.

[16]  Ken-ichi Funahashi,et al.  On the approximate realization of continuous mappings by neural networks , 1989, Neural Networks.

[17]  Ilan Modai,et al.  Sensitivity profile of 41 psychiatric parameters determined by neural network in relation to 8-week outcome , 1995 .

[18]  Vincenzo Piuri,et al.  Sensitivity to errors in artificial neural networks: a behavioral approach , 1995 .

[19]  Andries P. Engelbrecht,et al.  Selective learning using sensitivity analysis , 1998, 1998 IEEE International Joint Conference on Neural Networks Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98CH36227).

[20]  Yann LeCun,et al.  Optimal Brain Damage , 1989, NIPS.

[21]  Ferdinand Hergert,et al.  Improving model selection by nonconvergent methods , 1993, Neural Networks.

[22]  Chong-Ho Choi,et al.  Sensitivity analysis of multilayer perceptron with differentiable activation functions , 1992, IEEE Trans. Neural Networks.

[23]  Halbert White,et al.  On learning the derivatives of an unknown mapping with multilayer feedforward networks , 1992, Neural Networks.

[24]  Gregory J. Wolff,et al.  Optimal Brain Surgeon: Extensions and performance comparisons , 1993, NIPS 1993.

[25]  Andries P. Engelbrecht,et al.  Optimizing the number of hidden nodes of a feedforward artificial neural network , 1998, 1998 IEEE International Joint Conference on Neural Networks Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98CH36227).

[26]  Patrick Gallinari,et al.  Variable selection with neural networks , 1996, Neurocomputing.

[27]  L. Fu,et al.  Sensitivity analysis for input vector in multilayer feedforward neural networks , 1993, IEEE International Conference on Neural Networks.

[28]  Sang-Hoon Oh,et al.  Sensitivity analysis of single hidden-layer neural networks with threshold functions , 1995, IEEE Trans. Neural Networks.