A comprehensive evaluation of random vector functional link networks

With randomly generated weights between input and hidden layers, a random vector functional link network is a universal approximator for continuous functions on compact sets with fast learning property. Though it was proposed two decades ago, the classification ability of this family of networks has not been fully investigated yet. Through a very comprehensive evaluation by using 121 UCI datasets, the effect of bias in the output layer, direct links from the input layer to the output layer and type of activation functions in the hidden layer, scaling of parameter randomization as well as the solution procedure for the output weights are investigated in this work. Surprisingly, we found that the direct link plays an important performance enhancing role in RVFL, while the bias term in the output neuron had no significant effect. The ridge regression based closed-form solution was better than those with Moore-Penrose pseudoinverse. Instead of using a uniform randomization in - 1,+1 for all datasets, tuning the scaling of the uniform randomization range for each dataset enhances the overall performance. Six commonly used activation functions were investigated in this work and we found that hardlim and sign activation functions degenerate the overall performance. These basic conclusions can serve as general guidelines for designing RVFL networks based classifiers.

[1]  Tommy W. S. Chow,et al.  Comments on "Stochastic choice of basis functions in adaptive function approximation and the functional-link net" [and reply] , 1997, IEEE Trans. Neural Networks.

[2]  Peter L. Bartlett,et al.  The Sample Complexity of Pattern Classification with Neural Networks: The Size of the Weights is More Important than the Size of the Network , 1998, IEEE Trans. Inf. Theory.

[3]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[4]  Yoh-Han Pao,et al.  Unconstrained word-based approach for off-line script recognition using density-based random-vector functional-link net , 2000, Neurocomputing.

[5]  Janez Demsar,et al.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[6]  Allan Pinkus,et al.  Multilayer Feedforward Networks with a Non-Polynomial Activation Function Can Approximate Any Function , 1991, Neural Networks.

[7]  Dianhui Wang,et al.  Fast decorrelated neural network ensembles with random weights , 2014, Inf. Sci..

[8]  Senén Barro,et al.  Do we need hundreds of classifiers to solve real world classification problems? , 2014, J. Mach. Learn. Res..

[9]  Yoh-Han Pao,et al.  Stochastic choice of basis functions in adaptive function approximation and the functional-link net , 1995, IEEE Trans. Neural Networks.

[10]  Tianyou Chai,et al.  Multisource Data Ensemble Modeling for Clinker Free Lime Content Estimate in Rotary Kiln Sintering Processes , 2015, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[11]  S. R. LeClair,et al.  Intelligent rate control for MPEG-4 coders , 2000 .

[12]  Umesh V. Vazirani,et al.  An Introduction to Computational Learning Theory , 1994 .

[13]  Dejan J. Sobajic,et al.  Neural-net computing and the intelligent control of systems , 1992 .

[14]  David J. Sheskin,et al.  Handbook of Parametric and Nonparametric Statistical Procedures , 1997 .

[15]  Brian D. Ripley,et al.  Pattern Recognition and Neural Networks , 1996 .

[16]  C. L. Philip Chen,et al.  A rapid learning and dynamic stepwise updating algorithm for flat neural networks and the application to time-series prediction , 1999, IEEE Trans. Syst. Man Cybern. Part B.

[17]  Dirk Husmeier,et al.  Modeling Conditional Probabilities with Committees of RVFL Networks , 1997, ICANN.

[18]  Jooyoung Park,et al.  Universal Approximation Using Radial-Basis-Function Networks , 1991, Neural Computation.

[19]  Sung-Bae Cho,et al.  A comprehensive survey on functional link neural networks and an adaptive PSO–BP learning for CFLNN , 2010, Neural Computing and Applications.

[20]  Dirk Husmeier,et al.  Neural Networks for Predicting Conditional Probability Densities: Improved Training Scheme Combining EM and RVFL , 1998, Neural Networks.

[21]  Yoh-Han Pao,et al.  Adaptive pattern recognition and neural networks , 1989 .

[22]  L. Breiman Arcing classifier (with discussion and a rejoinder by the author) , 1998 .

[23]  Robert P. W. Duin,et al.  Feedforward neural networks with random weights , 1992, Proceedings., 11th IAPR International Conference on Pattern Recognition. Vol.II. Conference B: Pattern Recognition Methodology and Systems.

[24]  Yoh-Han Pao,et al.  The functional link net and learning optimal control , 1995, Neurocomputing.

[25]  Lawrence D. Jackel,et al.  Handwritten Digit Recognition with a Back-Propagation Network , 1989, NIPS.

[26]  Okan K. Ersoy,et al.  A statistical self-organizing learning system for remote sensing classification , 2005, IEEE Transactions on Geoscience and Remote Sensing.

[27]  R. E. Lee,et al.  Distribution-free multiple comparisons between successive treatments , 1995 .

[28]  Dianhui Wang,et al.  Distributed learning for Random Vector Functional-Link networks , 2015, Inf. Sci..

[29]  Dejan J. Sobajic,et al.  Learning and generalization characteristics of the random vector Functional-link net , 1994, Neurocomputing.

[30]  Alexander Gammerman,et al.  Ridge Regression Learning Algorithm in Dual Variables , 1998, ICML.

[31]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[32]  Benjamin Recht,et al.  Random Features for Large-Scale Kernel Machines , 2007, NIPS.

[33]  Isabelle Guyon,et al.  Neural Network Recognizer for Hand-Written Zip Code Digits , 1988, NIPS.

[34]  Shan Juan Xie,et al.  A High Accuracy Pedestrian Detection System Combining a Cascade AdaBoost Detector and Random Vector Functional-Link Net , 2014, TheScientificWorldJournal.

[35]  Emilio Soria-Olivas,et al.  Hardware implementation methods in Random Vector Functional-Link Networks , 2013, Applied Intelligence.

[36]  Svetha Venkatesh,et al.  Face Recognition Using Kernel Ridge Regression , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[37]  Ivan Tyukin,et al.  Feasibility of random basis function approximators for modeling and control , 2009, 2009 IEEE Control Applications, (CCA) & Intelligent Control, (ISIC).

[38]  M. Friedman The Use of Ranks to Avoid the Assumption of Normality Implicit in the Analysis of Variance , 1937 .

[39]  M. Friedman A Comparison of Alternative Tests of Significance for the Problem of $m$ Rankings , 1940 .

[40]  Shan Juan Xie,et al.  Random vector functional-link net based pedestrian detection using multi-feature combination , 2013, 2013 6th International Congress on Image and Signal Processing (CISP).