Letter: On non-iterative learning algorithms with closed-form solution

Abstract This letter discusses non-iterative learning methods with closed-form solution such as the kernel ridge regression and randomization based single hidden layer feedforward neural networks like random vector functional link (RVFL). Similarities and differences between these methods are also discussed. Irrelevance of kernel-trick for randomized neural networks is explained. The need for dual formulation or constrained optimization formulation for kernel methods and RVFL is distinguished. Finally, the articles in this special issue focusing on non-iterative learning methods with closed-form solution are summarized. A common conclusion in these articles is that the RVFL developed in the early 1990s outperforms the extreme learning machines (ELM). This conclusion is consistent with the earlier findings [1] , [2] , [3] that the direct links enhance the performance of the RVFL.

[1]  Bijaya K. Panigrahi,et al.  Indian summer monsoon rainfall prediction: A comparison of iterative and non-iterative approaches , 2018, Appl. Soft Comput..

[2]  Bernhard Schölkopf,et al.  Kernel Principal Component Analysis , 1997, ICANN.

[3]  Le Zhang,et al.  An ensemble of decision trees with random vector functional link networks for multi-class classification , 2017, Appl. Soft Comput..

[4]  Hubert A.B. Te Braake,et al.  Random activation weight neural net (RAWN) for fast non-iterative training. , 1995 .

[5]  Robert P. W. Duin,et al.  Feedforward neural networks with random weights , 1992, Proceedings., 11th IAPR International Conference on Pattern Recognition. Vol.II. Conference B: Pattern Recognition Methodology and Systems.

[6]  Ping Guo,et al.  A Vest of the Pseudoinverse Learning Algorithm , 2018, ArXiv.

[7]  David G. Stork,et al.  Pattern Classification , 1973 .

[8]  Le Zhang,et al.  Visual Tracking With Convolutional Random Vector Functional Link Network , 2017, IEEE Transactions on Cybernetics.

[9]  L. P. Wang,et al.  Comments on "The Extreme Learning Machine" , 2008, IEEE Trans. Neural Networks.

[10]  B. Scholkopf,et al.  Fisher discriminant analysis with kernels , 1999, Neural Networks for Signal Processing IX: Proceedings of the 1999 IEEE Signal Processing Society Workshop (Cat. No.98TH8468).

[11]  Alexander Gammerman,et al.  Ridge Regression Learning Algorithm in Dual Variables , 1998, ICML.

[12]  Y. Takefuji,et al.  Functional-link net computing: theory, system architecture, and functionalities , 1992, Computer.

[13]  Kurt Hornik,et al.  Approximation capabilities of multilayer feedforward networks , 1991, Neural Networks.

[14]  Gonzalo A. Ruz,et al.  A non-iterative method for pruning hidden neurons in neural networks with random weights , 2018, Appl. Soft Comput..

[15]  Najdan Vukovic,et al.  A comprehensive experimental evaluation of orthogonal polynomial expanded random vector functional link neural networks for regression , 2017, Appl. Soft Comput..

[16]  Leonardo Ramos Rodrigues,et al.  Building selective ensembles of Randomization Based Neural Networks with the successive projections algorithm , 2017, Appl. Soft Comput..

[17]  Le Zhang,et al.  A survey of randomized algorithms for training neural networks , 2016, Inf. Sci..

[18]  Dianhui Wang,et al.  Randomness in neural networks: an overview , 2017, WIREs Data Mining Knowl. Discov..

[19]  Xiaoming Zhang,et al.  Multi-modal kernel ridge regression for social image classification , 2018, Appl. Soft Comput..

[20]  Mathias Quoy,et al.  Structure and Dynamics of Random Recurrent Neural Networks , 2006, Adapt. Behav..

[21]  P. N. Suganthan,et al.  A comprehensive evaluation of random vector functional link networks , 2016, Inf. Sci..

[22]  Ling Tang,et al.  A non-iterative decomposition-ensemble learning paradigm using RVFL network for crude oil price forecasting , 2017, Appl. Soft Comput..

[23]  Jaehoon Lee,et al.  Deep Neural Networks as Gaussian Processes , 2017, ICLR.

[24]  Ponnuthurai N. Suganthan,et al.  Random vector functional link network for short-term electricity load demand forecasting , 2016, Inf. Sci..

[25]  P. N. Suganthan,et al.  Benchmarking Ensemble Classifiers with Novel Co-Trained Kernal Ridge Regression and Random Vector Functional Link Ensembles [Research Frontier] , 2017, IEEE Computational Intelligence Magazine.

[26]  Pradipta Kishore Dash,et al.  Short-term wind speed and wind power prediction using hybrid empirical mode decomposition and kernel ridge regression , 2017, Appl. Soft Comput..