Random Vector Functional Link Neural Network based Ensemble Deep Learning

In this paper, we propose a deep learning framework based on randomized neural network. In particular, inspired by the principles of Random Vector Functional Link (RVFL) network, we present a deep RVFL network (dRVFL) with stacked layers. The parameters of the hidden layers of the dRVFL are randomly generated within a suitable range and kept fixed while the output weights are computed using the closed form solution as in a standard RVFL network. We also propose an ensemble deep network (edRVFL) that can be regarded as a marriage of ensemble learning with deep learning. Unlike traditional ensembling approaches that require training several models independently from scratch, edRVFL is obtained by training a single dRVFL network once. Both dRVFL and edRVFL frameworks are generic and can be used with any RVFL variant. To illustrate this, we integrate the deep learning networks with a recently proposed sparse-pretrained RVFL (SP-RVFL). Extensive experiments on benchmark datasets from diverse domains show the superior performance of our proposed deep RVFL networks.

[1]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[2]  J. Spencer.  Ten lectures on the probabilistic method , 1987 .

[3]  Kevin Barraclough,et al.  I and i , 2001, BMJ : British Medical Journal.

[4]  Abraham J. Wyner,et al.  Modern Neural Networks Generalize on Small Data Sets , 2018, NeurIPS.

[5]  Ming-Wei Chang,et al.  Load Forecasting Using Support Vector Machines: A Study on EUNITE Competition 2001 , 2004, IEEE Transactions on Power Systems.

[6]  Ponnuthurai N. Suganthan,et al.  Ensemble incremental learning Random Vector Functional Link network for short-term electric load forecasting , 2018, Knowl. Based Syst..

[7]  P. N. Suganthan,et al.  An Ensemble of Kernel Ridge Regression for Multi-class Classification , 2017, ICCS.

[8]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Bijaya K. Panigrahi,et al.  Indian summer monsoon rainfall prediction: A comparison of iterative and non-iterative approaches , 2018, Appl. Soft Comput..

[10]  Jürgen Schmidhuber,et al.  Training Very Deep Networks , 2015, NIPS.

[11]  Ponnuthurai Nagaratnam Suganthan,et al.  Empirical Mode Decomposition based ensemble deep learning for load demand time series forecasting , 2017, Appl. Soft Comput..

[12]  Chee Kheong Siew,et al.  Extreme learning machine: Theory and applications , 2006, Neurocomputing.

[13]  A. Timmermann.  Forecast Combinations , 2005 .

[14]  Yusheng Huang,et al.  A new crude oil price forecasting model based on variational mode decomposition , 2021, Knowl. Based Syst..

[15]  P. N. Suganthan,et al.  Benchmarking Ensemble Classifiers with Novel Co-Trained Kernal Ridge Regression and Random Vector Functional Link Ensembles [Research Frontier] , 2017, IEEE Computational Intelligence Magazine.

[16]  P. N. Suganthan,et al.  Stacked Autoencoder Based Deep Random Vector Functional Link Neural Network for Classification , 2019, Appl. Soft Comput..

[17]  Serge J. Belongie,et al.  Residual Networks Behave Like Ensembles of Relatively Shallow Networks , 2016, NIPS.

[18]  Y. Takefuji,et al.  Functional-link net computing: theory, system architecture, and functionalities , 1992, Computer.

[19]  Ping Guo,et al.  A Vest of the Pseudoinverse Learning Algorithm , 2018, ArXiv.

[20]  P. N. Suganthan,et al.  A Comparative Study of Empirical Mode Decomposition-Based Short-Term Wind Speed Forecasting Methods , 2015, IEEE Transactions on Sustainable Energy.

[21]  Ponnuthurai N. Suganthan,et al.  Ensemble Classification and Regression-Recent Developments, Applications and Future Directions [Review Article] , 2016, IEEE Computational Intelligence Magazine.

[22]  Ran Li,et al.  Deep Learning for Household Load Forecasting—A Novel Pooling Deep RNN , 2018, IEEE Transactions on Smart Grid.

[23]  Bernard Widrow,et al.  The No-Prop algorithm: A new learning algorithm for multilayer neural networks , 2013, Neural Networks.

[24]  George Edwards,et al.  A Review of Deep Learning Methods Applied on Load Forecasting , 2017, 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA).

[25]  P. N. Suganthan,et al.  A comprehensive evaluation of random vector functional link networks , 2016, Inf. Sci..

[26]  Okan Duru,et al.  Parsimonious fuzzy time series modelling , 2020, Expert Syst. Appl..

[27]  Ghulam Hafeez,et al.  Electric load forecasting based on deep learning and optimized by heuristic algorithm in smart grid , 2020 .

[28]  J. W. Taylor,et al.  Short-Term Load Forecasting With Exponentially Weighted Methods , 2012, IEEE Transactions on Power Systems.

[29]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[30]  Ponnuthurai N. Suganthan,et al.  On the origins of randomization-based feedforward neural networks , 2021, Appl. Soft Comput..

[31]  Jing Liu,et al.  Time-Series Forecasting Based on High-Order Fuzzy Cognitive Maps and Wavelet Transform , 2018, IEEE Transactions on Fuzzy Systems.

[32]  Maarouf Saad,et al.  An efficient approach for short term load forecasting using artificial neural networks , 2006 .

[33]  Claudio Gallicchio,et al.  Deep reservoir computing: A critical experimental analysis , 2017, Neurocomputing.

[34]  Saeid Nahavandi,et al.  A Novel Evolutionary-Based Deep Convolutional Neural Network Model for Intelligent Load Forecasting , 2021, IEEE Transactions on Industrial Informatics.

[35]  Halbert White,et al.  Chapter 9 Approximate Nonlinear Forecasting Methods , 2006 .

[36]  W. Marsden.  I and J , 2012 .

[37]  Mathias Quoy,et al.  Structure and Dynamics of Random Recurrent Neural Networks , 2006, Adapt. Behav..

[38]  Yoshua Bengio,et al.  Extracting and composing robust features with denoising autoencoders , 2008, ICML '08.

[39]  Ling Tang,et al.  A non-iterative decomposition-ensemble learning paradigm using RVFL network for crude oil price forecasting , 2017, Appl. Soft Comput..

[40]  Martin J. Wainwright,et al.  Divide and conquer kernel ridge regression: a distributed algorithm with minimax optimal rates , 2013, J. Mach. Learn. Res..

[41]  Geoffrey E. Hinton,et al.  Deep Boltzmann Machines , 2009, AISTATS.

[42]  Senén Barro,et al.  Do we need hundreds of classifiers to solve real world classification problems? , 2014, J. Mach. Learn. Res..

[43]  R. Stephenson.  A and V , 1962, The British journal of ophthalmology.

[44]  Gabriel Rilling,et al.  Empirical mode decomposition as a filter bank , 2004, IEEE Signal Processing Letters.

[45]  Le Zhang,et al.  Ensemble deep learning for regression and time series forecasting , 2014, 2014 IEEE Symposium on Computational Intelligence in Ensemble Learning (CIEL).

[46]  Rob J Hyndman,et al.  Another look at measures of forecast accuracy , 2006 .

[47]  Ponnuthurai N. Suganthan,et al.  Letter: On non-iterative learning algorithms with closed-form solution , 2018, Appl. Soft Comput..

[48]  Claudio Gallicchio,et al.  Design of deep echo state networks , 2018, Neural Networks.

[49]  Ponnuthurai N. Suganthan,et al.  Heterogeneous oblique random forest , 2020, Pattern Recognit..

[50]  Muhammad Tariq,et al.  Load Forecasting Through Estimated Parametrized Based Fuzzy Inference System in Smart Grids , 2021, IEEE Transactions on Fuzzy Systems.

[51]  P. Casazza.  THE ART OF FRAME THEORY , 1999, math/9910168.

[52]  Rayan Saab,et al.  Random Vector Functional Link Networks for Function Approximation on Manifolds , 2020, ArXiv.

[53]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[54]  Ponnuthurai N. Suganthan,et al.  Random vector functional link network for short-term electricity load demand forecasting , 2016, Inf. Sci..

[55]  Robert P. W. Duin,et al.  Feedforward neural networks with random weights , 1992, Proceedings., 11th IAPR International Conference on Pattern Recognition. Vol.II. Conference B: Pattern Recognition Methodology and Systems.

[56]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[57]  Kilian Q. Weinberger,et al.  Snapshot Ensembles: Train 1, get M for free , 2017, ICLR.

[58]  Ponnuthurai N. Suganthan,et al.  Oblique random forest ensemble via Least Square Estimation for time series forecasting , 2017, Inf. Sci..

[59]  Mohammad Navid Fekri,et al.  Deep learning for load forecasting with smart meter data: Online Adaptive Recurrent Neural Network , 2021, Applied Energy.

[60]  Hubert A.B. Te Braake,et al.  Random activation weight neural net (RAWN) for fast non-iterative training. , 1995 .

[61]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[62]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[63]  Liang Du,et al.  Robust empirical wavelet fuzzy cognitive map for time series forecasting , 2020, Eng. Appl. Artif. Intell..

[64]  J. Contreras,et al.  ARIMA Models to Predict Next-Day Electricity Prices , 2002, IEEE Power Engineering Review.

[65]  Tim Salimans,et al.  Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks , 2016, NIPS.

[66]  Umesh V. Vazirani,et al.  An Introduction to Computational Learning Theory , 1994 .

[67]  Eran Segal,et al.  Regularization Learning Networks , 2018, NeurIPS.

[68]  Yuan Zhang,et al.  Short-Term Residential Load Forecasting Based on LSTM Recurrent Neural Network , 2019, IEEE Transactions on Smart Grid.

[69]  Sepp Hochreiter,et al.  Self-Normalizing Neural Networks , 2017, NIPS.

[70]  Ponnuthurai N. Suganthan,et al.  Walk-forward empirical wavelet random vector functional link for time series forecasting , 2021, Appl. Soft Comput..

[71]  José Manuel Benítez,et al.  On the use of cross-validation for time series predictor evaluation , 2012, Inf. Sci..

[72]  C. Granger,et al.  Handbook of Economic Forecasting , 2006 .

[73]  Philip S. Yu,et al.  An unsupervised parameter learning model for RVFL neural network , 2019, Neural Networks.

[74]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[75]  Vladlen Koltun,et al.  An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling , 2018, ArXiv.

[76]  Ömer Faruk Ertugrul,et al.  A novel type of activation function in artificial neural networks: Trained activation function , 2018, Neural Networks.

[77]  Le Zhang,et al.  An ensemble of decision trees with random vector functional link networks for multi-class classification , 2017, Appl. Soft Comput..

[78]  Alexander Gammerman,et al.  Ridge Regression Learning Algorithm in Dual Variables , 1998, ICML.

[79]  Min Han,et al.  Laplacian Echo State Network for Multivariate Time Series Prediction , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[80]  Gonzalo A. Ruz,et al.  Twitter Sentiment Classification Based on Deep Random Vector Functional Link , 2018, 2018 International Joint Conference on Neural Networks (IJCNN).

[81]  Dejan J. Sobajic,et al.  Learning and generalization characteristics of the random vector Functional-link net , 1994, Neurocomputing.

[82]  Manisa Pipattanasomporn,et al.  Robust short-term electrical load forecasting framework for commercial buildings using deep recurrent neural networks , 2020 .

[83]  Azim Heydari,et al.  Short-term electricity price and load forecasting in isolated power grids based on composite neural network and gravitational search optimization algorithm , 2020 .

[84]  Jérôme Gilles,et al.  Empirical Wavelet Transform , 2013, IEEE Transactions on Signal Processing.

[85]  J. J. Mulawka,et al.  A modified backpropagation algorithm , 1994, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94).

[86]  Le Zhang,et al.  Visual Tracking With Convolutional Random Vector Functional Link Network , 2017, IEEE Transactions on Cybernetics.

[87]  Guang-Bin Huang,et al.  Extreme Learning Machine for Multilayer Perceptron , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[88]  Davide Anguita,et al.  Energy Load Forecasting Using Empirical Mode Decomposition and Support Vector Regression , 2013, IEEE Transactions on Smart Grid.

[89]  Najdan Vukovic,et al.  A comprehensive experimental evaluation of orthogonal polynomial expanded random vector functional link neural networks for regression , 2017, Appl. Soft Comput..

[90]  Ponnuthurai N. Suganthan,et al.  Enhancing Multi-Class Classification of Random Forest using Random Vector Functional Neural Network and Oblique Decision Surfaces , 2018, 2018 International Joint Conference on Neural Networks (IJCNN).

[91]  Ponnuthurai N. Suganthan,et al.  A Novel Empirical Mode Decomposition With Support Vector Regression for Wind Speed Forecasting , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[92]  Janez Demsar,et al.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[93]  L. Pazvakawambwa,et al.  Forecasting methods and applications. , 2013 .