The Improvement of Dropout Strategy Based on Two Evolutionary Algorithms

Dropout strategy is a simple and common regularization method in the construction of deep network that it can control the status of units in the Dropout layers according to the constant probability values in the training processes to prevent the training from overfitting. However, the probability values of the Dropout strategy are single and decided by users, which means that we need more training iterations to receive better results and avoid less fitting problem. In this paper, two evolutionary algorithms, genetic algorithm and differential evolution algorithm are used to optimize the set probability values of network units to improve dropout strategy and they are proved to be able to increase the accuracy of the original method to about 5%.

[1]  Trevor Darrell,et al.  Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[2]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[3]  Joan Bruna,et al.  Intriguing properties of neural networks , 2013, ICLR.

[4]  Li Zhang.  Speech Recognition Based o , 2003 .

[5]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Zhiyuan Liu,et al.  A C-LSTM Neural Network for Text Classification , 2015, ArXiv.

[7]  K. Stanislaw,et al.  A new constraint tournament selection method for multicriteria optimization using genetic algorithm , 2000, Proceedings of the 2000 Congress on Evolutionary Computation. CEC00 (Cat. No.00TH8512).

[8]  William M. Spears,et al.  A Study of Crossover Operators in Genetic Programming , 1991, ISMIS.

[9]  Rainer Storn,et al.  Differential Evolution – A Simple and Efficient Heuristic for global Optimization over Continuous Spaces , 1997, J. Glob. Optim..

[10]  P. Rocca,et al.  Differential evolution as applied to electromagnetics: Advances, comparisons, and applications , 2012, 2012 6th European Conference on Antennas and Propagation (EUCAP).

[11]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[12]  Xie Bai-qiao.  Overview of differential evolution algorithm and its improved algorithms , 2008 .