Evolving neural networks for strategic decision-making problems

Evolution of neural networks, or neuroevolution, has been a successful approach to many low-level control problems such as pole balancing, vehicle control, and collision warning. However, certain types of problems-such as those involving strategic decision-making-have remained difficult for neuroevolution to solve. This paper evaluates the hypothesis that such problems are difficult because they are fractured: The correct action varies discontinuously as the agent moves from state to state. A method for measuring fracture using the concept of function variation is proposed and, based on this concept, two methods for dealing with fracture are examined: neurons with local receptive fields, and refinement based on a cascaded network architecture. Experiments in several benchmark domains are performed to evaluate how different levels of fracture affect the performance of neuroevolution methods, demonstrating that these two modifications improve performance significantly. These results form a promising starting point for expanding neuroevolution to strategic tasks.

[1]  Lin Guo,et al.  Combining genetic optimisation with hybrid learning algorithm for radial basis function neural networks , 2003 .

[2]  Shimon Whiteson,et al.  Comparing evolutionary and temporal difference methods in a reinforcement learning domain , 2006, GECCO.

[3]  Vladimir Vapnik,et al.  Chervonenkis: On the uniform convergence of relative frequencies of events to their probabilities , 1971 .

[4]  Stephen A. Billings,et al.  Radial basis function network configuration using genetic algorithms , 1995, Neural Networks.

[5]  Héctor Pomares,et al.  Parallel multiobjective memetic RBFNNs design and feature selection for function approximation problems , 2009, Neurocomputing.

[6]  Héctor Pomares,et al.  Multiobjective evolutionary optimization of the size, shape, and position parameters of radial basis function networks for function approximation , 2003, IEEE Trans. Neural Networks.

[7]  Kenneth A. De Jong,et al.  Cooperative Coevolution: An Architecture for Evolving Coadapted Subcomponents , 2000, Evolutionary Computation.

[8]  A. P. Wieland,et al.  Evolving neural network controllers for unstable systems , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[9]  Peter Stone,et al.  Keepaway Soccer: From Machine Learning Testbed to Benchmark , 2005, RoboCup.

[10]  Nicholas J. Radcliffe,et al.  Genetic set recombination and its application to neural network topology optimisation , 1993, Neural Computing & Applications.

[11]  Thomas G. Dietterich What is machine learning? , 2020, Archives of Disease in Childhood.

[12]  Joydeep Ghosh,et al.  An overview of radial basis function networks , 2001 .

[13]  Zhiye Zhao,et al.  Design of structural modular neural networks with genetic algorithm , 2003 .

[14]  Risto Miikkulainen,et al.  Competitive Coevolution through Evolutionary Complexification , 2011, J. Artif. Intell. Res..

[15]  Yoshua Bengio,et al.  Scaling learning algorithms towards AI , 2007 .

[16]  John Moody,et al.  Fast Learning in Networks of Locally-Tuned Processing Units , 1989, Neural Computation.

[17]  Bruce A. Whitehead,et al.  Cooperative-competitive genetic evolution of radial basis function centers and widths for time series prediction , 1996, IEEE Trans. Neural Networks.

[18]  Risto Miikkulainen,et al.  Evolving a real-world vehicle warning system , 2006, GECCO.

[19]  Stewart W. Wilson Classifier Conditions Using Gene Expression Programming , 2008, IWLCS.

[20]  Александр Сергеевич Леонов,et al.  Замечания о полной вариации функций нескольких переменных и многомерном аналоге принципа выбора Хелли@@@On the total variation for functions of several variables and a multidimensional analog of Helly's selection principle , 1998 .

[21]  Risto Miikkulainen,et al.  Real-time neuroevolution in the NERO video game , 2005, IEEE Transactions on Evolutionary Computation.

[22]  Justinian Rosca,et al.  Hierarchical learning with procedural abstraction mechanisms , 1997 .

[23]  Xin Yao,et al.  Evolving artificial neural networks , 1999, Proc. IEEE.

[24]  J. Mehler,et al.  LANGUAGE AND COGNITION , 1998 .

[25]  David B. Fogel,et al.  Evolving Neural Control Systems , 1995, IEEE Expert.

[26]  Jan M. Maciejowski,et al.  Model discrimination using an algorithmic information criterion , 1979, Autom..

[27]  John C. Platt A Resource-Allocating Network for Function Interpolation , 1991, Neural Computation.

[28]  P. Angeline Evolving basis functions with dynamic receptive fields , 1997, 1997 IEEE International Conference on Systems, Man, and Cybernetics. Computational Cybernetics and Simulation.

[29]  David McLean,et al.  On global-local artificial neural networks for function approximation , 2006, IEEE Trans. Neural Networks.

[30]  Jason Weston,et al.  Large-scale kernel machines , 2007 .

[31]  Jorma Rissanen,et al.  The Minimum Description Length Principle in Coding and Modeling , 1998, IEEE Trans. Inf. Theory.

[32]  Franz Oppacher,et al.  Combining Competitive And Cooperative Coevolution For Training Cascade Neural Networks , 2002, GECCO.

[33]  S. Lawrence,et al.  Function Approximation with Neural Networks and Local Methods: Bias, Variance and Smoothness , 1996 .

[34]  Larry Bull,et al.  Accuracy-based Neuro And Neuro-fuzzy Classifier Systems , 2002, GECCO.

[35]  Jooyoung Park,et al.  Universal Approximation Using Radial-Basis-Function Networks , 1991, Neural Computation.

[36]  Martin V. Butz,et al.  Context-dependent predictions and cognitive arm control with XCSF , 2008, GECCO '08.

[37]  E. Kamke Das Lebesgue-Stieltjes-Integral , 1958 .

[38]  Matthew Taylor and Shimon Whiteson and Peter Stone,et al.  Comparing Evolutionary and Temporal Difference Methods for Reinforcement Learning , 2006 .

[39]  Risto Miikkulainen,et al.  Coevolving Strategies for General Game Playing , 2007, 2007 IEEE Symposium on Computational Intelligence and Games.

[40]  Héctor Pomares,et al.  Improving the Performance of Multi-objective Genetic Algorithm for Function Approximation Through Parallel Islands Specialisation , 2006, Australian Conference on Artificial Intelligence.

[41]  Larry Bull,et al.  Self-adaptive constructivism in Neural XCS and XCSF , 2008, GECCO '08.

[42]  Yoshua. Bengio,et al.  Learning Deep Architectures for AI , 2007, Found. Trends Mach. Learn..

[43]  Martin V. Butz Kernel-based, ellipsoidal conditions in the real-valued XCS classifier system , 2005, GECCO '05.

[44]  David E. Goldberg,et al.  Genetic Algorithms with Sharing for Multimodalfunction Optimization , 1987, ICGA.

[45]  Zuhair Bandar,et al.  Neural network architectures and overtopping predictions , 2005 .

[46]  Didier Guériot,et al.  RBF neural network, basis functions and genetic algorithm , 1997, Proceedings of International Conference on Neural Networks (ICNN'97).

[47]  Li Jun,et al.  Q-Learning with a growing RBF network for behavior learning in mobile robotics , 2005 .

[48]  Todd Peterson,et al.  An RBF network alternative for a hybrid architecture , 1998, 1998 IEEE International Joint Conference on Neural Networks Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98CH36227).

[49]  Daniele Loiacono,et al.  Classifier prediction based on tile coding , 2006, GECCO '06.

[50]  A. Kolmogorov Three approaches to the quantitative definition of information , 1968 .

[51]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[52]  Hans-Martin Gutmann,et al.  A Radial Basis Function Method for Global Optimization , 2001, J. Glob. Optim..

[53]  Risto Miikkulainen,et al.  Efficient Non-linear Control Through Neuroevolution , 2006, ECML.

[54]  David P Stoten,et al.  UKACC International Conference on CONTROL , 2008 .

[55]  Alan F. Murray,et al.  IEEE International Conference on Neural Networks , 1997 .

[56]  Héctor Pomares,et al.  Parallel Multi-objective Memetic RBFNNs Design and Feature Selection for Function Approximation Problems , 2007, IWANN.

[57]  Ali M. S. Zalzala,et al.  Evolving hybrid RBF-MLP networks using combined genetic/unsupervised/supervised learning , 1998 .

[58]  Risto Miikkulainen,et al.  Evolving Soccer Keepaway Players Through Task Decomposition , 2005, Machine Learning.

[59]  Haralambos Sarimveis,et al.  A new algorithm for developing dynamic radial basis function neural network models based on genetic algorithms , 2002, Comput. Chem. Eng..

[60]  Robert Kozma,et al.  The KIV model of intentional dynamics and decision making , 2009, Neural Networks.

[61]  Risto Miikkulainen,et al.  Evolving Neural Networks through Augmenting Topologies , 2002, Evolutionary Computation.

[62]  L. D. Whitley,et al.  Genetic Reinforcement Learning for Neurocontrol Problems , 2004, Machine Learning.

[63]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[64]  Gregory J. Chaitin,et al.  A recent technical report , 1974, SIGA.

[65]  Stewart W. Wilson Classifiers that approximate functions , 2002, Natural Computing.

[66]  Risto Miikkulainen,et al.  Solving Non-Markovian Control Tasks with Neuro-Evolution , 1999, IJCAI.

[67]  Risto Miikkulainen,et al.  Efficient Reinforcement Learning through Symbiotic Evolution , 1996, Machine Learning.

[68]  Ming Li,et al.  An Introduction to Kolmogorov Complexity and Its Applications , 1997, Texts in Computer Science.

[69]  Risto Miikkulainen,et al.  Neuroevolution of an automobile crash warning system , 2005, GECCO '05.

[70]  Jun Li,et al.  Q-RAN: A Constructive Reinforcement Learning Approach for Robot Behavior Learning , 2006, 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[71]  Larry D. Pyeatt,et al.  A comparison between cellular encoding and direct encoding for genetic neural networks , 1996 .

[72]  Daniel S. Levine,et al.  2009 Special Issue: Brain pathways for cognitive-emotional decision making in the human animal , 2009 .

[73]  Daniele Loiacono,et al.  XCS with computed prediction in multistep environments , 2005, GECCO '05.

[74]  Risto Miikkulainen,et al.  Evolving a Roving Eye for Go , 2004, GECCO.

[75]  Minoru Asada,et al.  Non-Physical Intervention in Robot Learning Based on LfE Method , 1995 .

[76]  Charles W. Anderson,et al.  Comparison of CMACs and radial basis functions for local function approximators in reinforcement learning , 1997, Proceedings of International Conference on Neural Networks (ICNN'97).

[77]  Tin Kam Ho,et al.  Complexity Measures of Supervised Classification Problems , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[78]  Daniele Loiacono,et al.  XCS with computed prediction for the learning of Boolean functions , 2005, 2005 IEEE Congress on Evolutionary Computation.

[79]  Peter J. Angeline,et al.  An evolutionary algorithm that constructs recurrent neural networks , 1994, IEEE Trans. Neural Networks.

[80]  S. Bochner,et al.  Lectures on Fourier integrals : with an author's supplement on monotonic functions, Stieltjes integrals, and harmonic analysis , 1959 .

[81]  Risto Miikkulainen,et al.  Evolving Keepaway Soccer Players through Task Decomposition , 2003, GECCO.

[82]  Richard S. Sutton,et al.  Generalization in Reinforcement Learning: Successful Examples Using Sparse Coarse Coding , 1995, NIPS.