Synthesis of sigma-pi neural networks by the breeder genetic programming

Genetic programming has been successfully applied to evolve computer programs for solving a variety of interesting problems. The breeder genetic programming (BGP) method has Occam's razor in its fitness measure to evolve minimal size multilayer perceptrons. In this paper, we apply the method to synthesis of sigma-pi neural networks. Unlike perceptron architectures, sigma-pi networks use product units as well as summation units to build higher-order terms. The effectiveness of the method is demonstrated on benchmark problems. Simulation results on noisy data suggest that BGP not only improves the generalization performance, but it can also accelerate the convergence speed.<<ETX>>

[1]  Peter M. Todd,et al.  Designing Neural Networks using Genetic Algorithms , 1989, ICGA.

[2]  David E. Rumelhart,et al.  Product Units: A Computationally Powerful and Biologically Plausible Extension to Backpropagation Networks , 1989, Neural Computation.

[3]  Peter Secretan.  Learning , 1965, Mental Health.

[4]  Shun-ichi Amari,et al.  Dualistic geometry of the manifold of higher-order neurons , 1991, Neural Networks.

[5]  Frédéric Gruau,et al.  Genetic synthesis of Boolean neural networks with a cell rewriting developmental process , 1992, [Proceedings] COGANN-92: International Workshop on Combinations of Genetic Algorithms and Neural Networks.

[6]  A. G. Ivakhnenko,et al.  Polynomial Theory of Complex Systems , 1971, IEEE Trans. Syst. Man Cybern..

[7]  John R. Koza,et al.  Genetic programming - on the programming of computers by means of natural selection , 1993, Complex adaptive systems.

[8]  Heinz Mühlenbein,et al.  Parallel Genetic Algorithms in Combinatorial Optimization , 1992, Computer Science and Operations Research.

[9]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[10]  Byoung-Tak Zhang,et al.  Evolving Optimal Neural Networks Using Genetic Algorithms with Occam's Razor , 1993, Complex Syst..

[11]  Hillol Kargupta,et al.  System Identification with Evolving Polynomial Networks , 1991, ICGA.

[12]  Tariq Samad,et al.  Towards the Genetic Synthesisof Neural Networks , 1989, ICGA.

[13]  Schloss Birlinghoven.  Evolution in Time and Space -the Parallel Genetic Algorithm , 1991 .

[14]  Jan M. Zytkow,et al.  Discovery of Equations: Experimental Evaluation of Convergence , 1992, AAAI.

[15]  Thomas Bäck,et al.  An Overview of Evolutionary Algorithms for Parameter Optimization , 1993, Evolutionary Computation.

[16]  Marvin Minsky,et al.  Perceptrons: An Introduction to Computational Geometry , 1969 .

[17]  Heinz Mühlenbein,et al.  Predictive Models for the Breeder Genetic Algorithm I. Continuous Parameter Optimization , 1993, Evolutionary Computation.

[18]  Kenneth E. Kinnear,et al.  Generality and Difficulty in Genetic Programming: Evolving a Sort , 1993, ICGA.

[19]  Jerome A. Feldman,et al.  Connectionist Models and Their Properties , 1982, Cogn. Sci..

[20]  Hiroaki Kitano,et al.  Designing Neural Networks Using Genetic Algorithms with Graph Generation System , 1990, Complex Syst..

[21]  Hitoshi Iba,et al.  System Identification using Structured Genetic Algorithms , 1993, ICGA.

[22]  R. Sorkin.  A quantitative occam's razor , 1983, astro-ph/0511780.

[23]  Zbigniew Michalewicz,et al.  Genetic Algorithms + Data Structures = Evolution Programs , 1996, Springer Berlin Heidelberg.

[24]  Colin Giles,et al.  Learning, invariance, and generalization in high-order neural networks. , 1987, Applied optics.

[25]  Geoffrey E. Hinton,et al.  A general framework for parallel distributed processing , 1986 .

[26]  D. E. Goldberg,et al.  Genetic Algorithms in Search , 1989 .

[27]  Byoung-Tak Zhang,et al.  Genetic Programming of Minimal Neural Nets Using Occam's Razor , 1993, ICGA.

[28]  Byoung-Tak Zhang,et al.  Accelerated Learning by Active Example Selection , 1994, Int. J. Neural Syst..

[29]  L. Darrell Whitley,et al.  Genetic algorithms and neural networks: optimizing connections and connectivity , 1990, Parallel Comput..

[30]  Walter Alden Tackett,et al.  Genetic Programming for Feature Discovery and Image Discrimination , 1993, ICGA.

[31]  Peter J. Angeline,et al.  Competitive Environments Evolve Better Solutions for Complex Tasks , 1993, ICGA.

[32]  H. M. Uhlenbein.  Evolutionary Algorithms: Theory and Applications , 1993 .

[33]  John R. Koza,et al.  Genetic Programming: The Movie , 1992 .

[34]  Zbigniew Michalewicz,et al.  Genetic Algorithms + Data Structures = Evolution Programs , 1992, Artificial Intelligence.

[35]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .