Rule-Based Evolutionary Online Learning Systems: Learning Bounds, Classification, and Prediction

Rule-based evolutionary online learning systems, often referred to as Michigan-style learning classifier systems (LCSs), were proposed nearly thirty years ago (Holland, 1976; Holland, 1977) originally calling them cognitive systems. LCSs combine the strength of reinforcement learning with the generalization capabilities of genetic algorithms promising a flexible, online generalizing, solely reinforcement dependent learning system. However, despite several initial successful applications of LCSs and their interesting relations with animal learning and cognition, understanding of the systems remained somewhat obscured. Questions concerning learning complexity or convergence remained unanswered. Performance in different problem types, problem structures, concept spaces, and hypothesis spaces stayed nearly unpredictable. This thesis has the following three major objectives: (1) to establish a facetwise theory approach for LCSs that promotes system analysis, understanding, and design; (2) to analyze, evaluate, and enhance the XCS classifier system (Wilson, 1995) by the means of the facetwise approach establishing a fundamental XCS learning theory; (3) to identify both the major advantages of an LCS-based learning approach as well as the most promising potential application areas. Achieving these three objectives leads to a rigorous understanding of LCS functioning that enables the successful application of LCSs to diverse problem types and problem domains. The quantitative analysis of XCS shows that the interactive, evolutionary-based online learning mechanism works machine learning competitively yielding a low-order polynomial learning complexity. Moreover, the facetwise analysis approach facilitates the successful design of more advanced LCSs including Holland's originally envisioned cognitive systems.

[1]  Martin V. Butz,et al.  Anticipation for learning, cognition and education , 2004, On the Horizon.

[2]  Martin V. Butz,et al.  Anticipatory Behavior in Adaptive Learning Systems , 2003, Lecture Notes in Computer Science.

[3]  Martin V. Butz,et al.  Gradient descent methods in learning classifier systems: improving XCS performance in multistep problems , 2005, IEEE Transactions on Evolutionary Computation.

[4]  Pat Langley,et al.  Estimating Continuous Distributions in Bayesian Classifiers , 1995, UAI.

[5]  M.H. Hassoun,et al.  Fundamentals of Artificial Neural Networks , 1996, Proceedings of the IEEE.

[6]  Fernando G. Lobo,et al.  Extended Compact Genetic Algorithm in C , 1999 .

[7]  M. Colombetti,et al.  An extension to the XCS classifier system for stochastic environments , 1999 .

[8]  David Maxwell Chickering,et al.  Learning Bayesian Networks: The Combination of Knowledge and Statistical Data , 1994, Machine Learning.

[9]  Ian H. Witten,et al.  Generating Accurate Rule Sets Without Global Optimization , 1998, ICML.

[10]  David E. Goldberg,et al.  The Design of Innovation: Lessons from and for Competent Genetic Algorithms , 2002 .

[11]  Christopher Stone,et al.  For Real! XCS with Continuous-Valued Inputs , 2003, Evolutionary Computation.

[12]  Kalyanmoy Deb,et al.  Genetic Algorithms, Noise, and the Sizing of Populations , 1992, Complex Syst..

[13]  Tim Kovacs,et al.  What Makes a Problem Hard for XCS? , 2000, IWLCS.

[14]  K. Dejong,et al.  An analysis of the behavior of a class of genetic adaptive systems , 1975 .

[15]  P. Lanzi,et al.  Adaptive Agents with Reinforcement Learning and Internal Memory , 2000 .

[16]  G. Schwarz Estimating the Dimension of a Model , 1978 .

[17]  David E. Goldberg,et al.  An Analysis of Reproduction and Crossover in a Binary-Coded Genetic Algorithm , 1987, ICGA.

[18]  Gregory F. Cooper,et al.  A Bayesian Method for the Induction of Probabilistic Networks from Data , 1992 .

[19]  Martin V. Butz,et al.  Internal Models and Anticipations in Adaptive Learning Systems , 2003, ABiALS.

[20]  David E. Goldberg,et al.  Implicit Niching in a Learning Classifier System: Nature's Way , 1994, Evolutionary Computation.

[21]  Rocco A. Servedio,et al.  Efficient algorithms in computational learning theory , 2001 .

[22]  Samir W. Mahfoud Niching methods for genetic algorithms , 1996 .

[23]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[24]  D.E. Goldberg,et al.  Classifier Systems and Genetic Algorithms , 1989, Artif. Intell..

[25]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[26]  A. Feng,et al.  Neural basis of hearing in real-world situations. , 2000, Annual review of psychology.

[27]  Ian H. Witten,et al.  Data mining: practical machine learning tools and techniques with Java implementations , 2002, SGMD.

[28]  Wray L. Buntine Theory Refinement on Bayesian Networks , 1991, UAI.

[29]  D. E. Goldberg,et al.  Genetic Algorithms as a Computational Theory of Conceptual Design , 1991 .

[30]  John C. Platt,et al.  Fast training of support vector machines using sequential minimal optimization, advances in kernel methods , 1999 .

[31]  Dana H. Ballard,et al.  Learning to perceive and act by trial and error , 1991, Machine Learning.

[32]  David E. Goldberg,et al.  Bayesian optimization algorithm, decision graphs, and Occam's razor , 2001 .

[33]  L. Valiant,et al.  Leslie G. Valiant. A theory of the learnable. Communications of the ACM, , 2022 .

[34]  David E. Goldberg,et al.  The Race, the Hurdle, and the Sweet Spot , 1998 .

[35]  Thomas G. Dietterich What is machine learning? , 2020, Archives of Disease in Childhood.

[36]  Andrew W. Moore,et al.  Reinforcement Learning: A Survey , 1996, J. Artif. Intell. Res..

[37]  David E. Goldberg,et al.  Genetic Algorithms with Sharing for Multimodalfunction Optimization , 1987, ICGA.

[38]  Leslie G. Valiant,et al.  A theory of the learnable , 1984, CACM.

[39]  Martin V. Butz,et al.  Anticipatory Learning Classifier Systems , 2002, Genetic Algorithms and Evolutionary Computation.

[40]  Michael R. James,et al.  Learning and discovery of predictive state representations in dynamical systems with reset , 2004, ICML.

[41]  David W. Aha,et al.  Instance-Based Learning Algorithms , 1991, Machine Learning.

[42]  Martin V. Butz,et al.  Efiective Online Detection of Task-Independent Landmarks , 2004 .

[43]  Andrew G. Barto,et al.  Using relative novelty to identify useful temporal abstractions in reinforcement learning , 2004, ICML.

[44]  Stewart W. Wilson Mining Oblique Data with XCS , 2000, IWLCS.

[45]  T. Kovacs XCS Classifier System Reliably Evolves Accurate, Complete, and Minimal Representations for Boolean Functions , 1998 .

[46]  Duncan Potts,et al.  Incremental learning of linear model trees , 2004, ICML.

[47]  Ingo Rechenberg,et al.  Evolutionsstrategie : Optimierung technischer Systeme nach Prinzipien der biologischen Evolution , 1973 .

[48]  Chris Drummond,et al.  Accelerating Reinforcement Learning by Composing Solutions of Automatically Identified Subtasks , 2011, J. Artif. Intell. Res..

[49]  Stewart W. Wilson Classifier systems and the animat problem , 2004, Machine Learning.

[50]  Ronald A. Howard,et al.  Influence Diagrams , 2005, Decis. Anal..

[51]  Pier Luca Lanzi,et al.  A Study of the Generalization Capabilities of XCS , 1997, ICGA.

[52]  Stewart W. Wilson Classifier Systems for Continuous Payoff Environments , 2004, GECCO.

[53]  Samir W. Mahfoud Crowding and Preselection Revisited , 1992, PPSN.

[54]  J. C. Johnston,et al.  Attention and performance. , 2001, Annual review of psychology.

[55]  Kenneth A. De Jong,et al.  Learning Concept Classification Rules Using Genetic Algorithms , 1991, IJCAI.

[56]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[57]  Ester Bernadó-Mansilla,et al.  Accuracy-Based Learning Classifier Systems: Models, Analysis and Applications to Classification Tasks , 2003, Evolutionary Computation.

[58]  Andreas Rauber,et al.  The growing hierarchical self-organizing map , 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium.

[59]  Thomas G. Dietterich Machine-Learning Research Four Current Directions , 1997 .

[60]  Wolfgang Stolzmann,et al.  Anticipatory Classifier Systems: An introduction , 2001 .

[61]  F. W. Irwin Purposive Behavior in Animals and Men , 1932, The Psychological Clinic.

[62]  L. Baird Reinforcement Learning Through Gradient Descent , 1999 .

[63]  Dirk Thierens,et al.  Toward a Better Understanding of Mixing in Genetic Algorithms , 1993 .

[64]  Stewart W. Wilson Classifier Fitness Based on Accuracy , 1995, Evolutionary Computation.

[65]  Robert C. Holte,et al.  Very Simple Classification Rules Perform Well on Most Commonly Used Datasets , 1993, Machine Learning.

[66]  Martin V. Butz,et al.  An algorithmic description of XCS , 2000, Soft Comput..

[67]  Martin V. Butz,et al.  Toward a theory of generalization and learning in XCS , 2004, IEEE Transactions on Evolutionary Computation.

[68]  Stewart W. Wilson The animat path to AI , 1991 .

[69]  Kalyanmoy Deb,et al.  A Comparative Analysis of Selection Schemes Used in Genetic Algorithms , 1990, FOGA.

[70]  Olivier Sigaud,et al.  Adding a generalization mechanism to YACS , 2001 .

[71]  A. Martin V. Butz,et al.  The anticipatory classifier system and genetic generalization , 2002, Natural Computing.

[72]  Melanie Mitchell,et al.  The royal road for genetic algorithms: Fitness landscapes and GA performance , 1991 .

[73]  Stewart W. WilsonPrediction Function Approximation with a Classi er System , 2001 .

[74]  Richard S. Sutton,et al.  Predictive Representations of State , 2001, NIPS.

[75]  Martin V. Butz,et al.  Gradient-Based Learning Updates Improve XCS Performance in Multistep Problems , 2004, GECCO.

[76]  Herbert A. Simon,et al.  The Sciences of the Artificial , 1970 .

[77]  James E. Baker,et al.  Adaptive Selection Methods for Genetic Algorithms , 1985, International Conference on Genetic Algorithms.

[78]  Arthur Gelb,et al.  Applied Optimal Estimation , 1974 .

[79]  Martin V. Butz,et al.  Data Mining in Learning Classifier Systems: Comparing XCS with GAssist , 2005, IWLCS.

[80]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[81]  John H. Holland,et al.  Cognitive systems based on adaptive algorithms , 1977, SGAR.

[82]  G. Harik Linkage Learning via Probabilistic Modeling in the ECGA , 1999 .

[83]  Leonard Kleinrock,et al.  Queueing Systems: Volume I-Theory , 1975 .

[84]  Joachim Hoffmann,et al.  Anticipatory Behavioral Control , 2003, ABiALS.

[85]  David E. Goldberg,et al.  Probability Matching, the Magnitude of Reinforcement, and Classifier System Bidding , 1990, Machine Learning.

[86]  Luca Lanzi Pier,et al.  Extending the Representation of Classifier Conditions Part II: From Messy Coding to S-Expressions , 1999 .

[87]  Lashon B. Booker,et al.  Intelligent Behavior as an Adaptation to the Task Environment , 1982 .

[88]  MSc PhD Tim Kovacs BA Strength or Accuracy: Credit Assignment in Learning Classifier Systems , 2004, Distinguished Dissertations.

[89]  Richard S. Sutton,et al.  Reinforcement learning architectures for animats , 1991 .

[90]  Pier Luca Lanzi,et al.  An Analysis of Generalization in the XCS Classifier System , 1999, Evolutionary Computation.

[91]  G. Aschersleben,et al.  The Theory of Event Coding (TEC): a framework for perception and action planning. , 2001, The Behavioral and brain sciences.

[92]  David E. Goldberg,et al.  A Survey of Optimization by Building and Using Probabilistic Models , 2002, Comput. Optim. Appl..

[93]  Morgan B Kaufmann,et al.  Finite Markov Chain Analysis of Genetic Algorithms with Niching , 1993 .

[94]  D. Goldberg,et al.  Bounding Learning Time in XCS , 2004, GECCO.

[95]  Gilles Venturini,et al.  Adaptation in dynamic environments through a minimal probability of exploration , 1994 .

[96]  Stewart W. Wilson Knowledge Growth in an Artificial Animal , 1985, ICGA.

[97]  Nello Cristianini,et al.  Kernel Methods for Pattern Analysis , 2006 .

[98]  Martin V. Butz,et al.  Bounding the Population Size in XCS to Ensure Reproductive Opportunities , 2003, GECCO.

[99]  D. Goldberg,et al.  A practical schema theorem for genetic algorithm design and tuning , 2001 .

[100]  Xavier Llorà,et al.  Bounding the Effect of Noise in Multiobjective Learning Classifier Systems , 2003, Evolutionary Computation.

[101]  David E. Goldberg,et al.  A Critical Review of Classifier Systems , 1989, ICGA.

[102]  Tim Kovacs,et al.  Strength or Accuracy? Fitness Calculation in Learning Classifier Systems , 1999, Learning Classifier Systems.

[103]  D. Goldberg,et al.  Domino convergence, drift, and the temporal-salience structure of problems , 1998, 1998 IEEE International Conference on Evolutionary Computation Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98TH8360).

[104]  Stewart W. Wilson ZCS: A Zeroth Level Classifier System , 1994, Evolutionary Computation.

[105]  Sean R Eddy,et al.  What is dynamic programming? , 2004, Nature Biotechnology.

[106]  M. Pelikán,et al.  Analyzing the evolutionary pressures in XCS , 2001 .

[107]  Leemon C. Baird,et al.  Residual Algorithms: Reinforcement Learning with Function Approximation , 1995, ICML.

[108]  David E. Goldberg,et al.  Accuracy, Parsimony, and Generality in Evolutionary Learning Systems via Multiobjective Selection , 2002, IWLCS.

[109]  Sridhar Mahadevan,et al.  Recent Advances in Hierarchical Reinforcement Learning , 2003, Discret. Event Dyn. Syst..

[110]  David E. Goldberg,et al.  Bayesian optimization algorithm: from single level to hierarchy , 2002 .

[111]  T. Kovacs Deletion Schemes for Classi er Systems , 1999 .

[112]  D. Goldberg,et al.  Minimal Achievable Error in the LED problem , 2002 .

[113]  Lashon B. Booker,et al.  Recombination Distributions for Genetic Algorithms , 1992, FOGA.

[114]  Martin V. Butz,et al.  How XCS evolves accurate classifiers , 2001 .

[115]  Olivier Sigaud,et al.  YACS: Combining Dynamic Programming with Generalization in Classifier Systems , 2000, IWLCS.

[116]  Stewart W. Wilson The Genetic Algorithm and Simulated Evolution , 1987, ALIFE.

[117]  Pedro Larrañaga,et al.  A Review on Estimation of Distribution Algorithms , 2002, Estimation of Distribution Algorithms.

[118]  Richard S. Sutton,et al.  Integrated Architectures for Learning, Planning, and Reacting Based on Approximating Dynamic Programming , 1990, ML.

[119]  Luc De Raedt,et al.  Bellman goes relational , 2004, ICML.

[120]  Stewart W. Wilson Generalization in the XCS Classifier System , 1998 .

[121]  Olivier Sigaud,et al.  Designing Efficient Exploration with MACS: Modules and Function Approximation , 2003, GECCO.

[122]  Thomas G. Dietterich Hierarchical Reinforcement Learning with the MAXQ Value Function Decomposition , 1999, J. Artif. Intell. Res..

[123]  A. Gray,et al.  I. THE ORIGIN OF SPECIES BY MEANS OF NATURAL SELECTION , 1963 .

[124]  Doina Precup,et al.  Between MDPs and Semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning , 1999, Artif. Intell..

[125]  Stewart W. Wilson,et al.  Toward Optimal Classifier System Performance in Non-Markov Environments , 2000, Evolutionary Computation.

[126]  Pier Luca Lanzi Learning classifier systems from a reinforcement learning perspective , 2002, Soft Comput..

[127]  Tim Kovacs,et al.  Towards a Theory of Strong Overgeneral Classifiers , 2000, FOGA.

[128]  Peter Stone,et al.  Learning Predictive State Representations , 2003, ICML.

[129]  Ronald A. Howard,et al.  Readings on the Principles and Applications of Decision Analysis , 1989 .

[130]  Stewart W. Wilson Get Real! XCS with Continuous-Valued Inputs , 1999, Learning Classifier Systems.

[131]  Dirk Thierens,et al.  Mixing in Genetic Algorithms , 1993, ICGA.

[132]  David Maxwell Chickering,et al.  A Bayesian Approach to Learning Bayesian Networks with Local Structure , 1997, UAI.

[133]  H. Crichton-Miller Adaptation , 1926 .

[134]  S. Grossberg,et al.  Adaptive pattern classification and universal recoding: I. Parallel development and coding of neural feature detectors , 1976, Biological Cybernetics.

[135]  Nir Friedman,et al.  Learning Bayesian Networks with Local Structure , 1996, UAI.

[136]  H. Pashler The Psychology of Attention , 1997 .

[137]  Martin V. Butz,et al.  Strong, Stable, and Reliable Fitness Pressure in XCS due to Tournament Selection , 2005, Genetic Programming and Evolvable Machines.

[138]  J. Hoffmann,et al.  Anticipated Action Effects Affect the Selection, Initiation, and Execution of Actions , 2004, The Quarterly journal of experimental psychology. A, Human experimental psychology.

[139]  Max Henrion,et al.  Propagating uncertainty in bayesian networks by probabilistic logic sampling , 1986, UAI.

[140]  Stephen Grossberg,et al.  Adaptive pattern classification and universal recoding: II. Feedback, expectation, olfaction, illusions , 1976, Biological Cybernetics.

[141]  Chris Watkins,et al.  Learning from delayed rewards , 1989 .

[142]  Larry Bull Investigating fitness sharing in a simple payoff-based learning classifier system , 2003 .

[143]  D. Goldberg,et al.  BOA: the Bayesian optimization algorithm , 1999 .

[144]  T. Kovacs Deletion schemes for classifier systems , 1999 .

[145]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[146]  er SystemsTim KovacsOctober Evolving Optimal Populations with XCS Classi , 1996 .

[147]  LanziPier Luca An analysis of generalization in the xcs classifier system , 1999 .

[148]  C. Atkeson,et al.  Prioritized Sweeping: Reinforcement Learning with Less Data and Less Time , 1993, Machine Learning.

[149]  Herbert Jaeger,et al.  Observable Operator Models for Discrete Stochastic Time Series , 2000, Neural Computation.

[150]  Martin V. Butz,et al.  Analysis and Improvement of Fitness Exploitation in XCS: Bounding Models, Tournament Selection, and Bilateral Accuracy , 2003, Evolutionary Computation.

[151]  Stewart W. Wilson Function approximation with a classifier system , 2001 .