Substructrual surrogates for learning decomposable classification problems: implementation and first results

This paper presents a learning methodology based on a substructural classification model to solve decomposable classification problems. The proposed method consists of three important components: (1) a structural model that represents salient interactions between attributes for a given data, (2) a surrogate model which provides a functional approximation of the output as a function of attributes, and (3) a classification model which predicts the class for new inputs. The structural model is used to infer the functional form of the surrogate and its coefficients are estimated using linear regression methods. The classification model uses a maximally-accurate, least-complex surrogate to predict the output for given inputs. The structural model that yields an optimal classification model is searched using an iterative greedy search heuristic. Results show that the proposed method successfully detects the interacting variables in hierarchical problems, group them in linkages groups, and build maximally accurate classification models. The initial results on non-trivial hierarchical test problems indicate that the proposed method holds promise and have also shed light on several improvements to enhance the capabilities of the proposed method.

[1]  Martin Pelikan,et al.  Fitness Inheritance in the Bayesian Optimization Algorithm , 2004, GECCO.

[2]  Herbert A. Simon,et al.  The Sciences of the Artificial , 1970 .

[3]  G. Harik.  Linkage Learning via Probabilistic Modeling in the ECGA , 1999 .

[4]  Chih-Jen Lin,et al.  Asymptotic Behaviors of Support Vector Machines with Gaussian Kernel , 2003, Neural Computation.

[5]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .

[6]  Martin V. Butz,et al.  Automated Global Structure Extraction for Effective Local Building Block Processing in XCS , 2006, Evolutionary Computation.

[7]  Ester Bernadó-Mansilla,et al.  Bounding XCS's parameters for unbalanced datasets , 2006, GECCO '06.

[8]  David E. Goldberg,et al.  Efficiency enhancement of genetic algorithms via building-block-wise fitness estimation , 2004, Proceedings of the 2004 Congress on Evolutionary Computation (IEEE Cat. No.04TH8753).

[9]  Ester Bernadó-Mansilla,et al.  The Class Imbalance Problem in UCS Classifier System: A Preliminary Study , 2005, IWLCS.

[10]  D. Goldberg,et al.  A matrix approach for finding extrema: problems with modularity, hierarchy, and overlap , 2006 .

[11]  T. Kovacs.  Deletion schemes for classifier systems , 1999 .

[12]  N. Draper,et al.  Applied Regression Analysis , 1966 .

[13]  Stewart W. Wilson.  Classifier Fitness Based on Accuracy , 1995, Evolutionary Computation.

[14]  Martin V. Butz,et al.  Rule-Based Evolutionary Online Learning Systems - A Principled Approach to LCS Analysis and Design , 2006, Studies in Fuzziness and Soft Computing.

[15]  John C. Platt,et al.  Fast training of support vector machines using sequential minimal optimization, advances in kernel methods , 1999 .

[16]  D. Goldberg,et al.  Probabilistic Model Building and Competent Genetic Programming , 2003 .

[17]  T. Kovacs.  XCS Classifier System Reliably Evolves Accurate, Complete, and Minimal Representations for Boolean Functions , 1998 .

[18]  Stewart W. Wilson.  Generalization in the XCS Classifier System , 1998 .

[19]  Ester Bernadó-Mansilla,et al.  Accuracy-Based Learning Classifier Systems: Models, Analysis and Applications to Classification Tasks , 2003, Evolutionary Computation.

[20]  Martin Pelikan,et al.  Hierarchical Bayesian optimization algorithm: toward a new generation of evolutionary algorithms , 2010, SICE 2003 Annual Conference (IEEE Cat. No.03TH8734).

[21]  Thomas G. Dietterich.  Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms , 1998, Neural Computation.

[22]  Dale Borowiak,et al.  Linear Models, Least Squares and Alternatives , 2001, Technometrics.

[23]  N. Draper,et al.  Applied Regression Analysis. , 1967 .

[24]  David E. Goldberg,et al.  The Design of Innovation: Lessons from and for Competent Genetic Algorithms , 2002 .

[25]  Martin V. Butz,et al.  How XCS evolves accurate classifiers , 2001 .

[26]  Xavier Llorà,et al.  Do not match, inherit: fitness surrogates for genetics-based machine learning techniques , 2007, GECCO '07.

[27]  Aiko M. Hormann,et al.  Programs for Machine Learning. Part I , 1962, Inf. Control..

[28]  Kenneth A. De Jong,et al.  Learning Concept Classification Rules Using Genetic Algorithms , 1991, IJCAI.

[29]  E. Reed.  The Ecological Approach to Visual Perception , 1989 .

[30]  Stewart W. Wilson.  Quasi-Darwinian Learning in a Classifier System , 1987 .

[31]  Ehl Emile Aarts,et al.  Simulated annealing and Boltzmann machines , 2003 .

[32]  Pier Luca Lanzi,et al.  An Analysis of Generalization in the XCS Classifier System , 1999, Evolutionary Computation.

[33]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[34]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[35]  David E. Goldberg,et al.  Evaluation relaxation using substructural information and linear estimation , 2006, GECCO '06.

[36]  Shumeet Baluja,et al.  Incorporating a priori Knowledge in Probabilistic-Model Based Optimization , 2006, Scalable Optimization via Probabilistic Modeling.