Evolving model trees for mining data sets with continuous-valued classes

This paper presents a genetic programming (GP) approach to extract symbolic rules from data sets with continuous-valued classes, called GPMCC. The GPMCC makes use of a genetic algorithm (GA) to evolve multi-variate non-linear models [Potgieter, G., & Engelbrecht, A. (2007). Genetic algorithms for the structural optimisation of learned polynomial expressions. Applied Mathematics and Computation] at the terminal nodes of the GP. Several mechanisms have been developed to optimise the GP, including a fragment pool of candidate non-linear models, k-means clustering of the training data to facilitate the use of stratified sampling methods, and specialized mutation and crossover operators to evolve structurally optimal and accurate models. It is shown that the GPMCC is insensitive to control parameter values. Experimental results show that the accuracy of the GPMCC is comparable to that of NeuroLinear and Cubist, while producing significantly less rules with less complex antecedents.

[1]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .

[2]  Hussein A. Abbass,et al.  Data Mining: A Heuristic Approach , 2002 .

[3]  Kalyanmoy Deb,et al.  A Comparative Analysis of Selection Schemes Used in Genetic Algorithms , 1990, FOGA.

[4]  Heekuck Oh,et al.  Neural Networks for Pattern Recognition , 1993, Adv. Comput..

[5]  Zbigniew Michalewicz,et al.  Evolutionary Computation 2 : Advanced Algorithms and Operators , 2000 .

[6]  Elie Bienenstock,et al.  Neural Networks and the Bias/Variance Dilemma , 1992, Neural Computation.

[7]  Geum Yong Lee.  Genetic recursive regression for modeling and forecasting real-world chaotic time series , 1999 .

[8]  J. R. Quinlan.  Learning With Continuous Classes , 1992 .

[9]  Gavin Potgieter.  Mining continuous classes using evolutionary computing , 2003 .

[10]  Earl B. Hunt,et al.  Machine learning: An artificial intelligence approach (vol. 2): R. S. Michalski, J. G. Carbonell, and T. M. Mitchell (Eds.). Los Alton, CA: Morgan Kaufmann, 1986. Pp. x + 738. $39.95 , 1987 .

[11]  Alex A. Freitas,et al.  A Genetic Programming Framework for Two Data Mining Tasks: Classification and Generalized Rule Induction , 1997 .

[12]  J. Douglas Faires,et al.  Numerical Analysis , 1981 .

[13]  John E. Dennis,et al.  Numerical methods for unconstrained optimization and nonlinear equations , 1983, Prentice Hall series in computational mathematics.

[14]  Peter Clark,et al.  The CN2 Induction Algorithm , 1989, Machine Learning.

[15]  J. Ross Quinlan,et al.  Learning Efficient Classification Procedures and Their Application to Chess End Games , 1983 .

[16]  Jacek M. Zurada,et al.  Extraction of rules from artificial neural networks for nonlinear regression , 2002, IEEE Trans. Neural Networks.

[17]  M. A. Kaboudan,et al.  Genetic evolution of regression models for business and economic forecasting , 1999, Proceedings of the 1999 Congress on Evolutionary Computation-CEC99 (Cat. No. 99TH8406).

[18]  J. Miller.  Numerical Analysis , 1966, Nature.

[19]  Rudy Setiono,et al.  Generating Linear Regression Rules from Neural Networks Using Local Least Squares Approximation , 2001, IWANN.

[20]  Andries Petrus Engelbrecht,et al.  Genetic algorithms for the structural optimisation of learned polynomial expressions , 2007, Appl. Math. Comput..

[21]  E. Forgy.  Cluster analysis of multivariate data : efficiency versus interpretability of classifications , 1965 .

[22]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[23]  John R. Koza,et al.  Genetic programming 1997 : proceedings of the Second Annual Conference, July 13-16, 1997, Stanford University , 1997 .

[24]  Peter Clark,et al.  Rule Induction with CN2: Some Recent Improvements , 1991, EWSL.

[25]  Andries P. Engelbrecht,et al.  A Building Block Approach to Genetic Programming for Rule Discovery , 2002 .

[26]  LiMin Fu,et al.  Rule Generation from Neural Networks , 1994, IEEE Trans. Syst. Man Cybern. Syst..

[27]  Jacek M. Zurada,et al.  Introduction to artificial neural systems , 1992 .

[28]  Krister Svanberg.  Structural Optimization , 2009, Encyclopedia of Optimization.

[29]  G.G. Langdon,et al.  Data compression , 1988, IEEE Potentials.

[30]  John R. Anderson,et al.  Machine learning - an artificial intelligence approach , 1982, Symbolic computation.

[31]  Rudy Setiono,et al.  Use of a quasi-Newton method in a feedforward neural network construction algorithm , 1995, IEEE Trans. Neural Networks.

[32]  Jano I. van Hemert,et al.  Adapting the Fitness Function in GP for Data Mining , 1999, EuroGP.

[33]  John R. Koza,et al.  Genetic programming - on the programming of computers by means of natural selection , 1993, Complex adaptive systems.

[34]  Riccardo Poli,et al.  Foundations of Genetic Programming , 1999, Springer Berlin Heidelberg.

[35]  Wee Kheng Leow,et al.  Pruned Neural Networks for Regression , 2000, PRICAI.

[36]  David B. Fogel,et al.  Evolution-ary Computation 1: Basic Algorithms and Operators , 2000 .

[37]  Andries Petrus Engelbrecht,et al.  Supervised Training Using an Unsupervised Approach to Active Learning , 2002, Neural Processing Letters.