Incremental Induction of Decision Trees

This article presents an incremental algorithm for inducing decision trees equivalent to those formed by Quinlan's nonincremental ID3 algorithm, given the same training instances. The new algorithm, named ID5R, lets one apply the ID3 induction process to learning tasks in which training instances are presented serially. Although the basic tree-building algorithms differ only in how the decision trees are constructed, experiments show that incremental training makes it possible to select training instances more carefully, which can result in smaller decision trees. The ID3 algorithm and its variants are compared in terms of theoretical complexity and empirical behavior.

[1]  Philip M. Lewis,et al.  The characteristic selection problem in recognition systems , 1962, IRE Trans. Inf. Theory.

[2]  Tom Michael Mitchell,et al.  Model-directed learning of production rules , 1977, SGAR.

[3]  Tom M. Mitchell,et al.  MODEL-DIRECTED LEARNING OF PRODUCTION RULES1 , 1978 .

[4]  Tom Michael Mitchell.  Version spaces: an approach to concept learning. , 1979 .

[5]  Steven A. Vere,et al.  Multilevel Counterfactuals for Generalizations of Relational Concepts and Productions , 1980, Artif. Intell..

[6]  Bernard M. E. Moret,et al.  Decision Trees and Diagrams , 1982, CSUR.

[7]  Leo Breiman,et al.  Classification and Regression Trees , 1984 .

[8]  Ryszard S. Michalski,et al.  A Theory and Methodology of Inductive Learning , 1983, Artificial Intelligence.

[9]  J. Ross Quinlan,et al.  Learning Efficient Classification Procedures and Their Application to Chess End Games , 1983 .

[10]  Satosi Watanabe,et al.  Pattern Recognition: Human and Mechanical , 1985 .

[11]  A G Barto,et al.  Learning by statistical cooperation of self-interested neuron-like computing elements. , 1985, Human neurobiology.

[12]  Douglas H. Fisher,et al.  A Case Study of Incremental Concept Induction , 1986, AAAI.

[13]  J. R. B. Cockett.  Discrete Decision Theory: Manipulations , 1987, Theor. Comput. Sci..

[14]  J. Ross Quinlan,et al.  An Empirical Comparison of Genetic and Decision-Tree Classifiers , 1988, ML.

[15]  Paul E. Utgoff,et al.  ID5: An Incremental ID3 , 1987, ML Workshop.

[16]  Stephen I. Gallant,et al.  Connectionist expert systems , 1988, CACM.

[17]  Paul E. Utgoff,et al.  Improved Training Via Incremental Learning , 1989, ML.

[18]  Shinichi Morishita,et al.  On Classification and Regression , 1998, Discovery Science.

[19]  J. C. Schlimmer,et al.  Incremental learning from noisy data , 2004, Machine Learning.

[20]  J. Ross Quinlan,et al.  Induction of Decision Trees , 1986, Machine Learning.

[21]  Stewart W. Wilson.  Classifier systems and the animat problem , 2004, Machine Learning.

[22]  Douglas H. Fisher,et al.  Knowledge Acquisition Via Incremental Conceptual Clustering , 1987, Machine Learning.

[23]  Stewart W. Wilson.  Classifier Systems and the Animat Problem , 1987, Machine Learning.

[24]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .