Efficient Stepwise Selection in Decomposable Models

In this paper, we present an efficient algorithm for performing stepwise selection in the class of decomposable models. We focus on the forward selection procedure, but we also discuss how backward selection and the combination of the two can be performed efficiently. The main contributions of this paper are (1) a simple characterization for the edges that can be added to a decomposable model while retaining its decomposability and (2) an efficient algorithm for enumerating all such edges for a given decomposable model in O(n2) time, where n is the number of variables in the model. We also analyze the complexity of the overall stepwise selection procedure (which includes the complexity of enumerating eligible edges as well as the complexity of deciding how to "progress"). We use the KL divergence of the model from the saturated model as our metric, but the results we present here extend to many other metrics as well.

[1]  K. Pearson.  Biometrika , 1902, The American Naturalist.

[2]  N. Wermuth.  Model Search among Multiplicative Models , 1976 .

[3]  Catriel Beeri,et al.  On the Desirability of Acyclic Database Schemes , 1983, JACM.

[4]  M. Frydenberg,et al.  Decomposition of maximum likelihood in mixed graphical interaction models , 1989 .

[5]  Francesco M. Malvestuto,et al.  Approximating discrete probability distributions with decomposable models , 1991, IEEE Trans. Syst. Man Cybern..

[6]  Frank Jensen,et al.  Optimal junction Trees , 1994, UAI.

[7]  Janyce Wiebe,et al.  Word-Sense Disambiguation Using Decomposable Models , 1994, ACL.

[8]  Michel Habib,et al.  Chordal Graphs and Their Clique Graphs , 1995, WG.

[9]  D. Edwards.  Introduction to graphical modelling , 1995 .

[10]  Jeffrey F. Naughton,et al.  On the Computation of Multidimensional Aggregates , 1996, VLDB.

[11]  John D. Lafferty,et al.  Inducing Features of Random Fields , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[12]  Song-Chun Zhu,et al.  Minimax Entropy Principle and Its Application to Texture Modeling , 1997, Neural Computation.

[13]  Michael I. Jordan.  Graphical Models , 1998 .

[14]  P. Green,et al.  Decomposable graphical Gaussian model determination , 1999 .

[15]  Michael I. Jordan,et al.  Learning with Mixtures of Trees , 2001, J. Mach. Learn. Res..

[16]  R. Christensen.  Introduction to Graphical Modeling , 2001 .

[17]  Rajeev Rastogi,et al.  Independence is good: dependency-based histogram synopses for high-dimensional data , 2001, SIGMOD '01.