Relational Dependency Networks

Recent work on graphical models for relational data has demonstrated significant improvements in classification and inference when models represent the dependencies among instances. Despite its use in conventional statistical models, the assumption of instance independence is contradicted by most relational data sets. For example, in citation data there are dependencies among the topics of a paper's references, and in genomic data there are dependencies among the functions of interacting proteins. In this paper, we present relational dependency networks (RDNs), graphical models that are capable of expressing and reasoning with such dependencies in a relational setting. We discuss RDNs in the context of relational Bayes networks and relational Markov networks and outline the relative strengths of RDNs---namely, the ability to represent cyclic dependencies, simple methods for parameter estimation, and efficient structure learning techniques. The strengths of RDNs are due to the use of pseudolikelihood learning techniques, which estimate an efficient approximation of the full joint distribution. We present learned RDNs for a number of real-world data sets and evaluate the models in a prediction context, showing that RDNs identify and exploit cyclic relational dependencies to achieve significant performance gains over conventional conditional models. In addition, we use synthetic data to explore model performance under various relational data characteristics, showing that RDN learning and inference techniques are accurate over a wide range of conditions.

[1]  Tom M. Mitchell,et al.  Learning to Extract Symbolic Knowledge from the World Wide Web , 1998, AAAI/IAAI.

[2]  Andrew McCallum,et al.  Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data , 2001, ICML.

[3]  Andrew McCallum,et al.  A Machine Learning Approach to Building Domain-Specific Search Engines , 1999, IJCAI.

[4]  Jennifer Neville,et al.  Supporting Relational Knowledge Discovery: Lessons in Architecture and Algorithm Design , 2002 .

[5]  Luc De Raedt,et al.  Basic Principles of Learning Bayesian Logic Programs , 2008, Probabilistic Inductive Logic Programming.

[6]  Foster J. Provost,et al.  Classification in Networked Data: a Toolkit and a Univariate Case Study , 2007, J. Mach. Learn. Res..

[7]  Corinna Cortes,et al.  Communities of interest , 2001, Intell. Data Anal..

[8]  Jennifer Neville,et al.  Linkage and Autocorrelation Cause Feature Selection Bias in Relational Learning , 2002, ICML.

[9]  Foster Provost,et al.  A Simple Relational Classifier , 2003 .

[10]  Tom Fawcett,et al.  Adaptive Fraud Detection , 1997, Data Mining and Knowledge Discovery.

[11]  Jennifer Neville,et al.  Why collective inference improves relational classification , 2004, KDD.

[12]  Pedro M. Domingos,et al.  Learning the structure of Markov logic networks , 2005, ICML.

[13]  Jennifer Neville,et al.  Simple estimators for relational Bayesian classifiers , 2003, Third IEEE International Conference on Data Mining.

[14]  Ben Taskar,et al.  Discriminative Probabilistic Models for Relational Data , 2002, UAI.

[15]  Michael I. Jordan,et al.  Loopy Belief Propagation for Approximate Inference: An Empirical Study , 1999, UAI.

[16]  Matthew Richardson,et al.  Mining the network value of customers , 2001, KDD '01.

[17]  Peter A. Flach,et al.  IBC: A First-Order Bayesian Classifier , 1999, ILP.

[18]  Steffen L. Lauritzen,et al.  Graphical Models for Genetic Analyses , 2003 .

[19]  J. Besag Statistical Analysis of Non-Lattice Data , 1975 .

[20]  Andrew McCallum,et al.  Efficiently Inducing Features of Conditional Random Fields , 2002, UAI.

[21]  Manfred Jaeger,et al.  Relational Bayesian Networks , 1997, UAI.

[22]  Jennifer Neville,et al.  Dependency networks for relational data , 2004, Fourth IEEE International Conference on Data Mining (ICDM'04).

[23]  Chris Volinsky,et al.  Network-Based Marketing: Identifying Likely Adopters Via Consumer Networks , 2006, math/0606278.

[24]  David Heckerman,et al.  Probabilistic Models for Relational Data , 2004 .

[25]  Foster J. Provost,et al.  Aggregation-based feature invention and relational concept classes , 2003, KDD '03.

[26]  David M. Pennock,et al.  Statistical relational learning for document mining , 2003, Third IEEE International Conference on Data Mining.

[27]  Ben Taskar,et al.  Probabilistic Classification and Clustering in Relational Data , 2001, IJCAI.

[28]  J. Besag Spatial Interaction and the Statistical Analysis of Lattice Systems , 1974 .

[29]  Kristian Kersting Representational Power of Probabilistic-Logical Models : From Upgrading to Downgrading ∗ , 2003 .

[30]  Pedro M. Domingos,et al.  Dynamic Probabilistic Relational Models , 2003, IJCAI.

[31]  B. Gidas Consistency of Maximum Likelihood and Pseudo-Likelihood Estimators for Gibbs Distributions , 1988 .

[32]  Piotr Indyk,et al.  Enhanced hypertext categorization using hyperlinks , 1998, SIGMOD '98.

[33]  Adrian Pagan,et al.  Estimation, Inference and Specification Analysis. , 1996 .

[34]  Jennifer Neville,et al.  Avoiding Bias when Aggregating Relational Data with Degree Disparity , 2003, ICML.

[35]  Jennifer Neville,et al.  Collective Classification with Relational Dependency Networks , 2003 .

[36]  F. Comets On Consistency of a Class of Estimators for Exponential Families of Markov Random Fields on the Lattice , 1992 .

[37]  Jennifer Neville,et al.  Learning relational probability trees , 2003, KDD '03.

[38]  Lise Getoor,et al.  Learning Probabilistic Relational Models , 1999, IJCAI.

[39]  Jennifer Neville,et al.  Iterative Classification in Relational Data , 2000 .

[40]  Bianca Zadrozny,et al.  Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers , 2001, ICML.

[41]  E. L. Lehmann,et al.  Theory of point estimation , 1950 .

[42]  Lise Getoor,et al.  Link-Based Classification , 2003, Encyclopedia of Machine Learning and Data Mining.

[43]  Jennifer Neville,et al.  Bias / Variance Analysis for Network Data , 2006 .

[44]  David Maxwell Chickering,et al.  Dependency Networks for Inference, Collaborative Filtering, and Data Visualization , 2000, J. Mach. Learn. Res..

[45]  Matthew Richardson,et al.  Markov logic networks , 2006, Machine Learning.

[46]  Neil Immerman,et al.  A Visual Query Language for Relational Knowledge Discovery TITLE2 , 2001 .

[47]  G. Casella,et al.  Statistical Inference , 2003, Encyclopedia of Social Network Analysis and Mining.

[48]  Jennifer Neville,et al.  Using relational knowledge discovery to prevent securities fraud , 2005, KDD '05.

[49]  Abraham Bernstein,et al.  The Relational Vector-Space Model , 2003 .