A Comparison of Algorithms for Learning Hidden Variables in Normal Graphs

A Bayesian factor graph reduced to normal form consists in the interconnection of diverter units (or equal constraint units) and Single-Input/Single-Output (SISO) blocks. In this framework localized adaptation rules are explicitly derived from a constrained maximum likelihood (ML) formulation and from a minimum KL-divergence criterion using KKT conditions. The learning algorithms are compared with two other updating equations based on a Viterbi-like and on a variational approximation respectively. The performance of the various algorithm is verified on synthetic data sets for various architectures. The objective of this paper is to provide the programmer with explicit algorithms for rapid deployment of Bayesian graphs in the applications.

[1]  Mordecai Avriel,et al.  Nonlinear programming , 1976 .

[2]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[3]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[4]  Michael I. Jordan.  Learning in Graphical Models , 1999, NATO ASI Series.

[5]  H. Sebastian Seung,et al.  Algorithms for Non-negative Matrix Factorization , 2000, NIPS.

[6]  G. Forney,et al.  Codes on graphs: normal realizations , 2000, 2000 IEEE International Symposium on Information Theory (Cat. No.00CH37060).

[7]  Brendan J. Frey,et al.  Factor graphs and the sum-product algorithm , 2001, IEEE Trans. Inf. Theory.

[8]  Nevin Lianwen Zhang,et al.  Hierarchical latent class models for cluster analysis , 2002, J. Mach. Learn. Res..

[9]  X. Jin.  Factor graphs and the Sum-Product Algorithm , 2002 .

[10]  John M. Winn,et al.  Variational Message Passing and its Applications , 2004 .

[11]  Stuart J. Russell,et al.  Adaptive Probabilistic Networks with Hidden Variables , 1997, Machine Learning.

[12]  H.-A. Loeliger,et al.  An introduction to factor graphs , 2004, IEEE Signal Processing Magazine.

[13]  Charles M. Bishop,et al.  Variational Message Passing , 2005, J. Mach. Learn. Res..

[14]  William T. Freeman,et al.  Constructing free-energy approximations and generalized belief propagation algorithms , 2005, IEEE Transactions on Information Theory.

[15]  Matthew J. Beal,et al.  Variational Bayesian learning of directed graphical models with hidden variables , 2006 .

[16]  Radford M. Neal.  Pattern Recognition and Machine Learning , 2007, Technometrics.

[17]  Justin Dauwels,et al.  On Variational Message Passing on Factor Graphs , 2007, 2007 IEEE International Symposium on Information Theory.

[18]  Richard E. Neapolitan,et al.  Learning Bayesian networks , 2007, KDD '07.

[19]  Yoshua. Bengio,et al.  Learning Deep Architectures for AI , 2007, Found. Trends Mach. Learn..

[20]  David Heckerman,et al.  A Tutorial on Learning with Bayesian Networks , 1999, Innovations in Bayesian Networks.

[21]  L. Williams,et al.  Contents , 2020, Ophthalmology (Rochester, Minn.).

[22]  P. Dooren,et al.  Non-negative matrix factorization with fixed row and column sums , 2008 .

[23]  Nir Friedman,et al.  Probabilistic Graphical Models - Principles and Techniques , 2009 .

[24]  Francesco Palmieri,et al.  Building a bayesian factor tree from examples , 2010, 2010 2nd International Workshop on Cognitive Information Processing.

[25]  Martin J. Wainwright,et al.  Major Advances and Emerging Developments of Graphical Models [From the Guest Editors] , 2010 .

[26]  Christopher K. I. Williams,et al.  Greedy Learning of Binary Latent Trees , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Vincent Y. F. Tan,et al.  High-Dimensional Graphical Model Selection: Tractable Graph Families and Necessary Conditions , 2011, NIPS.

[28]  Francesco Palmieri,et al.  Probability Learning and Soft Quantization in Bayesian Factor Graphs , 2012, WIRN.

[29]  Francesco Palmieri,et al.  Learning Non-Linear Functions With Factor Graphs , 2013, IEEE Transactions on Signal Processing.