Structure of the high-order Boltzmann machine from independence maps

In this paper we consider the determination of the structure of the high-order Boltzmann machine (HOBM), a stochastic recurrent network for approximating probability distributions. We obtain the structure of the HOBM, the hypergraph of connections, from conditional independences of the probability distribution to model. We assume that an expert provides these conditional independences and from them we build independence maps, Markov and Bayesian networks, which represent conditional independences through undirected graphs and directed acyclic graphs respectively. From these independence maps we construct the HOBM hypergraph. The central aim of this paper is to obtain a minimal hypergraph. Given that different orderings of the variables provide in general different Bayesian networks, we define their intersection hypergraph. We prove that the intersection hypergraph of all the Bayesian networks (N!) of the distribution is contained by the hypergraph of the Markov network, it is more simple, and we give a procedure to determine a subset of the Bayesian networks that verifies this property. We also prove that the Markov network graph establishes a minimum connectivity for the hypergraphs from Bayesian networks.

[1]  Frank Harary,et al.  Graph Theory , 2016 .

[2]  Catriel Beeri,et al.  On the Desirability of Acyclic Database Schemes , 1983, JACM.

[3]  V. Isham An Introduction to Spatial Point Processes and Markov Random Fields , 1981 .

[4]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[5]  Geoffrey E. Hinton,et al.  Learning and relearning in Boltzmann machines , 1986 .

[6]  Geoffrey E. Hinton,et al.  A Learning Algorithm for Boltzmann Machines , 1985, Cogn. Sci..

[7]  T. Sejnowski Higher‐order Boltzmann machines , 1987 .

[8]  Judea Pearl,et al.  Causal networks: semantics and expressiveness , 2013, UAI.

[9]  Manuel Graña,et al.  The high-order Boltzmann machine: learned distribution and topology , 1995, IEEE Trans. Neural Networks.

[10]  N. Wermuth,et al.  Graphical and recursive models for contingency tables , 1983 .

[11]  J. N. R. Jeffers,et al.  Graphical Models in Applied Multivariate Statistics. , 1990 .

[12]  Manuel Graña,et al.  Convergence Properties of High-order Boltzmann Machines , 1996, Neural Networks.

[13]  Robert E. Tarjan,et al.  Simple Linear-Time Algorithms to Test Chordality of Graphs, Test Acyclicity of Hypergraphs, and Selectively Reduce Acyclic Hypergraphs , 1984, SIAM J. Comput..

[14]  J. M. Hammersley,et al.  Markov fields on finite graphs and lattices , 1971 .

[15]  T. Speed,et al.  Markov Fields and Log-Linear Interaction Models for Contingency Tables , 1980 .