Information Geometry on Hierarchical Decomposition of Stochastic Interactions

A joint probability distribution represents stochastic dependency among a number of random variables. They in general have complex structure of dependencies. A pairwise dependency is easily represented by correlation, but it is more diicult to extract triplewise or higher-order interactions (dependencies) among these variables. They form a hierarchy of dependencies. The present paper decomposes the higher order dependency into an "orthog-onal" sum of pairwise, triplewise and further higher-order dependencies, by using Information Geometry. This naturally gives a new invariant decomposition of joint entropy. This problem is important for extracting intrinsic interactions in ring patterns of an ensemble of neurons and for estimating its functional connections. The results are generalized to give an orthogonal decomposition in a wide class of hierarchical structures. As an example, we decompose higher order Markov chains into various lower order Markov 1 chains. This type of decomposition is related to the mean eld approximation in physics.