Fast Exact Inference for Recursive Cardinality Models

Cardinality potentials are a generally useful class of high order potential that affect probabilities based on how many of D binary variables are active. Maximum a posteriori (MAP) inference for cardinality potential models is well-understood, with efficient computations taking O(D log D) time. Yet efficient marginalization and sampling have not been addressed as thoroughly in the machine learning community. We show that there exists a simple algorithm for computing marginal probabilities and drawing exact joint samples that runs in O(D log2 D) time, and we show how to frame the algorithm as efficient belief propagation in a low order tree-structured model that includes additional auxiliary variables. We then develop a new, more general class of models, termed Recursive Cardinality models, which take advantage of this efficiency. Finally, we show how to do efficient exact inference in models composed of a tree structure and a cardinality potential. We explore the expressive power of Recursive Cardinality models and empirically demonstrate their utility.

[1]  M. Fisher Statistical Mechanics of Dimers on a Plane Lattice , 1961 .

[2]  Robert G. Gallager,et al.  Low-density parity-check codes , 1962, IRE Trans. Inf. Theory.

[3]  M. Gail,et al.  Likelihood calculations for matched case-control studies and survival studies with tied death times , 1981 .

[4]  R. E. Barlow,et al.  Computing k-out-of-n System Reliability , 1984, IEEE Transactions on Reliability.

[5]  I. L. Belfore An O(n/spl middot/(log/sub 2/(n))/sup 2/) algorithm for computing the reliability of k-out-of-n:G and k-to-l-out-of-n:G systems , 1995 .

[6]  Jean-Charles Régin,et al.  Generalized Arc Consistency for Global Cardinality Constraint , 1996, AAAI/IAAI, Vol. 1.

[7]  Thomas G. Dietterich,et al.  Solving the Multiple Instance Problem with Axis-Parallel Rectangles , 1997, Artif. Intell..

[8]  Jun S. Liu,et al.  STATISTICAL APPLICATIONS OF THE POISSON-BINOMIAL AND CONDITIONAL BERNOULLI DISTRIBUTIONS , 1997 .

[9]  Daniel P. Huttenlocher,et al.  Efficient Belief Propagation for Early Vision , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[10]  Dan Roth,et al.  Lifted First-Order Probabilistic Inference , 2005, IJCAI.

[11]  Peter V. Gehler,et al.  Deterministic Annealing for Multiple-Instance Learning , 2007, AISTATS.

[12]  Bert Huang,et al.  Loopy Belief Propagation for Bipartite Maximum Weight b-Matching , 2007, AISTATS.

[13]  P. Zhao,et al.  Grouped and Hierarchical Model Selection through Composite Absolute Penalties , 2007 .

[14]  Rahul Gupta,et al.  Efficient inference with cardinality-based clique potentials , 2007, ICML '07.

[15]  Leslie Pack Kaelbling,et al.  Lifted Probabilistic Inference with Counting Formulas , 2008, AAAI.

[16]  Pushmeet Kohli,et al.  Exact inference in multi-label CRFs with higher order cliques , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[17]  Tai Sing Lee,et al.  Efficient belief propagation for higher-order cliques using linear constraint nodes , 2008, Comput. Vis. Image Underst..

[18]  Pushmeet Kohli,et al.  Robust Higher Order Potentials for Enforcing Label Consistency , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[19]  Pushmeet Kohli,et al.  Minimizing sparse higher order energy functions of discrete variables , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[20]  Vladimir Kolmogorov,et al.  Joint optimization of segmentation and appearance models , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[21]  Andreas Krause,et al.  Efficient Minimization of Decomposable Submodular Functions , 2010, NIPS.

[22]  Richard S. Zemel,et al.  HOP-MAP: Efficient Message Passing with High Order Potentials , 2010, AISTATS.

[23]  Nando de Freitas,et al.  Inductive Principles for Restricted Boltzmann Machine Learning , 2010, AISTATS.

[24]  M. Fernandez,et al.  Closed-Form Expression for the Poisson-Binomial Probability Density Function , 2010, IEEE Transactions on Aerospace and Electronic Systems.

[25]  Ben Taskar,et al.  Posterior Regularization for Structured Latent Variable Models , 2010, J. Mach. Learn. Res..

[26]  Stephen Gould,et al.  Max-margin Learning for Lower Linear Envelope Potentials in Binary Markov Random Fields , 2011, ICML.

[27]  Yair Weiss,et al.  Globally Optimizing Graph Partitioning Problems Using Message Passing , 2012, AISTATS.