On the Spatial Statistics of Optical Flow

We present an analysis of the spatial and temporal statistics of “natural” optical flow fields and a novel flow algorithm that exploits their spatial statistics. Training flow fields are constructed using range images of natural scenes and 3D camera motions recovered from hand-held and car-mounted video sequences. A detailed analysis of optical flow statistics in natural scenes is presented and machine learning methods are developed to learn a Markov random field model of optical flow. The prior probability of a flow field is formulated as a Field-of-Experts model that captures the spatial statistics in overlapping patches and is trained using contrastive divergence. This new optical flow prior is compared with previous robust priors and is incorporated into a recent, accurate algorithm for dense optical flow computation. Experiments with natural and synthetic sequences illustrate how the learned optical flow prior quantitatively improves flow accuracy and how it captures the rich spatial structure found in natural scene motion.

[1]  T. Kailath.  The Divergence and Bhattacharyya Distance Measures in Signal Selection , 1967 .

[2]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[3]  Berthold K. P. Horn,et al.  Determining Optical Flow , 1981, Other Conferences.

[4]  Tomaso Poggio,et al.  Probabilistic Solution of Ill-Posed Problems in Computational Vision , 1987 .

[5]  David W. Murray,et al.  Scene Segmentation from Visual Motion Using Global Optimization , 1987, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Eric Dubois,et al.  Multigrid Bayesian Estimation Of Image Motion Using Stochastic Relaxation , 1988, [1988 Proceedings] Second International Conference on Computer Vision.

[7]  Edward H. Adelson,et al.  Probability distributions of optical flow , 1991, Proceedings. 1991 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[8]  Michael J. Black,et al.  Robust dynamic motion estimation over time , 1991, Proceedings. 1991 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[9]  Patrick Bouthemy,et al.  Multimodal Estimation of Discontinuous Optical Flow using Markov Random Fields , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[10]  D. Ruderman.  The statistics of natural images , 1994 .

[11]  Luc Van Gool,et al.  Determination of Optical Flow and its Discontinuities using Non-Linear Diffusion , 1994, ECCV.

[12]  D. Field,et al.  Natural image statistics and efficient coding. , 1996, Network.

[13]  Michael J. Black,et al.  The Robust Estimation of Multiple Motions: Parametric and Piecewise-Smooth Flow Fields , 1996, Comput. Vis. Image Underst..

[14]  Song-Chun Zhu,et al.  Prior Learning and Gibbs Reaction-Diffusion , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[15]  E. Adelson,et al.  Slow and Smooth: A Bayesian theory for the combination of local motion signals in human vision , 1998 .

[16]  D. Ruderman,et al.  INDEPENDENT COMPONENT ANALYSIS OF NATURAL IMAGE SEQUENCES YIELDS SPATIOTEMPORAL FILTERS SIMILAR TO SIMPLE CELLS IN PRIMARY VISUAL CORTEX , 1998 .

[17]  D. Ruderman,et al.  Independent component analysis of natural image sequences yields spatio-temporal filters similar to simple cells in primary visual cortex , 1998, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[18]  Geoffrey E. Hinton.  Products of experts , 1999 .

[19]  Michal Irani,et al.  Multi-frame optical flow estimation using subspace constraints , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[20]  David Mumford,et al.  Statistics of natural images and models , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[21]  David Mumford,et al.  Statistics of range images , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[22]  Anuj Srivastava,et al.  Probability Models for Clutter in Natural Images , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[23]  G D Lewen,et al.  Neural coding of naturalistic motion stimuli , 2001, Network.

[24]  Yiannis Aloimonos,et al.  The Statistics of Optical Flow , 2001, Comput. Vis. Image Underst..

[25]  Jitendra Malik,et al.  A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[26]  Patrick Bouthemy,et al.  Non parametric motion recognition using temporal multiscale Gibbs models , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[27]  Geoffrey E. Hinton.  Training Products of Experts by Minimizing Contrastive Divergence , 2002, Neural Computation.

[28]  David J. Fleet,et al.  Bayesian inference of visual motion boundaries , 2003 .

[29]  Gerhard Lakemeyer,et al.  Exploring artificial intelligence in the new millennium , 2003 .

[30]  Antonio Torralba,et al.  Statistics of natural image categories , 2003, Network.

[31]  Konrad P. Körding,et al.  The world from a cat’s perspective – statistics of natural videos , 2003, Biological Cybernetics.

[32]  Yee Whye Teh,et al.  Energy-Based Models for Sparse Overcomplete Representations , 2003, J. Mach. Learn. Res..

[33]  Eero P. Simoncelli,et al.  On Advances in Statistical Modeling of Natural Images , 2004, Journal of Mathematical Imaging and Vision.

[34]  Patrick Pérez,et al.  Hierarchical Estimation and Segmentation of Dense Motion Fields , 2002, International Journal of Computer Vision.

[35]  David J. Fleet,et al.  Design and Use of Linear Models for Image Motion Analysis , 2000, International Journal of Computer Vision.

[36]  David Mumford,et al.  Occlusion Models for Natural Images: A Statistical Study of a Scale-Invariant Dead Leaves Model , 2004, International Journal of Computer Vision.

[37]  Florentin Wörgötter,et al.  Statistics of optic flow for self-motion through natural scenes , 2004 .

[38]  Hanno Scharr,et al.  Optimal Filters for Extended Optical Flow , 2004, IWCM.

[39]  Michael G. Ross,et al.  A Systematic Approach to Learning Object Segmentation from Motion , 2004 .

[40]  Joachim Weickert,et al.  Reliable Estimation of Dense Optical Flow Fields with Large Displacements , 2000, International Journal of Computer Vision.

[41]  Timothy A. Davis,et al.  A column pre-ordering strategy for the unsymmetric-pattern multifrontal method , 2004, TOMS.

[42]  Song-Chun Zhu,et al.  Filters, Random Fields and Maximum Entropy (FRAME): Towards a Unified Theory for Texture Modeling , 1998, International Journal of Computer Vision.

[43]  Joachim Weickert,et al.  Variational Optic Flow Computation with a Spatio-Temporal Smoothness Constraint , 2001, Journal of Mathematical Imaging and Vision.

[44]  Joachim Weickert,et al.  Lucas/Kanade Meets Horn/Schunck: Combining Local and Global Optic Flow Methods , 2005, International Journal of Computer Vision.

[45]  A. Yuille,et al.  Motion estimation by Swendsen-Wang Cuts , 2004, CVPR 2004.

[46]  Antonio Torralba,et al.  Contextual Priming for Object Detection , 2003, International Journal of Computer Vision.

[47]  Alan L. Yuille,et al.  Ideal Observers for Detecting Motion: Correspondence Noise , 2005, NIPS.

[48]  Hanno Scharr,et al.  Accurate optical flow in noisy image sequences using flow adapted anisotropic diffusion , 2005, Signal Process. Image Commun..

[49]  Michael J. Black,et al.  Fields of Experts: a framework for learning image priors , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[50]  Thomas Brox,et al.  Universität Des Saarlandes Fachrichtung 6.1 – Mathematik Highly Accurate Optic Flow Computation with Theoretically Justified Warping Highly Accurate Optic Flow Computation with Theoretically Justified Warping , 2022 .

[51]  David J. Fleet,et al.  Performance of optical flow techniques , 1994, International Journal of Computer Vision.

[52]  Michael J. Black,et al.  On the Spatial Statistics of Optical Flow , 2005, ICCV.

[53]  Daniel Cremers,et al.  Motion Competition: A Variational Approach to Piecewise Parametric Motion Segmentation , 2005, International Journal of Computer Vision.

[54]  Leslie Pack Kaelbling,et al.  Learning Static Object Segmentation from Motion Segmentation , 2005, AAAI.

[55]  Nir A. Sochen,et al.  A General Framework and New Alignment Criterion for Dense Optical Flow , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[56]  Richard Szeliski,et al.  A Comparative Study of Energy Minimization Methods for Markov Random Fields , 2006, ECCV.

[57]  R. Mester,et al.  On the equivalence of variational and statistical differential motion estimation , 2006, 2006 IEEE Southwest Symposium on Image Analysis and Interpretation.