Learning Optical Flow

Assumptions of brightness constancy and spatial smoothness underlie most optical flow estimation methods. In contrast to standard heuristic formulations, we learn a statistical model of both brightness constancy error and the spatial properties of optical flow using image sequences with associated ground truth flow fields. The result is a complete probabilistic model of optical flow. Specifically, the ground truth enables us to model how the assumption of brightness constancy is violated in naturalistic sequences, resulting in a probabilistic model of "brightness inconstancy". We also generalize previous high-order constancy assumptions, such as gradient constancy, by modeling the constancy of responses to various linear filters in a high-order random field framework. These filters are free variables that can be learned from training data. Additionally we study the spatial structure of the optical flow and how motion boundaries are related to image intensity boundaries. Spatial smoothness is modeled using a Steerable Random Field, where spatial derivatives of the optical flow are steered by the image brightness structure. These models provide a statistical motivation for previous methods and enable the learning of all parameters from training data. All proposed models are quantitatively compared on the Middlebury flow dataset.

[1]  Berthold K. P. Horn,et al.  Determining Optical Flow , 1981, Other Conferences.

[2]  Edward H. Adelson,et al.  PYRAMID METHODS IN IMAGE PROCESSING. , 1984 .

[3]  Hans-Hellmut Nagel,et al.  An Investigation of Smoothness Constraints for the Estimation of Displacement Vector Fields from Image Sequences , 1983, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[4]  Andrew Blake,et al.  Visual Reconstruction , 1987, Deep Learning for EEG-Based Brain–Computer Interfaces.

[5]  S. Negahdaripour,et al.  Relaxing the Brightness Constancy Assumption in Computing Optical Flow , 1987 .

[6]  Edward H. Adelson,et al.  Probability distributions of optical flow , 1991, Proceedings. 1991 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[7]  G. Sandini,et al.  Computer Vision — ECCV'92 , 1992, Lecture Notes in Computer Science.

[8]  Michael J. Black Combining Intensity and Motion for Incremental Segmentation and Tracking Over Long Image Sequences , 1992, ECCV.

[9]  Michael J. Black,et al.  The Robust Estimation of Multiple Motions: Parametric and Piecewise-Smooth Flow Fields , 1996, Comput. Vis. Image Underst..

[10]  David J. Fleet,et al.  Learning parameterized models of image motion , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[11]  Song-Chun Zhu Filters, Random Fields and Maximum Entropy (FRAME): Towards a Unified Theory for Texture Modeling , 1998 .

[12]  Martin J. Wainwright,et al.  Scale Mixtures of Gaussians and the Statistics of Natural Images , 1999, NIPS.

[13]  Til Aach,et al.  Illumination-invariant change detection , 2000, 4th IEEE Southwest Symposium on Image Analysis and Interpretation.

[14]  Yiannis Aloimonos,et al.  The Statistics of Optical Flow , 2001, Comput. Vis. Image Underst..

[15]  David J. Fleet,et al.  Computing Optical Flow with Physical Models of Brightness Variation , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[16]  Geoffrey E. Hinton Training Products of Experts by Minimizing Contrastive Divergence , 2002, Neural Computation.

[17]  David J. Fleet,et al.  Bayesian inference of visual motion boundaries , 2003 .

[18]  Gerhard Lakemeyer,et al.  Exploring artificial intelligence in the new millennium , 2003 .

[19]  Jiří Matas,et al.  Computer Vision - ECCV 2004 , 2004, Lecture Notes in Computer Science.

[20]  William T. Freeman,et al.  c ○ 2000 Kluwer Academic Publishers. Manufactured in The Netherlands. Learning Low-Level Vision , 2022 .

[21]  Thomas Brox,et al.  High Accuracy Optical Flow Estimation Based on a Theory for Warping , 2004, ECCV.

[22]  Joachim Weickert,et al.  Lucas/Kanade Meets Horn/Schunck: Combining Local and Global Optic Flow Methods , 2005, International Journal of Computer Vision.

[23]  Michael J. Black,et al.  Fields of Experts: a framework for learning image priors , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[24]  Michael J. Black,et al.  On the Spatial Statistics of Optical Flow , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[25]  Rachid Deriche,et al.  Symmetrical Dense Optical Flow Estimation with Occlusions Detection , 2002, International Journal of Computer Vision.

[26]  Richard Szeliski,et al.  A Database and Evaluation Methodology for Optical Flow , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[27]  Christopher Joseph Pal,et al.  On Learning Conditional Random Fields for Stereo , 2007, International Journal of Computer Vision.

[28]  Michael J. Black,et al.  Steerable Random Fields , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[29]  Richard S. Zemel,et al.  Learning Flexible Features for Conditional Random Fields , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  Carsten Rother,et al.  FusionFlow: Discrete-continuous optimization for optical flow estimation , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[31]  Daniel Cremers,et al.  Continuous Energy Minimization Via Repeated Binary Fusion , 2008, ECCV.

[32]  Daniel P. Huttenlocher,et al.  Learning for Optical Flow Using Stochastic Optimization , 2008, ECCV.

[33]  Nassir Navab,et al.  Optical flow estimation with uncertainties through dynamic MRFs , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[34]  Carsten Rother,et al.  Discrete-Continuous Optimization for Optical Flow Estimation , 2009, Statistical and Geometrical Approaches to Visual Motion Analysis.

[35]  Li Xu,et al.  A Segmentation Based Variational Model for Accurate Optical Flow Estimation , 2008, ECCV.