Specular Flow and the Recovery of Surface Structure

In scenes containing specular objects, the image motion observed by a moving camera may be an intermixed combination of optical flow resulting from diffuse reflectance (diffuse flow) and specular reflection (specular flow). Here, with few assumptions, we formalize the notion of specular flow, show how it relates to the 3D structure of the world, and develop an algorithm for estimating scene structure from 2D image motion. Unlike previous work on isolated specular highlights we use two image frames and estimate the semi-dense flow arising from the specular reflections of textured scenes. We parametrically model the image motion of a quadratic surface patch viewed from a moving camera. The flow is modeled as a probabilistic mixture of diffuse and specular components and the 3D shape is recovered using an Expectation-Maximization algorithm. Rather than treating specular reflections as noise to be removed or ignored, we show that the specular flow provides additional constraints on scene geometry that improve estimation of 3D structure when compared with reconstruction from diffuse flow alone. We demonstrate this for a set of synthetic and real sequences of mixed specular-diffuse objects.

[1]  KanadeTakeo,et al.  Adapting optical-flow to measure object motion in reflectance and x-ray image sequences (abstract only) , 1984 .

[2]  C. Dyer,et al.  Dynamic shading, motion parallax and qualitative shape , 1993, [1993] Proceedings IEEE Workshop on Qualitative Vision.

[3]  Richard Szeliski,et al.  Video and Image Bayesian Demosaicing with a Two Color Image Prior , 2006, ECCV.

[4]  John D. Smith The remarkable Ibn al-Haytham , 1992, The Mathematical Gazette.

[5]  Takeo Kanade,et al.  Determining shape and reflectance of Lambertian, specular, and hybrid surfaces using extended sources , 1989, International Workshop on Industrial Applications of Machine Intelligence and Vision,.

[6]  P. Perona,et al.  Local analysis for 3D reconstruction of specular surfaces , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[7]  Katsushi Ikeuchi,et al.  Determining Surface Orientations of Specular Surfaces by Using the Photometric Stereo Method , 1981, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Michael J. Black,et al.  The Robust Estimation of Multiple Motions: Parametric and Piecewise-Smooth Flow Fields , 1996, Comput. Vis. Image Underst..

[9]  H. Bülthoff,et al.  Does the brain know the physics of specular reflection? , 1990, Nature.

[10]  Min Chen,et al.  Theory and application of specular path perturbation , 2000, TOGS.

[11]  Michal Irani,et al.  Separating Transparent Layers through Layer Information Exchange , 2004, ECCV.

[12]  Henrik Aanæs,et al.  A variational analysis of shape from specularities using sparse data , 2004, Proceedings. 2nd International Symposium on 3D Data Processing, Visualization and Transmission, 2004. 3DPVT 2004..

[13]  Shree K. Nayar,et al.  A Theory of Specular Surface Geometry , 2004, International Journal of Computer Vision.

[14]  Ramesh C. Jain,et al.  Segmentation through Variable-Order Surface Fitting , 1988, IEEE Trans. Pattern Anal. Mach. Intell..

[15]  M. Shizawa,et al.  Principle of superposition: a common computational framework for analysis of multiple motion , 1991, Proceedings of the IEEE Workshop on Visual Motion.

[16]  Andrew Blake,et al.  Geometry From Specularities , 1988, [1988 Proceedings] Second International Conference on Computer Vision.

[17]  Andrew Blake,et al.  Specular Stereo , 1985, IJCAI.

[18]  Shmuel Peleg,et al.  A Three-Frame Algorithm for Estimating Two-Component Image Motion , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[19]  David J. Fleet,et al.  Performance of optical flow techniques , 1994, International Journal of Computer Vision.

[20]  Stephen Lin,et al.  Diffuse-Specular Separation and Depth Recovery from Image Sequences , 2002, ECCV.

[21]  Harpreet S. Sawhney,et al.  Compact Representations of Videos Through Dominant and Multiple Motion Estimation , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[22]  Andrew Blake,et al.  The information available to a moving observer from specularities , 1989, Image and Vision Computing.

[23]  Pietro Perona,et al.  Local Analysis for 3D Reconstruction of Specular Surfaces - Part II , 2002, ECCV.

[24]  Jean-Yves Bouguet,et al.  Camera calibration toolbox for matlab , 2001 .

[25]  Richard Szeliski,et al.  Layer extraction from multiple images containing reflections and transparency , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[26]  Peter M. Neumann,et al.  REFLECTIONS ON REFLECTION IN A SPHERICAL MIRROR , 1998 .

[27]  Stefano Soatto,et al.  Variational multiframe stereo in the presence of specular reflections , 2002, Proceedings. First International Symposium on 3D Data Processing Visualization and Transmission.

[28]  Thomas Brox,et al.  High Accuracy Optical Flow Estimation Based on a Theory for Warping , 2004, ECCV.

[29]  Thomas O. Binford,et al.  Local shape from specularity , 1988, Comput. Vis. Graph. Image Process..

[30]  Richard Szeliski,et al.  On the Motion and Appearance of Specularities in Image Sequences , 2002, ECCV.

[31]  Takeo Kanade,et al.  Adapting optical-flow to measure object motion in reflectance and x-ray image sequences (abstract only) , 1984, COMG.

[32]  Geoffrey E. Hinton,et al.  A View of the Em Algorithm that Justifies Incremental, Sparse, and other Variants , 1998, Learning in Graphical Models.