Modeling Blurred Video with Layers

Videos contain complex spatially-varying motion blur due to the combination of object motion, camera motion, and depth variation with finite shutter speeds. Existing methods to estimate optical flow, deblur the images, and segment the scene fail in such cases. In particular, boundaries between differently moving objects cause problems, because here the blurred images are a combination of the blurred appearances of multiple surfaces. We address this with a novel layered model of scenes in motion. From a motion-blurred video sequence, we jointly estimate the layer segmentation and each layer’s appearance and motion. Since the blur is a function of the layer motion and segmentation, it is completely determined by our generative model. Given a video, we formulate the optimization problem as minimizing the pixel error between the blurred frames and images synthesized from the model, and solve it using gradient descent. We demonstrate our approach on synthetic and real sequences.

[1]  Brendan J. Frey,et al.  Learning appearance and transparency manifolds of occluded objects in layers , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[2]  Peyman Milanfar,et al.  Removing Motion Blur With Space–Time Processing , 2011, IEEE Transactions on Image Processing.

[3]  Stefano Soatto,et al.  Dynamic Shape and Appearance Modeling Via Moving and Deforming Layers , 2005, EMMCVPR.

[4]  Ying Wu,et al.  Removing partial blur in a single image , 2009, CVPR.

[5]  Michael J. Black,et al.  Layered segmentation and optical flow estimation over time , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[6]  William T. Freeman,et al.  Analyzing spatially-varying blur , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[7]  Chengtao Cai,et al.  Motion deblurring from a single image , 2016, 2016 IEEE 20th International Conference on Computer Supported Cooperative Work in Design (CSCWD).

[8]  Roberto Cipolla,et al.  Computer Vision — ECCV '96 , 1996, Lecture Notes in Computer Science.

[9]  Hai Tao,et al.  A background layer model for object tracking through occlusion , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[10]  Mads Nielsen,et al.  Computer Vision — ECCV 2002 , 2002, Lecture Notes in Computer Science.

[11]  Guillermo Sapiro,et al.  A Variational Framework for Simultaneous Motion Estimation and Restoration of Motion-Blurred Video , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[12]  Michael S. Brown,et al.  Motion Regularization for Matting Motion Blurred Objects , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Daniel P. Huttenlocher,et al.  Generating sharp panoramas from motion-blurred videos , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[14]  X.C. He,et al.  Motion estimation method for blurred videos and application of deblurring with spatially varying blur kernels , 2010, 5th International Conference on Computer Sciences and Convergence Information Technology.

[15]  Michel Barlaud,et al.  Two deterministic half-quadratic regularization algorithms for computed imaging , 1994, Proceedings of 1st International Conference on Image Processing.

[16]  Radim Sára,et al.  A Weak Structure Model for Regular Pattern Recognition Applied to Facade Images , 2010, ACCV.

[17]  William T. Freeman,et al.  Removing camera shake from a single photograph , 2006, SIGGRAPH 2006.

[18]  Sang Hwa Lee,et al.  Recovery of blurred video signals using iterative image restoration combined with motion estimation , 1997, Proceedings of International Conference on Image Processing.

[19]  Anat Levin,et al.  Blind Motion Deblurring Using Image Statistics , 2006, NIPS.

[20]  A. N. Rajagopalan,et al.  Non-uniform Motion Deblurring for Bilayer Scenes , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[21]  Stefano Soatto,et al.  Dynamic Shape and Appearance Modeling via Moving and Deforming Layers , 2005, International Journal of Computer Vision.

[22]  Michael J. Black,et al.  A Fully-Connected Layered Model of Foreground and Background Flow , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[23]  Wei Xiong,et al.  Rotational Motion Deblurring of a Rigid Object from a Single Image , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[24]  Seungyong Lee,et al.  Video deblurring for hand-held cameras using patch-based synthesis , 2012, ACM Trans. Graph..

[25]  Brendan J. Frey,et al.  Learning flexible sprites in video layers , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[26]  Michael J. Black,et al.  The Dense Estimation of Motion and Appearance in Layers , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[27]  Edward H. Adelson,et al.  Representing moving images with layers , 1994, IEEE Trans. Image Process..

[28]  Andrew Blake,et al.  Motion Deblurring and Super-resolution from an Image Sequence , 1996, ECCV.

[29]  Vladimir Kolmogorov,et al.  What energy functions can be minimized via graph cuts? , 2002, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  Jean Ponce,et al.  Non-uniform Deblurring for Shaken Images , 2012, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[31]  Andrew Zisserman,et al.  Learning Layered Motion Segmentations of Video , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[32]  Thomas S. Huang,et al.  Image processing , 1971 .

[33]  Steven M. Seitz,et al.  Filter flow , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[34]  Yasuyuki Matsushita,et al.  Motion detail preserving optical flow estimation , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[35]  Daniel Cremers,et al.  Variational space-time motion segmentation , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[36]  Ying Wu,et al.  Motion from blur , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[37]  Irfan A. Essa,et al.  Calibration-free rolling shutter removal , 2012, 2012 IEEE International Conference on Computational Photography (ICCP).

[38]  Sunghyun Cho,et al.  Fast motion deblurring , 2009, SIGGRAPH 2009.

[39]  Pushmeet Kohli,et al.  Unwrap mosaics: a new representation for video editing , 2008, SIGGRAPH 2008.

[40]  Takuma Yamaguchi,et al.  Video Deblurring and Super-Resolution Technique for Multiple Moving Objects , 2010, ACCV.

[41]  Yair Weiss,et al.  Smoothness in layers: Motion segmentation using nonparametric mixture estimation , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[42]  Worthy N. Martin,et al.  Image Motion Estimation From Motion Smear-A New Computational Model , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[43]  D. A. Fish,et al.  Blind deconvolution by means of the Richardson-Lucy algorithm. , 1995 .

[44]  Alfred M. Bruckstein,et al.  Variational Approach for Joint Optic-Flow Computation and Video Restoration , 2005 .

[45]  Li Zhang,et al.  Optical flow in the presence of spatially-varying motion blur , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[46]  Anita Sellent,et al.  Motion Field Estimation from Alternate Exposure Images , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[47]  David J. Fleet,et al.  A Layered Motion Representation with Occlusion and Compact Spatial Support , 2002, ECCV.

[48]  Takeo Kanade,et al.  Super-Resolution Optical Flow , 1999 .

[49]  Roberto Cipolla,et al.  Visual tracking in the presence of motion blur , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[50]  Daniel Cremers,et al.  A Coding-Cost Framework for Super-Resolution Motion Layer Decomposition , 2012, IEEE Transactions on Image Processing.

[51]  Richard Szeliski,et al.  An integrated Bayesian approach to layer extraction from image sequences , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.