Patches, Planes and Probabilities: A Non-Local Prior for Volumetric 3D Reconstruction

In this paper, we propose a non-local structured prior for volumetric multi-view 3D reconstruction. Towards this goal, we present a novel Markov random field model based on ray potentials in which assumptions about large 3D surface patches such as planarity or Manhattan world constraints can be efficiently encoded as probabilistic priors. We further derive an inference algorithm that reasons jointly about voxels, pixels and image segments, and estimates marginal distributions of appearance, occupancy, depth, normals and planarity. Key to tractable inference is a novel hybrid representation that spans both voxel and pixel space and that integrates non-local information from 2D image segmentations in a principled way. We compare our non-local prior to commonly employed local smoothness assumptions and a variety of state-of-the-art volumetric reconstruction baselines on challenging outdoor scenes with textureless and reflective surfaces. Our experiments indicate that regularizing over larger distances has the potential to resolve ambiguities where local regularizers fail.

[1]  Michael J. Black,et al.  Towards Probabilistic Volumetric Reconstruction Using Ray Potentials , 2015, 2015 International Conference on 3D Vision.

[2]  Marc Pollefeys,et al.  Discrete optimization of ray potentials for semantic 3D reconstruction , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Carlos Hernandez,et al.  Multi-View Stereo: A Tutorial , 2015, Found. Trends Comput. Graph. Vis..

[4]  H. Maitre,et al.  Using surface model to correct and fit disparity data in stereo vision , 1990, [1990] Proceedings. 10th International Conference on Pattern Recognition.

[5]  Adrien Bartoli,et al.  A random sampling strategy for piecewise planar scene segmentation , 2007, Comput. Vis. Image Underst..

[6]  Florent Lafarge,et al.  A Hybrid Multiview Stereo Algorithm for Modeling Urban Scenes , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[7]  Richard Szeliski,et al.  Manhattan-world stereo , 2009, CVPR.

[8]  Nikos Komodakis,et al.  Beyond pairwise energies: Efficient optimization for higher-order MRFs , 2009, CVPR.

[9]  J. Mixter Fast , 2012 .

[10]  Jean-Philippe Pons,et al.  Robust piecewise-planar 3D reconstruction and completion from large-scale unstructured point data , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[11]  Nir Friedman,et al.  Probabilistic Graphical Models - Principles and Techniques , 2009 .

[12]  Pau Gargallo,et al.  An Occupancy-Depth Generative Model of Multi-view Images , 2007, ACCV.

[13]  R. Cipolla,et al.  A probabilistic framework for space carving , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[14]  Andrew Calway,et al.  Dense 3-D Structure from Image Sequences Using Probabilistic Depth Carving , 2003, BMVC.

[15]  Silvio Savarese,et al.  Dense Object Reconstruction with Semantic Priors , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[16]  Marc Pollefeys,et al.  Joint 3D Scene Reconstruction and Class Segmentation , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[17]  Richard Szeliski,et al.  A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[18]  Marc Pollefeys,et al.  A Patch Prior for Dense 3D Reconstruction in Man-Made Environments , 2012, 2012 Second International Conference on 3D Imaging, Modeling, Processing, Visualization & Transmission.

[19]  Michael J. Black,et al.  Fields of Experts , 2009, International Journal of Computer Vision.

[20]  Andreas Geiger,et al.  Displets: Resolving stereo ambiguities using object knowledge , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Richard Szeliski,et al.  Piecewise planar stereo for image-based rendering , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[22]  Horst Bischof,et al.  Fusion of Feature- and Area-Based Information for Urban Buildings Modeling from Aerial Imagery , 2008, ECCV.

[23]  Michael J. Black,et al.  Preserving Modes and Messages via Diverse Particle Selection , 2014, ICML.

[24]  Konrad Schindler,et al.  Piecewise planar scene reconstruction from sparse correspondences , 2006, Image Vis. Comput..

[25]  Ian D. Reid,et al.  Dense Reconstruction Using 3D Object Shape Priors , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[26]  Andreas Geiger,et al.  Omnidirectional 3D reconstruction in augmented Manhattan worlds , 2014, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[27]  Kiriakos N. Kutulakos,et al.  A Probabilistic Theory of Occupancy and Emptiness , 2002, ECCV.

[28]  Jeremy S. De Bonet,et al.  Poxels: Probabilistic Voxelized Volume Reconstruction , 1999 .

[29]  Jan-Michael Frahm,et al.  Real-Time Plane-Sweeping Stereo with Multiple Sweeping Directions , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[30]  David A. McAllester,et al.  Particle Belief Propagation , 2009, AISTATS.

[31]  Raquel Urtasun,et al.  Robust Monocular Epipolar Flow Estimation , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[32]  John J. Leonard,et al.  A Mixture of Manhattan Frames: Beyond the Manhattan World , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[33]  Frank Dellaert,et al.  A Rao-Blackwellized MCMC algorithm for recovering piecewise planar 3D models from multiple view RGBD images , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[34]  Jan-Michael Frahm,et al.  Piecewise planar and non-planar stereo for urban scene reconstruction , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[35]  Pushmeet Kohli,et al.  Associative Hierarchical Random Fields , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[36]  Pushmeet Kohli,et al.  Robust Higher Order Potentials for Enforcing Label Consistency , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[37]  Andreas Geiger,et al.  Exploiting Object Similarity in 3D Reconstruction , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[38]  Zhuowen Tu,et al.  Image Segmentation by Data-Driven Markov Chain Monte Carlo , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[39]  Andrew Owens,et al.  Shape Anchors for Data-Driven Multi-view Reconstruction , 2013, 2013 IEEE International Conference on Computer Vision.

[40]  Ronald E. Goldstein,et al.  Principles and techniques , 2009 .

[41]  Tamir Hazan,et al.  Continuous Markov Random Fields for Robust Stereo Estimation , 2012, ECCV.

[42]  Steven M. Seitz,et al.  Multicore bundle adjustment , 2011, CVPR 2011.

[43]  András Bódis-Szomorú,et al.  Fast, Approximate Piecewise-Planar Modeling Based on Sparse Structure-from-Motion and Superpixels , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[44]  Jan Kautz,et al.  PMBP: PatchMatch Belief Propagation for Correspondence Field Estimation , 2014, International Journal of Computer Vision.

[45]  Joseph L. Mundy,et al.  Dynamic Probabilistic Volumetric Models , 2013, 2013 IEEE International Conference on Computer Vision.

[46]  D. Cooper,et al.  Statistical Inverse Ray Tracing for Image-Based 3D Modeling , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[47]  Steven M. Seitz,et al.  Photorealistic Scene Reconstruction by Voxel Coloring , 1997, International Journal of Computer Vision.

[48]  Larry S. Davis,et al.  A probabilistic framework for surface reconstruction from multiple images , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[49]  Jana Kosecka,et al.  Multi-view Superpixel Stereo in Urban Environments , 2010, International Journal of Computer Vision.

[50]  Kiriakos N. Kutulakos,et al.  A Theory of Shape by Space Carving , 2000, International Journal of Computer Vision.

[51]  Joseph L. Mundy,et al.  Change Detection in a 3-d World , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[52]  Richard Szeliski,et al.  Reconstructing building interiors from images , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[53]  Gabriel Taubin,et al.  High Resolution Surface Reconstruction from Multi-view Aerial Imagery , 2012, 2012 Second International Conference on 3D Imaging, Modeling, Processing, Visualization & Transmission.

[54]  David Mumford,et al.  Statistics of range images , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[55]  Changchang Wu,et al.  Towards Linear-Time Incremental Structure from Motion , 2013, 2013 International Conference on 3D Vision.

[56]  Marc Pollefeys,et al.  Class Specific 3D Object Shape Priors Using Surface Normals , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[57]  Pushmeet Kohli,et al.  Non-parametric Higher-Order Random Fields for Image Segmentation , 2014, ECCV.