Exemplar-Based Inpainting from a Variational Point of View

Among all methods for reconstructing missing regions in a digital image, the so-called exemplar-based algorithms are very efficient and often produce striking results. They are based on the simple idea—initially used for texture synthesis—that the unknown part of an image can be reconstructed by simply pasting samples extracted from the known part. Beyond heuristic considerations, there have been very few contributions in the literature to explain from a mathematical point of view the performances of these purely algorithmic and discrete methods. More precisely, a recent paper by Levina and Bickel [Ann. Statist., 34 (2006), pp. 1751–1773] provides a theoretical explanation of their ability to recover very well the texture, but nothing equivalent has been done so far for the recovery of geometry. Our purpose in this paper is twofold: (1) to propose well-posed variational models in the continuous domain that can be naturally associated to exemplar-based algorithms; (2) to investigate their ability to recons...

[1]  Mila Nikolova,et al.  Regularizing Flows for Constrained Matrix-Valued Images , 2004, Journal of Mathematical Imaging and Vision.

[2]  Hans-Peter Seidel,et al.  Image Compression with Anisotropic Diffusion , 2008, Journal of Mathematical Imaging and Vision.

[3]  Michael Ashikhmin,et al.  Synthesizing natural textures , 2001, I3D '01.

[4]  Tony F. Chan,et al.  Non-texture inpainting by curvature-driven diffusions (CDD) , 2001 .

[5]  Tony F. Chan,et al.  Total Variation Wavelet Inpainting , 2006, Journal of Mathematical Imaging and Vision.

[6]  D. Donoho,et al.  Simultaneous cartoon and texture image inpainting using morphological component analysis (MCA) , 2005 .

[7]  Steven J. Ruuth,et al.  Threshold dynamics for shape reconstruction and disocclusion , 2005, IEEE International Conference on Image Processing 2005.

[8]  David Tschumperlé,et al.  Fast Anisotropic Smoothing of Multi-Valued Images using Curvature-Preserving PDE's , 2006, International Journal of Computer Vision.

[9]  Onur G. Guleryuz,et al.  Nonlinear approximation based image recovery using adaptive sparse reconstructions and iterated denoising-part II: adaptive algorithms , 2006, IEEE Transactions on Image Processing.

[10]  Guillermo Sapiro,et al.  Image inpainting , 2000, SIGGRAPH.

[11]  D. Tschumperlé,et al.  Defining Some Variational Methods on the Space of Patches : Applications to Multi-Valued Image Denoising and Registration , 2008 .

[12]  Marc Levoy,et al.  Fast texture synthesis using tree-structured vector quantization , 2000, SIGGRAPH.

[13]  L. Ambrosio,et al.  Gradient Flows: In Metric Spaces and in the Space of Probability Measures , 2005 .

[14]  A one-shot inpainting algorithm based on the topological asymptotic analysis , 2006 .

[15]  Baining Guo,et al.  Real-time texture synthesis by patch-based sampling , 2001, TOGS.

[16]  Baining Guo,et al.  Chaos Mosaic: Fast and Memory Efficient Texture Synthesis , 2000 .

[17]  G. D. Maso,et al.  An Introduction to-convergence , 1993 .

[18]  Nikos Komodakis,et al.  Image Completion Using Efficient Belief Propagation Via Priority Scheduling and Dynamic Pruning , 2007, IEEE Transactions on Image Processing.

[19]  Manuel Menezes de Oliveira Neto,et al.  Fast Digital Image Inpainting , 2001, VIIP.

[20]  I. Daubechiesa,et al.  Variational image restoration by means of wavelets : Simultaneous decomposition , deblurring , and denoising , 2005 .

[21]  Yves Meyer,et al.  Oscillating Patterns in Image Processing and Nonlinear Evolution Equations: The Fifteenth Dean Jacqueline B. Lewis Memorial Lectures , 2001 .

[22]  Pierre Kornprobst,et al.  Mathematical problems in image processing - partial differential equations and the calculus of variations , 2010, Applied mathematical sciences.

[23]  Y. Meyer Oscillating Patterns in Some Nonlinear Evolution Equations , 2006 .

[24]  Harry Shum,et al.  Image completion with structure propagation , 2005, ACM Trans. Graph..

[25]  S. Osher,et al.  IMAGE DECOMPOSITION AND RESTORATION USING TOTAL VARIATION MINIMIZATION AND THE H−1 NORM∗ , 2002 .

[26]  Otmar Scherzer,et al.  Using the Complex Ginzburg-Landau Equation for Digital Inpainting in 2D and 3D , 2003, Scale-Space.

[27]  Guy Gilboa,et al.  Nonlocal Operators with Applications to Image Processing , 2008, Multiscale Model. Simul..

[28]  Kyōto Daigaku. Sūgakuka Lectures in mathematics , 1968 .

[29]  Sung Yong Shin,et al.  On pixel-based texture synthesis by non-parametric sampling , 2006, Comput. Graph..

[30]  Andrew Blake,et al.  PatchWorks: Example-Based Region Tiling for Image Editing , 2004 .

[31]  Guillermo Sapiro,et al.  A Variational Framework for Non-local Image Inpainting , 2009, EMMCVPR.

[32]  Jian-Feng Cai,et al.  A framelet-based image inpainting algorithm , 2008 .

[33]  Eli Shechtman,et al.  Space-Time Completion of Video , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[34]  L. Ambrosio,et al.  A direct variational approach to a problem arising in image reconstruction , 2003 .

[35]  Andrea L. Bertozzi,et al.  Inpainting of Binary Images Using the Cahn–Hilliard Equation , 2007, IEEE Transactions on Image Processing.

[36]  G. Aubert,et al.  Modeling Very Oscillating Signals. Application to Image Processing , 2005 .

[37]  Wotao Yin,et al.  A comparison of three total variation based texture extraction models , 2007, J. Vis. Commun. Image Represent..

[38]  Stanley Osher,et al.  Image Decomposition and Restoration Using Total Variation Minimization and the H1 , 2003, Multiscale Model. Simul..

[39]  Guillermo Sapiro,et al.  Navier-stokes, fluid dynamics, and image and video inpainting , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[40]  Louis Laborelli,et al.  Missing data correction in still images and image sequences , 2002, MULTIMEDIA '02.

[41]  Folkmar Bornemann,et al.  Fast Image Inpainting Based on Coherence Transport , 2007, Journal of Mathematical Imaging and Vision.

[42]  Luminita A. Vese,et al.  Image Decomposition Using Total Variation and div(BMO) , 2005, Multiscale Model. Simul..

[43]  Yann Gousseau,et al.  Are Natural Images of Bounded Variation? , 2001, SIAM J. Math. Anal..

[44]  Antonin Chambolle,et al.  Dual Norms and Image Decomposition Models , 2005, International Journal of Computer Vision.

[45]  Tony F. Chan,et al.  Aspects of Total Variation Regularized L[sup 1] Function Approximation , 2005, SIAM J. Appl. Math..

[46]  P. Bickel,et al.  Texture synthesis and nonparametric resampling of random fields , 2006, math/0611258.

[47]  Marcel J. T. Reinders,et al.  Edge-based image restoration , 2005, IEEE Transactions on Image Processing.

[48]  Patrick Pérez,et al.  Geometrically Guided Exemplar-Based Inpainting , 2011, SIAM J. Imaging Sci..

[49]  Onur G. Guleryuz,et al.  Nonlinear approximation based image recovery using adaptive sparse reconstructions and iterated denoising-part I: theory , 2006, IEEE Transactions on Image Processing.

[50]  T. Chan,et al.  Image inpainting by correspondence maps: A deterministic approach , 2003 .

[51]  P Perona,et al.  Preattentive texture discrimination with early vision mechanisms. , 1990, Journal of the Optical Society of America. A, Optics and image science.

[52]  L. Ambrosio,et al.  Functions of Bounded Variation and Free Discontinuity Problems , 2000 .

[53]  Antonin Chambolle,et al.  Image Decomposition into a Bounded Variation Component and an Oscillating Component , 2005, Journal of Mathematical Imaging and Vision.

[54]  Patrick Pérez,et al.  Object removal by exemplar-based inpainting , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[55]  J. Besag Spatial Interaction and the Statistical Analysis of Lattice Systems , 1974 .

[56]  Alfred S. Carasso,et al.  Singular Integrals, Image Smoothness, and the Recovery of Texture in Image Deblurring , 2003, SIAM J. Appl. Math..

[57]  Tony F. Chan,et al.  Nontexture Inpainting by Curvature-Driven Diffusions , 2001, J. Vis. Commun. Image Represent..

[58]  Daniel Cohen-Or,et al.  Fragment-based image completion , 2003, ACM Trans. Graph..

[59]  Béla Julesz,et al.  Visual Pattern Discrimination , 1962, IRE Trans. Inf. Theory.

[60]  Wotao Yin,et al.  Second-order Cone Programming Methods for Total Variation-Based Image Restoration , 2005, SIAM J. Sci. Comput..

[61]  Bernard P. Zeigler,et al.  Theory of modeling and simulation , 1976 .

[62]  Gabriel Peyré,et al.  Locally Parallel Texture Modeling , 2011, SIAM J. Imaging Sci..

[63]  Guillermo Sapiro,et al.  Filling-in by joint interpolation of vector fields and gray levels , 2001, IEEE Trans. Image Process..

[64]  Takashi Totsuka,et al.  Combining frequency and spatial domain information for fast interactive image noise removal , 1996, SIGGRAPH.

[65]  Jean-Michel Morel,et al.  Level lines based disocclusion , 1998, Proceedings 1998 International Conference on Image Processing. ICIP98 (Cat. No.98CB36269).

[66]  Paul Harrison,et al.  A Non-Hierarchical Procedure for Re-Synthesis of Complex Textures , 2001, WSCG.

[67]  Eero P. Simoncelli,et al.  Texture characterization via joint statistics of wavelet coefficient magnitudes , 1998, Proceedings 1998 International Conference on Image Processing. ICIP98 (Cat. No.98CB36269).

[68]  Gabriel Peyré,et al.  Manifold models for signals and images , 2009, Comput. Vis. Image Underst..

[69]  Yann Gousseau,et al.  The TVL1 Model: A Geometric Point of View , 2009, Multiscale Model. Simul..

[70]  Jérôme Darbon,et al.  Image Restoration with Discrete Constrained Total Variation Part I: Fast and Exact Optimization , 2006, Journal of Mathematical Imaging and Vision.

[71]  David Mumford,et al.  The 2.1-D sketch , 1990, [1990] Proceedings Third International Conference on Computer Vision.

[72]  B. Julesz Textons, the elements of texture perception, and their interactions , 1981, Nature.

[73]  Tony F. Chan,et al.  Structure-Texture Image Decomposition—Modeling, Algorithms, and Parameter Selection , 2006, International Journal of Computer Vision.

[74]  Lucas Pereira,et al.  Image replacement through texture synthesis , 1997, Proceedings of International Conference on Image Processing.

[75]  Jianhong Shen,et al.  Digital inpainting based on the Mumford–Shah–Euler image model , 2002, European Journal of Applied Mathematics.

[76]  G. Kanizsa,et al.  Organization in Vision: Essays on Gestalt Perception , 1979 .

[77]  Guillermo Sapiro,et al.  Simultaneous structure and texture image inpainting , 2003, IEEE Trans. Image Process..

[78]  Stanley Osher,et al.  Modeling Textures with Total Variation Minimization and Oscillating Patterns in Image Processing , 2003, J. Sci. Comput..

[79]  Jean-Michel Morel,et al.  A Review of Image Denoising Algorithms, with a New One , 2005, Multiscale Model. Simul..

[80]  Tony F. Chan,et al.  Image processing and analysis - variational, PDE, wavelet, and stochastic methods , 2005 .

[81]  Chi-Keung Tang,et al.  Image repairing: robust image synthesis by adaptive ND tensor voting , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[82]  Tony F. Chan,et al.  Mathematical Models for Local Nontexture Inpaintings , 2002, SIAM J. Appl. Math..

[83]  William T. Freeman,et al.  Learning low-level vision , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[84]  Alexei A. Efros,et al.  Image quilting for texture synthesis and transfer , 2001, SIGGRAPH.

[85]  A. Giacomini Piecewise rigidity , 2006 .

[86]  Jeremy S. De Bonet,et al.  Multiresolution sampling procedure for analysis and synthesis of texture images , 1997, SIGGRAPH.

[87]  B Julesz,et al.  Inability of Humans to Discriminate between Visual Textures That Agree in Second-Order Statistics—Revisited , 1973, Perception.

[88]  Tony F. Chan,et al.  Euler's Elastica and Curvature-Based Inpainting , 2003, SIAM J. Appl. Math..

[89]  Stefano Alliney,et al.  A property of the minimum vectors of a regularizing functional defined by means of the absolute norm , 1997, IEEE Trans. Signal Process..

[90]  T. Chan,et al.  Variational pde-based image segmentation and inpainting with applications in computer graphics , 2008 .

[91]  Anil K. Jain,et al.  Markov Random Field Texture Models , 1983, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[92]  Simon Masnou,et al.  Disocclusion: a variational approach using level lines , 2002, IEEE Trans. Image Process..

[93]  G. P. Leonardi,et al.  Locality of the mean curvature of rectifiable varifolds , 2009 .