Sparse Coding Approach for Multi-Frame Image Super Resolution

An image super-resolution method from multiple observation of low-resolution images is proposed. The method is based on sub-pixel accuracy block matching for estimating relative displacements of observed images, and sparse signal representation for estimating the corresponding high-resolution image. Relative displacements of small patches of observed low-resolution images are accurately estimated by a computationally efficient block matching method. Since the estimated displacements are also regarded as a warping component of image degradation process, the matching results are directly utilized to generate low-resolution dictionary for sparse image representation. The matching scores of the block matching are used to select a subset of low-resolution patches for reconstructing a high-resolution patch, that is, an adaptive selection of informative low-resolution images is realized. When there is only one low-resolution image, the proposed method works as a single-frame super-resolution method. The proposed method is shown to perform comparable or superior to conventional single- and multi-frame super-resolution methods through experiments using various real-world datasets.

[1]  Masato Inoue,et al.  Posterior-Mean Super-Resolution With a Causal Gaussian Markov Random Field Prior , 2011, IEEE Transactions on Image Processing.

[2]  Peng Wang,et al.  Super Resolution Reconstruction via Multiple Frames Joint Learning , 2011, 2011 International Conference on Multimedia and Signal Processing.

[3]  Michael Elad,et al.  On Single Image Scale-Up Using Sparse-Representations , 2010, Curves and Surfaces.

[4]  Li Min,et al.  Super-resolution based on improved sparse coding , 2010, 2010 International Conference on Computer Application and System Modeling (ICCASM 2010).

[5]  Shin Ishii,et al.  Superresolution with compound Markov random fields via the variational EM algorithm , 2009, Neural Networks.

[6]  Michael Elad,et al.  Fast and robust multiframe super resolution , 2004, IEEE Transactions on Image Processing.

[7]  Nanning Zheng,et al.  Image hallucination with primal sketch priors , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[8]  Michael Elad,et al.  Double Sparsity: Learning Sparse Dictionaries for Sparse Signal Approximation , 2010, IEEE Transactions on Signal Processing.

[9]  Thomas S. Huang,et al.  Image super-resolution as sparse representation of raw image patches , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[10]  Lei Zhang,et al.  Image Deblurring and Super-Resolution by Adaptive Sparse Domain Selection and Adaptive Regularization , 2010, IEEE Transactions on Image Processing.

[11]  Michael Elad,et al.  Sparse and Redundant Representations - From Theory to Applications in Signal and Image Processing , 2010 .

[12]  Michal Irani,et al.  Motion Analysis for Image Enhancement: Resolution, Occlusion, and Transparency , 1993, J. Vis. Commun. Image Represent..

[13]  Robert L. Stevenson,et al.  Spatial Resolution Enhancement of Low-Resolution Image Sequences A Comprehensive Review with Directions for Future Research , 1998 .

[14]  S. Mallat,et al.  Adaptive greedy approximations , 1997 .

[15]  D. Yeung,et al.  Super-resolution through neighbor embedding , 2004, CVPR 2004.

[16]  David J. Field,et al.  Emergence of simple-cell receptive field properties by learning a sparse code for natural images , 1996, Nature.

[17]  Stéphane Mallat,et al.  Matching pursuits with time-frequency dictionaries , 1993, IEEE Trans. Signal Process..

[18]  Jonathan W. Brandt,et al.  Improved Accuracy in Gradient-Based Optical Flow Estimation , 1997, International Journal of Computer Vision.

[19]  Y. C. Pati,et al.  Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition , 1993, Proceedings of 27th Asilomar Conference on Signals, Systems and Computers.

[20]  Sheng Chen,et al.  Orthogonal least squares methods and their application to non-linear system identification , 1989 .

[21]  Rajat Raina,et al.  Efficient sparse coding algorithms , 2006, NIPS.

[22]  Michael A. Saunders,et al.  Atomic Decomposition by Basis Pursuit , 1998, SIAM J. Sci. Comput..

[23]  Qi Tian,et al.  Algorithms for subpixel registration , 1986 .

[24]  Masatoshi Okutomi,et al.  Multi-Parameter Simultaneous Estimation on Area-Based Matching , 2006, International Journal of Computer Vision.

[25]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[26]  Russell C. Hardie,et al.  Joint MAP registration and high-resolution image estimation using a sequence of undersampled images , 1997, IEEE Trans. Image Process..

[27]  A. Bruckstein,et al.  K-SVD : An Algorithm for Designing of Overcomplete Dictionaries for Sparse Representation , 2005 .

[28]  Kai-Kuang Ma,et al.  A survey on super-resolution imaging , 2011, Signal Image Video Process..

[29]  Valia Guerra Ones.  Multi-frame Super Resolution based on Sparse Coding , 2016 .

[30]  Norimichi Tsumura,et al.  Video super-resolution using texton substitution , 2007, SIGGRAPH '07.

[31]  S T Roweis,et al.  Nonlinear dimensionality reduction by locally linear embedding. , 2000, Science.

[32]  David J. Field,et al.  How Close Are We to Understanding V1? , 2005, Neural Computation.

[33]  Luis Alvarez,et al.  Motion Estimation Techniques in Super-Resolution Image Reconstruction. A Performance Evaluation , 2006 .

[34]  R. Tibshirani,et al.  Least angle regression , 2004, math/0406456.

[35]  V. N. Dvornychenko,et al.  Bounds on (Deterministic) Correlation Functions with Application to Registration , 1983, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[36]  Jake K. Aggarwal,et al.  On the computation of motion from sequences of images-A review , 1988, Proc. IEEE.

[37]  David J. Fleet,et al.  Performance of optical flow techniques , 1994, International Journal of Computer Vision.

[38]  M. Elad,et al.  $rm K$-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation , 2006, IEEE Transactions on Signal Processing.

[39]  Balas K. Natarajan,et al.  Sparse Approximate Solutions to Linear Systems , 1995, SIAM J. Comput..

[40]  Xuelong Li,et al.  Geometry constrained sparse coding for single image super-resolution , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[41]  Jean-Michel Morel,et al.  Nonlocal Image and Movie Denoising , 2008, International Journal of Computer Vision.

[42]  Kjersti Engan,et al.  Method of optimal directions for frame design , 1999, 1999 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings. ICASSP99 (Cat. No.99CH36258).

[43]  William T. Freeman,et al.  Example-Based Super-Resolution , 2002, IEEE Computer Graphics and Applications.

[44]  Sebastián López,et al.  Analysis of fast block matching motion estimation algorithms for video super-resolution systems , 2008, IEEE Transactions on Consumer Electronics.

[45]  Thomas S. Huang,et al.  Image Super-Resolution Via Sparse Representation , 2010, IEEE Transactions on Image Processing.