Dictionaries for Sparse Representation Modeling

Sparse and redundant representation modeling of data assumes an ability to describe signals as linear combinations of a few atoms from a pre-specified dictionary. As such, the choice of the dictionary that sparsifies the signals is crucial for the success of this model. In general, the choice of a proper dictionary can be done using one of two ways: i) building a sparsifying dictionary based on a mathematical model of the data, or ii) learning a dictionary to perform best on a training set. In this paper we describe the evolution of these two paradigms. As manifestations of the first approach, we cover topics such as wavelets, wavelet packets, contourlets, and curvelets, all aiming to exploit 1-D and 2-D mathematical models for constructing effective dictionaries for signals and images. Dictionary learning takes a different route, attaching the dictionary to a set of examples it is supposed to serve. From the seminal work of Field and Olshausen, through the MOD, the K-SVD, the Generalized PCA and others, this paper surveys the various options such training has to offer, up to the most recent contributions and structures.

[1]  Dennis Gabor,et al.  Theory of communication , 1946 .

[2]  R. Stephenson A and V , 1962, The British journal of ophthalmology.

[3]  J. Tukey,et al.  An algorithm for the machine calculation of complex Fourier series , 1965 .

[4]  J.B. Allen,et al.  A unified approach to short-time Fourier analysis and synthesis , 1977, Proceedings of the IEEE.

[5]  J. Daugman Two-dimensional spectral analysis of cortical receptive field profiles , 1980, Vision Research.

[6]  M. Bastiaans,et al.  Gabor's expansion of a signal into Gaussian elementary signals , 1980, Proceedings of the IEEE.

[7]  A. Janssen Gabor representation of generalized functions , 1981 .

[8]  Peter J. Huber,et al.  Robust Statistics , 2005, Wiley Series in Probability and Statistics.

[9]  Edward H. Adelson,et al.  The Laplacian Pyramid as a Compact Image Code , 1983, IEEE Trans. Commun..

[10]  A. Grossmann,et al.  DECOMPOSITION OF HARDY FUNCTIONS INTO SQUARE INTEGRABLE WAVELETS OF CONSTANT SHAPE , 1984 .

[11]  J. Daugman Uncertainty relation for resolution in space, spatial frequency, and orientation optimized by two-dimensional visual cortical filters. , 1985, Journal of the Optical Society of America. A, Optics and image science.

[12]  Y. Meyer,et al.  Ondelettes et bases hilbertiennes. , 1986 .

[13]  I. Daubechies,et al.  PAINLESS NONORTHOGONAL EXPANSIONS , 1986 .

[14]  Y. Meyer Principe d'incertitude, bases hilbertiennes et algèbres d'opérateurs , 1986 .

[15]  I. Daubechies Orthonormal bases of compactly supported wavelets , 1988 .

[16]  John G. Daugman,et al.  Complete discrete 2-D Gabor transforms by neural networks for image analysis and compression , 1988, IEEE Trans. Acoust. Speech Signal Process..

[17]  Yehoshua Y. Zeevi,et al.  The Generalized Gabor Scheme of Image Representation in Biological and Machine Vision , 1988, IEEE Trans. Pattern Anal. Mach. Intell..

[18]  H. Feichtinger,et al.  Banach spaces related to integrable group representations and their atomic decompositions. Part II , 1989 .

[19]  Stéphane Mallat,et al.  A Theory for Multiresolution Signal Decomposition: The Wavelet Representation , 1989, IEEE Trans. Pattern Anal. Mach. Intell..

[20]  H. Feichtinger,et al.  Banach spaces related to integrable group representations and their atomic decompositions, I , 1989 .

[21]  Ingrid Daubechies,et al.  The wavelet transform, time-frequency localization and signal analysis , 1990, IEEE Trans. Inf. Theory.

[22]  Ronald R. Coifman,et al.  Wavelet analysis and signal processing , 1990 .

[23]  Jason Wexler,et al.  Discrete Gabor expansions , 1990, Signal Process..

[24]  Ingrid Daubechies,et al.  Ten Lectures on Wavelets , 1992 .

[25]  Stéphane Mallat,et al.  Singularity detection and processing with wavelets , 1992, IEEE Trans. Inf. Theory.

[26]  Edward H. Adelson,et al.  Shiftable multiscale transforms , 1992, IEEE Trans. Inf. Theory.

[27]  H. Feichtinger,et al.  Gabor wavelets and the Heisenberg group: Gabor expansions and short time Fourier transform from the group theoretical point of view , 1993 .

[28]  G. Beylkin On the representation of operators in bases of compactly supported wavelets , 1992 .

[29]  Joan L. Mitchell,et al.  JPEG: Still Image Data Compression Standard , 1992 .

[30]  Stéphane Mallat,et al.  Characterization of Signals from Multiscale Edges , 2011, IEEE Trans. Pattern Anal. Mach. Intell..

[31]  Y. C. Pati,et al.  Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition , 1993, Proceedings of 27th Asilomar Conference on Signals, Systems and Computers.

[32]  Stéphane Mallat,et al.  Matching pursuits with time-frequency dictionaries , 1993, IEEE Trans. Signal Process..

[33]  Shie Qian,et al.  Discrete Gabor transform , 1993, IEEE Trans. Signal Process..

[34]  Y. Meyer Wavelets and Operators , 1993 .

[35]  David J. Field,et al.  What Is the Goal of Sensory Coding? , 1994, Neural Computation.

[36]  B. Silverman,et al.  The Stationary Wavelet Transform and some Statistical Applications , 1995 .

[37]  D. Donoho,et al.  Translation-Invariant De-Noising , 1995 .

[38]  David J. Field,et al.  Emergence of simple-cell receptive field properties by learning a sparse code for natural images , 1996, Nature.

[39]  Bhaskar D. Rao,et al.  Sparse signal reconstruction from limited data using FOCUSS: a re-weighted minimum norm algorithm , 1997, IEEE Trans. Signal Process..

[40]  H. Feichtinger,et al.  Gabor Frames and Time-Frequency Analysis of Distributions* , 1997 .

[41]  Michael A. Saunders,et al.  Atomic Decomposition by Basis Pursuit , 1998, SIAM J. Sci. Comput..

[42]  R. DeVore,et al.  Nonlinear approximation , 1998, Acta Numerica.

[43]  S. Mallat A wavelet tour of signal processing , 1998 .

[44]  Kjersti Engan,et al.  Method of optimal directions for frame design , 1999, 1999 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings. ICASSP99 (Cat. No.99CH36258).

[45]  E. Candès,et al.  Ridgelets: a key to higher-dimensional intermittency? , 1999, Philosophical Transactions of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences.

[46]  D. Donoho Wedgelets: nearly minimax estimation of edges , 1999 .

[47]  E. Candès,et al.  Curvelets: A Surprisingly Effective Nonadaptive Representation for Objects with Edges , 2000 .

[48]  Høgskolen i Stavanger FRAME DESIGN USING FOCUSS WITH METHOD OF OPTIMAL DIRECTIONS (MOD) , 2000 .

[49]  Terrence J. Sejnowski,et al.  Learning Overcomplete Representations , 2000, Neural Computation.

[50]  P. Tseng,et al.  Block Coordinate Relaxation Methods for Nonparametric Wavelet Denoising , 2000 .

[51]  Emmanuel J. Candès,et al.  The curvelet transform for image denoising , 2001, Proceedings 2001 International Conference on Image Processing (Cat. No.01CH37205).

[52]  N. Kingsbury Complex Wavelets for Shift Invariant Analysis and Filtering of Signals , 2001 .

[53]  M. Vetterli,et al.  Contourlets: a new directional multiresolution image representation , 2002, Conference Record of the Thirty-Sixth Asilomar Conference on Signals, Systems and Computers, 2002..

[54]  Emmanuel J. Candès,et al.  The curvelet transform for image denoising , 2002, IEEE Trans. Image Process..

[55]  Bruno A. Olshausen,et al.  Learning Sparse Multiscale Image Representations , 2002, NIPS.

[56]  Joseph F. Murray,et al.  Dictionary Learning Algorithms for Sparse Representation , 2003, Neural Computation.

[57]  I. Daubechies,et al.  An iterative thresholding algorithm for linear inverse problems with a sparsity constraint , 2003, math/0307152.

[58]  L. Wood,et al.  From the Authors , 2003, European Respiratory Journal.

[59]  E. Candès,et al.  Continuous Curvelet Transform : I . Resolution of the Wavefront Set , 2003 .

[60]  E. Candès,et al.  Continuous curvelet transform , 2003 .

[61]  Robert D. Nowak,et al.  Platelets: a multiscale approach for recovering edges and surfaces in photon-limited medical imaging , 2003, IEEE Transactions on Medical Imaging.

[62]  B. Ripley,et al.  Robust Statistics , 2018, Wiley Series in Probability and Statistics.

[63]  Minh N. Do,et al.  Ieee Transactions on Image Processing the Contourlet Transform: an Efficient Directional Multiresolution Image Representation , 2022 .

[64]  Rémi Gribonval,et al.  Learning unions of orthonormal bases with thresholded singular value decomposition , 2005, Proceedings. (ICASSP '05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005..

[65]  Wang-Q Lim,et al.  Sparse multidimensional representation using shearlets , 2005, SPIE Optics + Photonics.

[66]  Stéphane Mallat,et al.  Sparse geometric image representations with bandelets , 2005, IEEE Transactions on Image Processing.

[67]  Stéphane Mallat,et al.  Surface compression with geometric bandelets , 2005, ACM Trans. Graph..

[68]  Lexing Ying,et al.  3D discrete curvelet transform , 2005, SPIE Optics + Photonics.

[69]  D. Donoho,et al.  Simultaneous cartoon and texture image inpainting using morphological component analysis (MCA) , 2005 .

[70]  Richard Baraniuk,et al.  The Dual-tree Complex Wavelet Transform , 2007 .

[71]  A. Bruckstein,et al.  K-SVD : An Algorithm for Designing of Overcomplete Dictionaries for Sparse Representation , 2005 .

[72]  S. Shankar Sastry,et al.  Generalized principal component analysis (GPCA) , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[73]  Baltasar Beferull-Lozano,et al.  Directionlets: anisotropic multidirectional representation with separable filtering , 2006, IEEE Transactions on Image Processing.

[74]  D. Labate,et al.  Resolution of the wavefront set using continuous shearlets , 2006, math/0605375.

[75]  Laurent Demanet,et al.  Fast Discrete Curvelet Transforms , 2006, Multiscale Model. Simul..

[76]  Mike E. Davies,et al.  Sparse and shift-Invariant representations of music , 2006, IEEE Transactions on Audio, Speech, and Language Processing.

[77]  Pierre Vandergheynst,et al.  MoTIF: An Efficient Algorithm for Learning Translation Invariant Dictionaries , 2006, 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings.

[78]  Michael Elad,et al.  Why Simple Shrinkage Is Still Relevant for Redundant Representations? , 2006, IEEE Transactions on Information Theory.

[79]  Minh N. Do,et al.  A New Contourlet Transform with Sharp Frequency Localization , 2006, 2006 International Conference on Image Processing.

[80]  Hayder Radha,et al.  Translation-Invariant Contourlet Transform and Its Application to Image Denoising , 2006, IEEE Transactions on Image Processing.

[81]  M. Elad,et al.  $rm K$-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation , 2006, IEEE Transactions on Signal Processing.

[82]  Minh N. Do,et al.  The Nonsubsampled Contourlet Transform: Theory, Design, and Applications , 2006, IEEE Transactions on Image Processing.

[83]  David L. Donoho,et al.  Sparse Solution Of Underdetermined Linear Equations By Stagewise Orthogonal Matching Pursuit , 2006 .

[84]  Michael Elad,et al.  Analysis versus synthesis in signal priors , 2006, 2006 14th European Signal Processing Conference.

[85]  Justin K. Romberg,et al.  Wavelet-domain approximation and compression of piecewise smooth images , 2006, IEEE Transactions on Image Processing.

[86]  Minh N. Do,et al.  Multidimensional Directional Filter Banks and Surfacelets , 2007, IEEE Transactions on Image Processing.

[87]  Kjersti Engan,et al.  Family of iterative LS-based dictionary learning algorithms, ILS-DLA, for sparse signal representation , 2007, Digit. Signal Process..

[88]  Alessandro Foi,et al.  Image Denoising by Sparse 3-D Transform-Domain Collaborative Filtering , 2007, IEEE Transactions on Image Processing.

[89]  Michael Elad,et al.  Sparse and Redundant Modeling of Image Content Using an Image-Signature-Dictionary , 2008, SIAM J. Imaging Sci..

[90]  G. Easley,et al.  Sparse directional image representations using the discrete shearlet transform , 2008 .

[91]  Stphane Mallat,et al.  A Wavelet Tour of Signal Processing, Third Edition: The Sparse Way , 2008 .

[92]  Michael Elad,et al.  Learning Multiscale Sparse Representations for Image and Video Restoration , 2007, Multiscale Model. Simul..

[93]  J. Tropp,et al.  CoSaMP: Iterative signal recovery from incomplete and inaccurate samples , 2008, Commun. ACM.

[94]  Mike E. Davies,et al.  Parametric Dictionary Design for Sparse Coding , 2009, IEEE Transactions on Signal Processing.

[95]  Venkat Chandrasekaran,et al.  Representation and Compression of Multidimensional Piecewise Functions Using Surflets , 2009, IEEE Transactions on Information Theory.

[96]  Michael Elad,et al.  From Sparse Solutions of Systems of Equations to Sparse Modeling of Signals and Images , 2009, SIAM Rev..

[97]  Heng Tao Shen,et al.  Principal Component Analysis , 2009, Encyclopedia of Biometrics.

[98]  Michael Elad,et al.  Double Sparsity: Learning Sparse Dictionaries for Sparse Signal Approximation , 2010, IEEE Transactions on Signal Processing.

[99]  SapiroGuillermo,et al.  Online Learning for Matrix Factorization and Sparse Coding , 2010 .

[100]  Kjersti Engan,et al.  Recursive Least Squares Dictionary Learning Algorithm , 2010, IEEE Transactions on Signal Processing.

[101]  Prashant Parikh A Theory of Communication , 2010 .

[102]  Guillermo Sapiro,et al.  Online Learning for Matrix Factorization and Sparse Coding , 2009, J. Mach. Learn. Res..

[103]  Deanna Needell,et al.  CoSaMP: Iterative signal recovery from incomplete and inaccurate samples , 2008, ArXiv.

[104]  W. Marsden I and J , 2012 .