Compressed Sensing and Dictionary Learning

Compressed sensing is a new field that arose as a response to inefficient traditional signal acquisition schemes. Under the assumption that the signal of interest is sparse, one wishes to take a small number of linear samples and later utilize a reconstruction algorithm to accurately recover the compressed signal. Typically, one assumes the signal is sparse itself or with respect to some fixed orthonormal basis. However, in applications one instead more often encounters signals sparse with respect to a tight frame which may be far from orthonormal. In the first part of these notes, we will introduce the compressed sensing problem as well as recent results extending the theory to the case of sparsity in tight frames. The second part of the notes focuses on dictionary learning which is also a new field and closely related to compressive sensing. Briefly speaking, a dictionary is a redundant system consisting of prototype signals that are used to express other signals. Due to the redundancy, for any given signal, there are many ways to represent it, but normally the sparsest representation is preferred for simplicity and easy interpretability. A good analog is the English language where the dictionary is the collection of all words (prototype signals) and sentences (signals) are short and concise combinations of words. Here we will introduce the problem of dictionary learning, its applications, and existing solutions.

[1]  M. Davies,et al.  Greedy-like algorithms for the cosparse analysis model , 2012, 1207.2456.

[2]  Heng Tao Shen,et al.  Principal Component Analysis , 2009, Encyclopedia of Database Systems.

[3]  Mike E. Davies,et al.  Iterative Hard Thresholding for Compressed Sensing , 2008, ArXiv.

[4]  D. Donoho,et al.  Sparse MRI: The application of compressed sensing for rapid MR imaging , 2007, Magnetic resonance in medicine.

[5]  Raja Giryes,et al.  Greedy Signal Space Methods for incoherence and beyond , 2013, ArXiv.

[6]  Guillermo Sapiro,et al.  Non-Parametric Bayesian Dictionary Learning for Sparse Image Representations , 2009, NIPS.

[7]  Martin J. Wainwright,et al.  A unified framework for high-dimensional analysis of $M$-estimators with decomposable regularizers , 2009, NIPS.

[8]  H. Rauhut On the Impossibility of Uniform Sparse Reconstruction using Greedy Methods , 2007 .

[9]  Michael I. Jordan,et al.  On Spectral Clustering: Analysis and an algorithm , 2001, NIPS.

[10]  Emmanuel J. Candès,et al.  Near-Optimal Signal Recovery From Random Projections: Universal Encoding Strategies? , 2004, IEEE Transactions on Information Theory.

[11]  Guangliang Chen,et al.  Spectral Curvature Clustering (SCC) , 2008, International Journal of Computer Vision.

[12]  P. Dutilleux An Implementation of the “algorithme à trous” to Compute the Wavelet Transform , 1989 .

[13]  Stephen L. Keeling Total variation based convex filters for medical imaging , 2003, Appl. Math. Comput..

[14]  Michael Elad Sparse and Redundant Representation Modeling—What Next? , 2012, IEEE Signal Processing Letters.

[15]  Michael Elad,et al.  Compression of facial images using the K-SVD algorithm , 2008, J. Vis. Commun. Image Represent..

[16]  A. Bruckstein,et al.  K-SVD : An Algorithm for Designing of Overcomplete Dictionaries for Sparse Representation , 2005 .

[17]  S. Mendelson,et al.  Uniform Uncertainty Principle for Bernoulli and Subgaussian Ensembles , 2006, math/0608665.

[18]  Babak Hassibi,et al.  New Null Space Results and Recovery Thresholds for Matrix Rank Minimization , 2010, ArXiv.

[19]  Joel A. Tropp,et al.  Just relax: convex programming methods for identifying sparse signals in noise , 2006, IEEE Transactions on Information Theory.

[20]  Nathan Srebro,et al.  Fast maximum margin matrix factorization for collaborative prediction , 2005, ICML.

[21]  Deanna Needell,et al.  CoSaMP: Iterative signal recovery from incomplete and inaccurate samples , 2008, Commun. ACM.

[22]  Emmanuel J. Candès,et al.  Decoding by linear programming , 2005, IEEE Transactions on Information Theory.

[23]  E. Candès,et al.  Stable signal recovery from incomplete and inaccurate measurements , 2005, math/0503066.

[24]  Michael B. Wakin,et al.  Compressive Sensing of Analog Signals Using Discrete Prolate Spheroidal Sequences , 2011, ArXiv.

[25]  Guillermo Sapiro,et al.  Online Learning for Matrix Factorization and Sparse Coding , 2009, J. Mach. Learn. Res..

[26]  Felix J. Herrmann,et al.  Higher Dimensional Blue-noise Sampling Schemes For Curvelet-based Seismic Data Recovery , 2009 .

[27]  F. Herrmann,et al.  Compressed wavefield extrapolation , 2007 .

[28]  Emmanuel J. Candès,et al.  A Probabilistic and RIPless Theory of Compressed Sensing , 2011, IEEE Transactions on Information Theory.

[29]  Emmanuel J. Candès,et al.  Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information , 2004, IEEE Transactions on Information Theory.

[30]  Michael Elad,et al.  The Cosparse Analysis Model and Algorithms , 2011, ArXiv.

[31]  Michael Elad,et al.  Dictionaries for Sparse Representation Modeling , 2010, Proceedings of the IEEE.


[33]  Emmanuel J. Candès,et al.  Tight Oracle Inequalities for Low-Rank Matrix Recovery From a Minimal Number of Noisy Random Measurements , 2011, IEEE Transactions on Information Theory.

[34]  Guangliang Chen,et al.  Multiscale geometric and spectral analysis of plane arrangements , 2011, CVPR 2011.

[35]  Laurent Demanet,et al.  Fast Discrete Curvelet Transforms , 2006, Multiscale Model. Simul..

[36]  Xiaodong Li,et al.  Stable Principal Component Pursuit , 2010, 2010 IEEE International Symposium on Information Theory.

[37]  Richard G. Baraniuk,et al.  Compressive Sensing DNA Microarrays , 2009, EURASIP J. Bioinform. Syst. Biol..

[38]  Guangliang Chen,et al.  Foundations of a Multi-way Spectral Clustering Framework for Hybrid Linear Modeling , 2009, Found. Comput. Math..

[39]  R. Baraniuk,et al.  Compressive Radar Imaging , 2007, 2007 IEEE Radar Conference.

[40]  Yonina C. Eldar,et al.  Xampling: Compressed Sensing of Analog Signals , 2012, Compressed Sensing.

[41]  Joel A. Tropp,et al.  Signal Recovery From Random Measurements Via Orthogonal Matching Pursuit , 2007, IEEE Transactions on Information Theory.

[42]  Amit Singer,et al.  A remark on global positioning from local distances , 2008, Proceedings of the National Academy of Sciences.

[43]  Ravishankar Krishnaswamy,et al.  Relax, No Need to Round: Integrality of Clustering Formulations , 2015, ITCS.

[44]  M. Bastiaans Gabor's expansion of a signal into Gaussian elementary signals , 1980, Proceedings of the IEEE.

[45]  Mojdeh Mohtashemi,et al.  Sparse sensing DNA microarray-based biosensor: Is it feasible? , 2010, 2010 IEEE Sensors Applications Symposium (SAS).

[46]  Tony F. Chan,et al.  Total Variation Wavelet Inpainting , 2006, Journal of Mathematical Imaging and Vision.

[47]  Vipin Kumar,et al.  A Fast and High Quality Multilevel Scheme for Partitioning Irregular Graphs , 1998, SIAM J. Sci. Comput..

[48]  Emmanuel J. Candès,et al.  Signal recovery from random projections , 2005, IS&T/SPIE Electronic Imaging.

[49]  Thomas Blumensath Sampling and Reconstructing Signals From a Union of Linear Subspaces , 2011, IEEE Transactions on Information Theory.

[50]  Yonina C. Eldar,et al.  Compressed Sensing with Coherent and Redundant Dictionaries , 2010, ArXiv.

[51]  Emmanuel J. Candès,et al.  Tight oracle bounds for low-rank matrix recovery from a minimal number of random measurements , 2010, ArXiv.

[52]  A. Ravi Kumar,et al.  Compressed-Sensing-Enabled Video Streaming for Wireless Multimedia Sensor Networks , 2014 .

[53]  E. Candès,et al.  New tight frames of curvelets and optimal representations of objects with piecewise C2 singularities , 2004 .

[54]  Hans-Peter Kriegel,et al.  Subspace clustering , 2012, Wiley Interdiscip. Rev. Data Min. Knowl. Discov..

[55]  Michael Elad,et al.  From Sparse Solutions of Systems of Equations to Sparse Modeling of Signals and Images , 2009, SIAM Rev..

[56]  E.J. Candes,et al.  An Introduction To Compressive Sampling , 2008, IEEE Signal Processing Magazine.

[57]  M. Rudelson,et al.  Sparse reconstruction by convex relaxation: Fourier and Gaussian measurements , 2006, 2006 40th Annual Conference on Information Sciences and Systems.

[58]  J.B. Allen,et al.  A unified approach to short-time Fourier analysis and synthesis , 1977, Proceedings of the IEEE.


[60]  J. MacQueen Some methods for classification and analysis of multivariate observations , 1967 .

[61]  A. Janssen Gabor representation of generalized functions , 1981 .

[62]  Mohamed-Jalal Fadili,et al.  The Undecimated Wavelet Decomposition and its Reconstruction , 2007, IEEE Transactions on Image Processing.

[63]  Nathan Srebro,et al.  Learning with matrix factorizations , 2004 .

[64]  Pablo A. Parrilo,et al.  Rank-Sparsity Incoherence for Matrix Decomposition , 2009, SIAM J. Optim..

[65]  M. Rudelson,et al.  On sparse reconstruction from Fourier and Gaussian measurements , 2008 .


[67]  Xiaoming Huo,et al.  Uncertainty principles and ideal atomic decomposition , 2001, IEEE Trans. Inf. Theory.

[68]  Michael Elad,et al.  Learning Multiscale Sparse Representations for Image and Video Restoration , 2007, Multiscale Model. Simul..

[69]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[70]  S. Frick Compressed Sensing , 2015 .

[71]  SapiroGuillermo,et al.  Online Learning for Matrix Factorization and Sparse Coding , 2010 .

[72]  Michael Elad,et al.  RIP-Based Near-Oracle Performance Guarantees for SP, CoSaMP, and IHT , 2012, IEEE Transactions on Signal Processing.

[73]  Thomas Strohmer,et al.  High-Resolution Radar via Compressed Sensing , 2009, IEEE Transactions on Signal Processing.

[74]  Stéphane Mallat,et al.  A Theory for Multiresolution Signal Decomposition: The Wavelet Representation , 1989, IEEE Trans. Pattern Anal. Mach. Intell..

[75]  R. O. Schmidt,et al.  Multiple emitter location and signal Parameter estimation , 1986 .

[76]  Zuowei Shen Affine systems in L 2 ( IR d ) : the analysis of the analysis operator , 1995 .

[77]  Shiqian Ma,et al.  An efficient algorithm for compressed MR imaging using total variation and wavelets , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[78]  Deanna Needell,et al.  Signal Space CoSaMP for Sparse Recovery With Redundant Dictionaries , 2013, IEEE Transactions on Information Theory.

[79]  Guangliang Chen,et al.  Multiscale geometric wavelets for the analysis of point clouds , 2010, 2010 44th Annual Conference on Information Sciences and Systems (CISS).

[80]  Richard G. Baraniuk,et al.  Random Filters for Compressive Sampling and Reconstruction , 2006, 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings.

[81]  D. Donoho,et al.  Redundant Multiscale Transforms and Their Application for Morphological Component Separation , 2004 .

[82]  Qun Wan,et al.  Total Variation Minimization Based Compressive Wideband Spectrum Sensing for Cognitive Radios , 2011, ArXiv.

[83]  M. Lustig,et al.  Compressed Sensing MRI , 2008, IEEE Signal Processing Magazine.

[84]  Dennis Gabor,et al.  Theory of communication , 1946 .

[85]  Stéphane Mallat,et al.  Matching pursuits with time-frequency dictionaries , 1993, IEEE Trans. Signal Process..

[86]  M. Maggioni,et al.  Multiscale Geometric Methods for Data Sets II: Geometric Multi-Resolution Analysis , 2011, 1105.4924.

[87]  Jean-Michel Morel,et al.  A Review of Image Denoising Algorithms, with a New One , 2005, Multiscale Model. Simul..

[88]  Michael Elad,et al.  Image Denoising Via Sparse and Redundant Representations Over Learned Dictionaries , 2006, IEEE Transactions on Image Processing.

[89]  E.J. Candes Compressive sampling , 2006 .

[90]  Michael A. Saunders,et al.  Atomic Decomposition by Basis Pursuit , 1998, SIAM J. Sci. Comput..

[91]  R. DeVore,et al.  Compressed sensing and best k-term approximation , 2008 .

[92]  David J. Field,et al.  Emergence of simple-cell receptive field properties by learning a sparse code for natural images , 1996, Nature.

[93]  Joel A. Tropp,et al.  The restricted isometry property for time–frequency structured random matrices , 2011, ArXiv.

[94]  J. Romberg Imaging via Compressive Sampling , 2008, IEEE Signal Processing Magazine.

[95]  Deanna Needell,et al.  Signal Recovery From Incomplete and Inaccurate Measurements Via Regularized Orthogonal Matching Pursuit , 2010, IEEE Journal of Selected Topics in Signal Processing.

[96]  Yi Ma,et al.  Robust principal component analysis? , 2009, JACM.

[97]  Deanna Needell,et al.  Uniform Uncertainty Principle and Signal Recovery via Regularized Orthogonal Matching Pursuit , 2007, Found. Comput. Math..

[98]  J. Haupt,et al.  A Generalized Restricted Isometry Property , 2008 .

[99]  Deanna Needell,et al.  Stable Image Reconstruction Using Total Variation Minimization , 2013, SIAM J. Imaging Sci..

[100]  Ingrid Daubechies,et al.  Ten Lectures on Wavelets , 1992 .

[101]  Michael Elad,et al.  On the Role of Sparse and Redundant Representations in Image Processing , 2010, Proceedings of the IEEE.

[102]  Guillermo Sapiro,et al.  Online dictionary learning for sparse coding , 2009, ICML '09.

[103]  Deanna Needell,et al.  Near-Optimal Compressed Sensing Guarantees for Total Variation Minimization , 2013, IEEE Transactions on Image Processing.

[104]  S. Kirolos,et al.  Analog-to-Information Conversion via Random Demodulation , 2006, 2006 IEEE Dallas/CAS Workshop on Design, Applications, Integration and Software.

[105]  Anthony Man-Cho So,et al.  Theory of semidefinite programming for Sensor Network Localization , 2005, SODA '05.

[106]  Rachel Ward,et al.  New and Improved Johnson-Lindenstrauss Embeddings via the Restricted Isometry Property , 2010, SIAM J. Math. Anal..

[107]  Ronald R. Coifman Wavelet analysis and signal processing , 1990 .

[108]  S. Mallat A wavelet tour of signal processing , 1998 .

[109]  Pablo A. Parrilo,et al.  Guaranteed Minimum-Rank Solutions of Linear Matrix Equations via Nuclear Norm Minimization , 2007, SIAM Rev..

[110]  S. Muthukrishnan,et al.  Data streams: algorithms and applications , 2005, SODA '03.

[111]  Jie Tang,et al.  Tomosynthesis via total variation minimization reconstruction and prior image constrained compressed sensing (PICCS) on a C-arm system , 2008, SPIE Medical Imaging.

[112]  Michael Elad,et al.  Generalizing the Nonlocal-Means to Super-Resolution Reconstruction , 2009, IEEE Transactions on Image Processing.

[113]  Richard G. Baraniuk,et al.  An Architecture for Compressive Imaging , 2006, 2006 International Conference on Image Processing.

[114]  Jens Frahm,et al.  Suppression of MRI Truncation Artifacts Using Total Variation Constrained Data Extrapolation , 2008, Int. J. Biomed. Imaging.

[115]  Jitendra Malik,et al.  Normalized cuts and image segmentation , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.