Efficient Algorithms for Convolutional Sparse Representations

When applying sparse representation techniques to images, the standard approach is to independently compute the representations for a set of overlapping image patches. This method performs very well in a variety of applications, but results in a representation that is multi-valued and not optimized with respect to the entire image. An alternative representation structure is provided by a convolutional sparse representation, in which a sparse representation of an entire image is computed by replacing the linear combination of a set of dictionary vectors by the sum of a set of convolutions with dictionary filters. The resulting representation is both single-valued and jointly optimized over the entire image. While this form of a sparse representation has been applied to a variety of problems in signal and image processing and computer vision, the computational expense of the corresponding optimization problems has restricted application to relatively small signals and images. This paper presents new, efficient algorithms that substantially improve on the performance of other recent methods, contributing to the development of this type of representation as a practical tool for a wider range of problems.

[1]  Michael Elad,et al.  From Sparse Solutions of Systems of Equations to Sparse Modeling of Signals and Images , 2009, SIAM Rev..

[2]  M. Elad,et al.  $rm K$-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation , 2006, IEEE Transactions on Signal Processing.

[3]  Youji Iiguni,et al.  Sparse image representations with shift-invariant tree-structured dictionaries , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[4]  Nadaniela Egidi,et al.  A Sherman-Morrison approach to the solution of linear systems , 2006 .

[5]  Pierre Vandergheynst,et al.  MoTIF: An Efficient Algorithm for Learning Translation Invariant Dictionaries , 2006, 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings.

[6]  José Carlos Príncipe,et al.  A fast proximal method for convolutional sparse coding , 2013, The 2013 International Joint Conference on Neural Networks (IJCNN).

[7]  Christoph Studer,et al.  Learning phase-invariant dictionaries , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[8]  Jean Ponce,et al.  Sparse Modeling for Image and Vision Processing , 2014, Found. Trends Comput. Graph. Vis..

[9]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[10]  Sotirios A. Tsaftaris,et al.  Explicit Shift-Invariant Dictionary Learning , 2014, IEEE Signal Processing Letters.

[11]  Ping-Keng Jao,et al.  Informed monaural source separation of music based on convolutional sparse coding , 2015, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[12]  Youji Iiguni,et al.  Sparse image representations with shift and rotation invariance constraints , 2009, 2009 International Symposium on Intelligent Signal Processing and Communication Systems (ISPACS).

[13]  Michael Elad,et al.  Multiscale Sparse Image Representationwith Learned Dictionaries , 2007, 2007 IEEE International Conference on Image Processing.

[14]  Rajesh P. N. Rao,et al.  Development of localized oriented receptive fields by learning a translation-invariant code for natural images. , 1998, Network.

[15]  Brendt Wohlberg,et al.  Efficient convolutional sparse coding , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[16]  Stéphane Mallat,et al.  Matching pursuits with time-frequency dictionaries , 1993, IEEE Trans. Signal Process..

[17]  Daniel N. Rockmore,et al.  Bayesian Learning of Sparse Multiscale Image Representations , 2013, IEEE Transactions on Image Processing.

[18]  Graham W. Taylor,et al.  Adaptive deconvolutional networks for mid and high level feature learning , 2011, 2011 International Conference on Computer Vision.

[19]  Kjersti Engan,et al.  Image compression using learned dictionaries by RLS-DLA and compared with K-SVD , 2011, 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[20]  Mike E. Davies,et al.  On Shift-Invariant Sparse Coding , 2004, ICA.

[21]  David B. Dunson,et al.  Deep Learning with Hierarchical Convolutional Factor Analysis , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  Kjersti Engan,et al.  Method of optimal directions for frame design , 1999, 1999 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings. ICASSP99 (Cat. No.99CH36258).

[23]  Bruno A. Olshausen,et al.  Learning Sparse Multiscale Image Representations , 2002, NIPS.

[24]  Bruno A. Olshausen,et al.  Learning sparse, overcomplete representations of time-varying natural images , 2003, Proceedings 2003 International Conference on Image Processing (Cat. No.03CH37429).

[25]  Mark D. Plumbley,et al.  Sparse representations of polyphonic music , 2006, Signal Process..

[26]  Michael A. Saunders,et al.  Atomic Decomposition by Basis Pursuit , 1998, SIAM J. Sci. Comput..

[27]  Michael Elad,et al.  Multi-Scale Dictionary Learning Using Wavelets , 2011, IEEE Journal of Selected Topics in Signal Processing.

[28]  Y. C. Pati,et al.  Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition , 1993, Proceedings of 27th Asilomar Conference on Signals, Systems and Computers.

[29]  Qiegen Liu,et al.  An augmented Lagrangian multi-scale dictionary learning algorithm , 2011, EURASIP J. Adv. Signal Process..

[30]  Mike E. Davies,et al.  Unsupervised learning of sparse and shift-invariant decompositions of polyphonic music , 2004, 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[31]  Adam M. Packer,et al.  Extracting regions of interest from biological images with convolutional sparse block coding , 2013, NIPS.

[32]  José M. Bioucas-Dias,et al.  Fast Image Recovery Using Variable Splitting and Constrained Optimization , 2009, IEEE Transactions on Image Processing.

[33]  Matthew E. P. Davies,et al.  Shift-invariant sparse coding for single channel blind source separation , 2005 .

[34]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[35]  Jayaraman J. Thiagarajan,et al.  Shift-invariant sparse representation of images using learned dictionaries , 2008, 2008 IEEE Workshop on Machine Learning for Signal Processing.

[36]  Graham W. Taylor,et al.  Deconvolutional networks , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[37]  Koji Kurata,et al.  Properties of basis functions generated by shift invariant sparse representations of natural images , 2000, Biological Cybernetics.

[38]  Guillermo Sapiro,et al.  Non-local sparse models for image restoration , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[39]  Eero P. Simoncelli,et al.  Recovery of Sparse Translation-Invariant Signals With Continuous Basis Pursuit , 2011, IEEE Transactions on Signal Processing.

[40]  Mike E. Davies,et al.  Sparse and shift-Invariant representations of music , 2006, IEEE Transactions on Audio, Speech, and Language Processing.

[41]  Karl Skretting,et al.  General design algorithm for sparse frame expansions , 2006, Signal Process..

[42]  Bruno A. Olshausen,et al.  Learning Sparse Image Codes using a Wavelet Pyramid Architecture , 2000, NIPS.

[43]  B. He,et al.  Alternating Direction Method with Self-Adaptive Penalty Parameters for Monotone Variational Inequalities , 2000 .

[44]  Rémi Gribonval,et al.  Fast matching pursuit with a multiscale dictionary of Gaussian chirps , 2001, IEEE Trans. Signal Process..

[45]  Kjersti Engan,et al.  Family of iterative LS-based dictionary learning algorithms, ILS-DLA, for sparse signal representation , 2007, Digit. Signal Process..

[46]  Yann LeCun,et al.  Pedestrian Detection with Unsupervised Multi-stage Feature Learning , 2012, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[47]  Pierre Vandergheynst,et al.  Shift-invariant dictionary learning for sparse representations: Extending K-SVD , 2008, 2008 16th European Signal Processing Conference.

[48]  Michael Elad,et al.  Learning Multiscale Sparse Representations for Image and Video Restoration , 2007, Multiscale Model. Simul..

[49]  Michael Elad,et al.  Dictionaries for Sparse Representation Modeling , 2010, Proceedings of the IEEE.

[50]  Anders P. Eriksson,et al.  Fast Convolutional Sparse Coding , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[51]  José M. Bioucas-Dias,et al.  An Augmented Lagrangian Approach to the Constrained Optimization Formulation of Imaging Inverse Problems , 2009, IEEE Transactions on Image Processing.

[52]  Brendt Wohlberg,et al.  Piano music transcription with fast convolutional sparse coding , 2015, 2015 IEEE 25th International Workshop on Machine Learning for Signal Processing (MLSP).

[53]  Quentin Barthelemy,et al.  Shift & 2D Rotation Invariant Sparse Coding for Multivariate Signals , 2012, IEEE Transactions on Signal Processing.

[54]  Mikkel N. Schmidt,et al.  Shift Invariant Sparse Coding of Image and Music Data , 2007 .

[55]  Terrence J. Sejnowski,et al.  Coding Time-Varying Signals Using Sparse, Shift-Invariant Representations , 1998, NIPS.

[56]  A. Bruckstein,et al.  K-SVD : An Algorithm for Designing of Overcomplete Dictionaries for Sparse Representation , 2005 .

[57]  Charless C. Fowlkes,et al.  Fast Convolutional Sparse Coding ( FCSC ) , 2014 .

[58]  Thomas S. Huang,et al.  Supervised translation-invariant sparse coding , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[59]  Mark J. Huiskes,et al.  The MIR flickr retrieval evaluation , 2008, MIR '08.

[60]  Y-Lan Boureau,et al.  Learning Convolutional Feature Hierarchies for Visual Recognition , 2010, NIPS.

[61]  Yann LeCun,et al.  Convolutional Matching Pursuit and Dictionary Training , 2010, ArXiv.

[62]  Morten Mørup,et al.  Transformation invariant sparse coding , 2011, 2011 IEEE International Workshop on Machine Learning for Signal Processing.

[63]  William W. Hager,et al.  Updating the Inverse of a Matrix , 1989, SIAM Rev..

[64]  Heiko Wersing,et al.  Sparse Coding with Invariance Constraints , 2003, ICANN.