Fast Sparse Decomposition by Iterative Detection-Estimation

Abstract—Finding sparse solutions of underdetermined sys-tems of linear equations is a fundamental problem in signalprocessing and statistics which has become a subject of interestin recent years. In general, these systems have infinitely ma nysolutions. However, it may be shown that sufficiently sparsesolutions may be identified uniquely. In other words, the cor -responding linear transformation will be invertible if we restrictits domain to sufficiently sparse vectors. This property may beused, for example, to solve the underdetermined Blind SourceSeparation (BSS) problem, or to find sparse representation o fa signal in an ‘overcomplete’ dictionary of primitive elements(i.e., the so-called atomic decomposition). The main drawback ofcurrent methods of finding sparse solutions is their computa tionalcomplexity. In this paper, we will show that by detecting ‘active’components of the (potential) solution, i.e., those componentshaving a considerable value, a framework for fast solution ofthe problem may be devised. The idea leads to a family ofalgorithms, called ‘Iterative Detection-Estimation (IDE)’, whichconverge to the solution by successive detection and estimation ofits active part. Comparing the performance of IDE(s) with oneof the most successful method to date, which is based on LinearProgramming (LP), an improvement in speed of about two tothree orders of magnitude is observed.

[1]  M. V. Van Hulle,et al.  Clustering approach to square and non-square blind source separation , 1999, Neural Networks for Signal Processing IX: Proceedings of the 1999 IEEE Signal Processing Society Workshop (Cat. No.98TH8468).

[2]  Michael Zibulevsky,et al.  Underdetermined blind source separation using sparse representations , 2001, Signal Process..

[3]  D. Donoho,et al.  Thresholds for the Recovery of Sparse Solutions via L1 Minimization , 2006, 2006 40th Annual Conference on Information Sciences and Systems.

[4]  D. Donoho For most large underdetermined systems of linear equations the minimal 𝓁1‐norm solution is also the sparsest solution , 2006 .

[5]  Michael Elad,et al.  A generalized uncertainty principle and sparse representation in pairs of bases , 2002, IEEE Trans. Inf. Theory.

[6]  M. Rudelson,et al.  Geometric approach to error-correcting codes and reconstruction of signals , 2005, math/0502299.

[7]  D K Smith,et al.  Numerical Optimization , 2001, J. Oper. Res. Soc..

[8]  Jean-Jacques Fuchs,et al.  On sparse representations in arbitrary redundant bases , 2004, IEEE Transactions on Information Theory.

[9]  Barak A. Pearlmutter,et al.  Blind Source Separation by Sparse Decomposition in a Signal Dictionary , 2001, Neural Computation.

[10]  David L Donoho,et al.  Compressed sensing , 2006, IEEE Transactions on Information Theory.

[11]  Emmanuel J. Candès,et al.  Near-Optimal Signal Recovery From Random Projections: Universal Encoding Strategies? , 2004, IEEE Transactions on Information Theory.

[12]  Jean-Francois Cardoso,et al.  Blind signal separation: statistical principles , 1998, Proc. IEEE.

[13]  Joel A. Tropp,et al.  Greed is good: algorithmic results for sparse approximation , 2004, IEEE Transactions on Information Theory.

[14]  Christian Jutten,et al.  A new approach for sparse decomposition and sparse source separation , 2006, 2006 14th European Signal Processing Conference.

[15]  L. Scharf,et al.  Statistical Signal Processing: Detection, Estimation, and Time Series Analysis , 1991 .

[16]  Michael Elad,et al.  Optimally sparse representation in general (nonorthogonal) dictionaries via ℓ1 minimization , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[17]  Rémi Gribonval,et al.  A survey of Sparse Component Analysis for blind source separation: principles, perspectives, and new challenges , 2006, ESANN.

[18]  Emmanuel J. Candès,et al.  Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information , 2004, IEEE Transactions on Information Theory.

[19]  Ingrid Daubechies,et al.  Time-frequency localization operators: A geometric phase space approach , 1988, IEEE Trans. Inf. Theory.

[20]  Stéphane Mallat,et al.  Matching pursuits with time-frequency dictionaries , 1993, IEEE Trans. Signal Process..

[21]  Xiaoming Huo,et al.  Uncertainty principles and ideal atomic decomposition , 2001, IEEE Trans. Inf. Theory.

[22]  Christian Jutten,et al.  Sparse ICA via cluster-wise PCA , 2006, Neurocomputing.

[23]  Nicholas I. M. Gould,et al.  On the Solution of Equality Constrained Quadratic Programming Problems Arising in Optimization , 2001, SIAM J. Sci. Comput..

[24]  Christian Jutten,et al.  Space or time adaptive signal processing by neural network models , 1987 .

[25]  S. Amari,et al.  SPARSE COMPONENT ANALYSIS FOR BLIND SOURCE SEPARATION WITH LESS SENSORS THAN SOURCES , 2003 .

[26]  Rémi Gribonval,et al.  Sparse representations in unions of bases , 2003, IEEE Trans. Inf. Theory.

[27]  Michael A. Saunders,et al.  Atomic Decomposition by Basis Pursuit , 1998, SIAM J. Sci. Comput..