A Note on Lewicki-Sejnowski Gradient for Learning Overcomplete Representations

Overcomplete representations have greater robustness in noise environment and also have greater flexibility in matching structure in the data. Lewicki and Sejnowski (2000) proposed an efficient extended natural gradient for learning the overcomplete basis and developed an overcomplete representation approach. However, they derived their gradient by many approximations, and their proof is very complicated. To give a stronger theoretical basis, we provide a brief and more rigorous mathematical proof for this gradient in this note. In addition, we propose a more robust constrained Lewicki-Sejnowski gradient.

[1]  Mark A. Girolami,et al.  A Variational Method for Learning Sparse and Overcomplete Representations , 2001, Neural Computation.

[2]  Lucas C. Parra,et al.  Convolutive blind separation of non-stationary sources , 2000, IEEE Trans. Speech Audio Process..

[3]  Terrence J. Sejnowski,et al.  Independent Component Analysis Using an Extended Infomax Algorithm for Mixed Sub-Gaussian and Super-Gaussian Sources , 1999, Neural Comput..

[4]  Yuanqing Li,et al.  K-hyperline clustering learning for sparse component analysis , 2009, Signal Process..

[5]  X. Liao,et al.  Estimating the mixing matrix by using less sparsity , 2009 .

[6]  Andrzej Cichocki,et al.  Estimation of Sparse Nonnegative Sources from Noisy Overcomplete Mixtures Using MAP , 2009, Neural Computation.

[7]  Terrence J. Sejnowski,et al.  Independent Component Analysis Using an Extended Infomax Algorithm for Mixed Subgaussian and Supergaussian Sources , 1999, Neural Computation.

[8]  Terrence J. Sejnowski,et al.  Blind source separation of more sources than mixtures using overcomplete representations , 1999, IEEE Signal Processing Letters.

[9]  Michael Zibulevsky,et al.  Underdetermined blind source separation using sparse representations , 2001, Signal Process..

[10]  ZuYuan Yang,et al.  Source number estimation and separation algorithms of underdetermined blind separation , 2008, Science in China Series F: Information Sciences.

[11]  Shun-ichi Amari,et al.  Natural Gradient Works Efficiently in Learning , 1998, Neural Computation.

[12]  Yuanqing Li,et al.  Analysis of Sparse Representation and Blind Source Separation , 2004, Neural Computation.

[13]  Zhaoshui He,et al.  Convolutive Blind Source Separation in the Frequency Domain Based on Sparse Representation , 2007, IEEE Transactions on Audio, Speech, and Language Processing.

[14]  Guoxu Zhou,et al.  New method for signal encryption using blind source separation based on subband decomposition , 2008 .

[15]  Guoxu Zhou,et al.  A new geometric approach to blind source separation of bounded sources , 2009 .

[16]  Andrzej Cichocki,et al.  A New Learning Algorithm for Blind Signal Separation , 1995, NIPS.

[17]  Terrence J. Sejnowski,et al.  Learning Overcomplete Representations , 2000, Neural Computation.