A Unifying Information-Theoretic Framework for Independent Component Analysis

We show that diierent theories recently proposed for Independent Component Analysis (ICA) lead to the same iterative learning algorithm for blind separation of mixed independent sources. We review those theories and suggest that information theory can be used to unify several lines of research. Pearlmutter and Parra (1996) and Cardoso (1997) showed that the infomax approach of Bell and Sejnowski (1995) and the maximum likelihood estimation approach are equivalent. We show that negentropy maximization also has equivalent properties and therefore all three approaches yield the same learning rule for a xed nonlinearity. Girolami and Fyfe (1997a) have shown that the nonlinear Principal Component Analysis (PCA) algorithm of Karhunen and Joutsensalo (1994) and Oja (1997) can also be viewed from information-theoretic principles since it minimizes the sum of squares of the fourth-order marginal cumulants and therefore approximately minimizes the mutual information (Comon, 1994). Lambert (1996) has proposed diierent Bussgang cost functions for multichannel blind deconvolution. We show how the Bussgang property relates to the infomax principle. Finally, we discuss convergence and stability as well as future research issues in blind source separation.

[1]  John G. Proakis,et al.  Probability, random variables and stochastic processes , 1985, IEEE Trans. Acoust. Speech Signal Process..

[2]  Robin Sibson,et al.  What is projection pursuit , 1987 .

[3]  Christian Jutten,et al.  Space or time adaptive signal processing by neural network models , 1987 .

[4]  Christian Jutten,et al.  Blind separation of sources, part I: An adaptive algorithm based on neuromimetic architecture , 1991, Signal Process..

[5]  J J Hopfield,et al.  Olfactory computation and object perception. , 1991, Proceedings of the National Academy of Sciences of the United States of America.

[6]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[7]  Ralph Linsker,et al.  Local Synaptic Learning Rules Suffice to Maximize Mutual Information in a Linear Network , 1992, Neural Computation.

[8]  Dinh Tuan Pham,et al.  Separation of a mixture of independent sources through a maximum likelihood approach , 1992 .

[9]  Andreas G. Andreou,et al.  Current-mode subthreshold MOS implementation of the Herault-Jutten autoadaptive network , 1992 .

[10]  J. Nadal.  Non linear neurons in the low noise limit : a factorial code maximizes information transferJean , 1994 .

[11]  Juha Karhunen,et al.  Representation and separation of signals using nonlinear PCA type learning , 1994, Neural Networks.

[12]  Pierre Comon,et al.  Independent component analysis, A new concept? , 1994, Signal Process..

[13]  C. D. Kemp,et al.  Kendall's Advanced Theory of Statistics, Vol. 1: Distribution Theory. , 1995 .

[14]  Andrzej Cichocki,et al.  Robust learning algorithm for blind separation of signals , 1994 .

[15]  Terrence J. Sejnowski,et al.  An Information-Maximization Approach to Blind Separation and Blind Deconvolution , 1995, Neural Computation.

[16]  Terrence J. Sejnowski,et al.  Adaptive separation of mixed broadband sound sources with delays by a beamforming Herault-Jutten network , 1995 .

[17]  S. Klinke,et al.  Exploratory Projection Pursuit , 1995 .

[18]  Erkki Oja,et al.  Signal Separation by Nonlinear Hebbian Learning , 1995 .

[19]  Tzyy-Ping Jung,et al.  Independent Component Analysis of Electroencephalographic Data , 1995, NIPS.

[20]  Andrzej Cichocki,et al.  A New Learning Algorithm for Blind Signal Separation , 1995, NIPS.

[21]  G. Deco,et al.  An Information-Theoretic Approach to Neural Computing , 1997, Perspectives in Neural Computing.

[22]  Yoram Baram,et al.  Multidimensional density shaping by sigmoids , 1996, IEEE Trans. Neural Networks.

[23]  Kari Torkkola,et al.  Blind separation of convolved sources based on information maximization , 1996, Neural Networks for Signal Processing VI. Proceedings of the 1996 IEEE Signal Processing Society Workshop.

[24]  Dinh-Tuan Pham,et al.  Blind separation of instantaneous mixture of sources via an independent component analysis , 1996, IEEE Trans. Signal Process..

[25]  Barak A. Pearlmutter,et al.  A Context-Sensitive Generalization of ICA , 1996 .

[26]  R. Lambert.  Multichannel blind deconvolution: FIR matrix algebra and separation of multipath mixtures , 1996 .

[27]  Jean-François Cardoso,et al.  Equivariant adaptive source separation , 1996, IEEE Trans. Signal Process..

[28]  Juha Karhunen,et al.  Neural approaches to independent component analysis and source separation , 1996, ESANN.

[29]  Andreas Ziehe,et al.  Adaptive On-line Learning in Changing Environments , 1996, NIPS.

[30]  Marian Stewart Bartlett,et al.  Viewpoint Invariant Face Recognition using Independent Component Analysis and Attractor Networks , 1996, NIPS.

[31]  Ehud Weinstein,et al.  Multichannel signal separation: methods and analysis , 1996, IEEE Trans. Signal Process..

[32]  Erkki Oja,et al.  The nonlinear PCA learning rule in independent component analysis , 1997, Neurocomputing.

[33]  C. Fyfe,et al.  Generalised independent component analysis through unsupervised learning with emergent Bussgang properties , 1997, Proceedings of International Conference on Neural Networks (ICNN'97).

[34]  Erkki Oja,et al.  A class of neural networks for independent component analysis , 1997, IEEE Trans. Neural Networks.

[35]  Ralph Linsker,et al.  A Local Learning Rule That Enables Information Maximization for Arbitrary Input Distributions , 1997, Neural Computation.

[36]  Juha Karhunen,et al.  A Maximum Likelihood Approach to Nonlinear Blind Source Separation , 1997, ICANN.

[37]  Tzyy-Ping Jung,et al.  Extended ICA Removes Artifacts from Electroencephalographic Recordings , 1997, NIPS.

[38]  Russell H. Lambert,et al.  Blind separation of multiple speakers in a multipath environment , 1997, 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[39]  Jack D. Cowan,et al.  Faithful Representation Of Separable Input Distributions , 1997, Neural Computation.

[40]  Reinhold Orglmeister,et al.  Blind source separation of real world signals , 1997, Proceedings of International Conference on Neural Networks (ICNN'97).

[41]  Christian Jutten,et al.  Nonlinear source separation: the post-nonlinear mixtures , 1997, ESANN.

[42]  Geoffrey E. Hinton,et al.  Generative models for discovering sparse distributed representations. , 1997, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[43]  Shun-ichi Amari,et al.  Adaptive Online Learning Algorithms for Blind Separation: Maximum Entropy and Minimum Mutual Information , 1997, Neural Computation.

[44]  Eric Moulines,et al.  Maximum likelihood for blind separation and deconvolution of noisy signals using mixture models , 1997, 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[45]  Colin Fyfe,et al.  Stochastic ICA Contrast Maximisation Using Oja's Nonlinear PCA Algorithm , 1997, Int. J. Neural Syst..

[46]  Shun-ichi Amari,et al.  Stability Analysis Of Adaptive Blind Source Separation , 1997 .

[47]  J. Cardoso.  Infomax and maximum likelihood for blind source separation , 1997, IEEE Signal Processing Letters.

[48]  Kari Torkkola.  Blind Separation of Radio Signals in Fading Channels , 1997, NIPS.

[49]  Te-Won Lee,et al.  Blind source separation of nonlinear mixing models , 1997, Neural Networks for Signal Processing VII. Proceedings of the 1997 IEEE Signal Processing Society Workshop.

[50]  Terrence J. Sejnowski,et al.  The “independent components” of natural scenes are edge filters , 1997, Vision Research.

[51]  Shun-ichi Amari,et al.  Neural Network Models for Blind Separation of Time Delayed and Convolved Signals , 1997 .

[52]  Terrence J. Sejnowski,et al.  Learning Nonlinear Overcomplete Representations for Efficient Coding , 1997, NIPS.

[53]  Néstor Parga,et al.  Redundancy Reduction and Independent Component Analysis: Conditions on Cumulants and Adaptive Approaches , 1997, Neural Computation.

[54]  Te-Won Lee,et al.  Independent Component Analysis , 1998, Springer US.

[55]  Mark A. Girolami,et al.  An Alternative Perspective on Adaptive Independent Component Analysis Algorithms , 1998, Neural Computation.

[56]  Terrence J. Sejnowski,et al.  Independent Component Analysis Using an Extended Infomax Algorithm for Mixed Sub-Gaussian and Super-Gaussian Sources , 1999, Neural Comput..

[57]  S. Amari.  Natural Gradient Works Eciently in Learning , 2022 .