A general approach for mutual information minimization and its application to blind source separation

In this paper, a nonparametric "gradient" of the mutual information is first introduced. It is used for showing that mutual information has no local minima. Using the introduced "gradient", two general gradient based approaches for minimizing mutual information in a parametric model are then presented. These approaches are quite general, and principally they can be used in any mutual information minimization problem. In blind source separation, these approaches provide powerful tools for separating any complicated (yet separable) mixing model. In this paper, they are used to develop algorithms for separating four separable mixing models: linear instantaneous, linear convolutive, post nonlinear (PNL) and convolutive post nonlinear (CPNL) mixtures.

[1]  Christian Jutten,et al.  MINIMIZATION-PROJECTION (MP) APPROACH FOR BLIND SOURCE SEPARATION IN DIFFERENT MIXING MODELS , 2003 .

[2]  Juha Karhunen,et al.  An Unsupervised Ensemble Learning Method for Nonlinear Dynamic State-Space Models , 2002, Neural Computation.

[3]  Dinh-Tuan Pham,et al.  Mutual information approach to blind separation of stationary sources , 2002, IEEE Trans. Inf. Theory.

[4]  Michel Verleysen,et al.  On the entropy minimization of a linear mixture of variables for source separation , 2005, Signal Process..

[5]  E. Oja,et al.  Independent Component Analysis , 2013 .

[6]  Luís B. Almeida,et al.  MISEP -- Linear and Nonlinear ICA Based on Mutual Information , 2003, J. Mach. Learn. Res..

[7]  Carine Simon.  Séparation aveugle des sources en mélange convolutif , 1999 .

[8]  Keinosuke Fukunaga,et al.  Introduction to statistical pattern recognition (2nd ed.) , 1990 .

[9]  Christian Jutten,et al.  Differential of the mutual information , 2004, IEEE Signal Processing Letters.

[10]  D. Cox.  A penalty method for nonparametric estimation of the logarithmic derivative of a density function , 1985 .

[11]  Christian Jutten,et al.  Source separation in post-nonlinear mixtures , 1999, IEEE Trans. Signal Process..

[12]  Shun-ichi Amari,et al.  Natural Gradient Works Efficiently in Learning , 1998, Neural Computation.

[13]  Jean-François Cardoso,et al.  Equivariant adaptive source separation , 1996, IEEE Trans. Signal Process..

[14]  Christian Jutten,et al.  Entropy Optimization - Application to Blind Source Separation , 1997, ICANN.

[15]  Christian Jutten,et al.  Parametric approach to blind deconvolution of nonlinear channels , 2002, ESANN.

[16]  Jean-Francois Cardoso,et al.  Blind signal separation: statistical principles , 1998, Proc. IEEE.

[17]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[18]  W. Härdle.  Smoothing Techniques: With Implementation in S , 1991 .

[19]  Christian Jutten,et al.  Three easy ways for separating nonlinear mixtures? , 2004, Signal Process..

[20]  P. J. Green,et al.  Density Estimation for Statistics and Data Analysis , 1987 .

[21]  D. Pham.  FAST ALGORITHM FOR ESTIMATING MUTUAL INFORMATION, ENTROPIES AND SCORE FUNCTIONS , 2003 .

[22]  Pierre Comon,et al.  Independent component analysis, A new concept? , 1994, Signal Process..

[23]  John G. Proakis,et al.  Probability, random variables and stochastic processes , 1985, IEEE Trans. Acoust. Speech Signal Process..

[24]  John W. Fisher,et al.  ICA Using Spacings Estimates of Entropy , 2003, J. Mach. Learn. Res..

[25]  Christian Jutten,et al.  Separating Convolutive Mixtures by Mutual Information Minimization , 2001, IWANN.

[26]  Christian Jutten,et al.  A minimization-projection (MP) approach for blind separating convolutive mixtures , 2004, 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[27]  L. Almeida,et al.  ICA of linear and nonlinear mixtures based on mutual information , 2001, IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222).

[28]  Ehud Weinstein,et al.  Criteria for multichannel signal separation , 1994, IEEE Trans. Signal Process..

[29]  Christian Jutten,et al.  A geometric approach for separating post non-linear mixtures , 2002, 2002 11th European Signal Processing Conference.

[30]  Christian Jutten,et al.  BLIND SEPARATING CONVOLUTIVE POST NON-LINEAR MIXTURES , 2001 .

[31]  Christian Jutten,et al.  USING MULTIVARIATE SCORE FUNCTIONS IN SOURCE SEPARATION: APPLICATION TO POST NON-LINEAR MIXTURES , 2002 .

[32]  P. Laguna,et al.  Signal Processing , 2002, Yearbook of Medical Informatics.

[33]  M. C. Jones,et al.  Spline Smoothing and Nonparametric Regression. , 1989 .

[34]  L. B. Almeida,et al.  Linear and nonlinear ICA based on mutual information , 2000, Proceedings of the IEEE 2000 Adaptive Systems for Signal Processing, Communications, and Control Symposium (Cat. No.00EX373).

[35]  Pavel Pudil,et al.  Introduction to Statistical Pattern Recognition , 2006 .