Advances in blind source separation (BSS) and independent component analysis (ICA) for nonlinear mixtures

In this paper, we review recent advances in blind source separation (BSS) and independent component analysis (ICA) for nonlinear mixing models. After a general introduction to BSS and ICA, we discuss in more detail uniqueness and separability issues, presenting some new results. A fundamental difficulty in the nonlinear BSS problem and even more so in the nonlinear ICA problem is that they provide non-unique solutions without extra constraints, which are often implemented by using a suitable regularization. In this paper, we explore two possible approaches. The first one is based on structural constraints. Especially, post-nonlinear mixtures are an important special case, where a nonlinearity is applied to linear mixtures. For such mixtures, the ambiguities are essentially the same as for the linear ICA or BSS problems. The second approach uses Bayesian inference methods for estimating the best statistical parameters, under almost unconstrained models in which priors can be easily added. In the later part of this paper, various separation techniques proposed for post-nonlinear mixtures and general nonlinear mixtures are reviewed.

[1]  Antti Honkela,et al.  Bayesian Non-Linear Independent Component Analysis by Multi-Layer Perceptrons , 2000 .

[2]  Christian Jutten,et al.  Nonlinear source separation: the post-nonlinear mixtures , 1997, ESANN.

[3]  Lei Xu,et al.  Temporal BYY learning for state space approach, hidden Markov model, and blind source separation , 2000, IEEE Trans. Signal Process..

[4]  Jacek M. Zurada,et al.  Nonlinear Blind Source Separation Using a Radial Basis Function Network , 2001 .

[5]  Gustavo Deco,et al.  Nonlinear higher-order statistical decorrelation by volume-conserving neural architectures , 1995, Neural Networks.

[6]  E. Lukács.  A Characterization of the Gamma Distribution , 1955 .

[7]  Aapo Hyvärinen,et al.  Emergence of Phase- and Shift-Invariant Features by Decomposition of Natural Images into Independent Feature Subspaces , 2000, Neural Computation.

[8]  J. Cardoso,et al.  Blind beamforming for non-gaussian signals , 1993 .

[9]  Fabian J. Theis,et al.  Pattern Repulsion Revisited , 2001, IWANN.

[10]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[11]  Raffaele Parisi,et al.  BLIND SOURCE SEPARATION IN NONLINEAR MIXTURES BY ADAPTIVE SPLINE NEURAL NETWORKS , 2001 .

[12]  Terrence J. Sejnowski,et al.  Blind separation and blind deconvolution: an information-theoretic approach , 1995, 1995 International Conference on Acoustics, Speech, and Signal Processing.

[13]  Juha Karhunen,et al.  A Maximum Likelihood Approach to Nonlinear Blind Source Separation , 1997, ICANN.

[14]  Dinh-Tuan Pham,et al.  Improving algorithm speed in PNL mixture separation and Wiener system inversion , 2003 .

[15]  Richard M. Everson,et al.  Independent Component Analysis: Principles and Practice , 2001 .

[16]  Juha Karhunen,et al.  An Ensemble Learning Approach to Nonlinear Independent Component Analysis , 2001 .

[17]  Juha Karhunen,et al.  An ensemble learning approach to nonlinear dynamic blind source separation using state-space models , 2002, Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN'02 (Cat. No.02CH37290).

[18]  Christian Jutten,et al.  Source separation in post-nonlinear mixtures , 1999, IEEE Trans. Signal Process..

[19]  E. Oja,et al.  Nonlinear Blind Source Separation by Variational Bayesian Learning , 2003, IEICE Trans. Fundam. Electron. Commun. Comput. Sci..

[20]  Barak A. Pearlmutter,et al.  Blind Source Separation by Sparse Decomposition in a Signal Dictionary , 2001, Neural Computation.

[21]  Christian Jutten,et al.  Quasi-nonparametric blind inversion of Wiener systems , 2001, IEEE Trans. Signal Process..

[22]  Ivica Kostanic,et al.  Principles of Neurocomputing for Science and Engineering , 2000 .

[23]  Christian Jutten,et al.  Entropy Optimization - Application to Blind Source Separation , 1997, ICANN.

[24]  Andreas Ziehe,et al.  Blind separation of post-nonlinear mixtures using gaussianizing transformations and temporal decorrelation , 2003 .

[25]  Andrzej Cichocki,et al.  Information-theoretic approach to blind separation of sources in non-linear mixture , 1998, Signal Process..

[26]  Jan Eriksson,et al.  Contributions to theory and algorithms of independent component analysis and signal separation , 2004 .

[27]  Terrence J. Sejnowski,et al.  ICA Mixture Models for Unsupervised Classification of Non-Gaussian Classes and Automatic Context Switching in Blind Signal Separation , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[28]  T. Sejnowski,et al.  Separation of post-nonlinear mixtures using ACE and temporal decorrelation , 2001 .

[29]  Heekuck Oh,et al.  Neural Networks for Pattern Recognition , 1993, Adv. Comput..

[30]  Juan K. Lin,et al.  Feature extraction approach to blind source separation , 1997, Neural Networks for Signal Processing VII. Proceedings of the 1997 IEEE Signal Processing Society Workshop.

[31]  Hagai Attias,et al.  Independent Factor Analysis , 1999, Neural Computation.

[32]  Alexander Ilin,et al.  On the Effect of the Form of the Posterior Approximation in Variational Learning of ICA Models , 2005, Neural Processing Letters.

[33]  최승진.  Nonlinear dynamic independent component analysis using state-space and neural network models , 1999 .

[34]  C. Jutten.  Calcul neuromimétique et traitement du signal : analyse en composantes indépendantes , 1987 .

[35]  Jean-François Cardoso,et al.  Equivariant adaptive source separation , 1996, IEEE Trans. Signal Process..

[36]  Visa Koivunen,et al.  Blind identifiability of class of nonlinear instantaneous ICA models , 2002, 2002 11th European Signal Processing Conference.

[37]  Lucas C. Parra,et al.  Symplectic Nonlinear Component Analysis , 1995, NIPS.

[38]  A. Honkela.  Speeding up cyclic update schemes by pattern searches , 2002, Proceedings of the 9th International Conference on Neural Information Processing, 2002. ICONIP '02..

[39]  Calyampudi R. Rao,et al.  Extension of Darmois-Skitovic Theorem to Functions of Random Variables Satisfying an Addition Theorem , 1973 .

[40]  Wan-Chi Siu,et al.  A Semi-Parametric Hybrid Neural Model for Nonlinear Blind Signal Separation , 2000, Int. J. Neural Syst..

[41]  Pierre-Olivier Amblard,et al.  Blind Equalization of a Nonlinear Satellite System Using MCMC Simulation Methods , 2002, EURASIP J. Adv. Signal Process..

[42]  R. Liu,et al.  AMUSE: a new blind identification algorithm , 1990, IEEE International Symposium on Circuits and Systems.

[43]  Erkki Oja,et al.  DETECTING PROCESS STATE CHANGES BY NONLINEAR BLIND SOURCE SEPARATION , 2003 .

[44]  L. Parra,et al.  Redundancy reduction with information-preserving nonlinear maps , 1995 .

[45]  Erkki Oja,et al.  Nonlinear dynamical factor analysis for state change detection , 2004, IEEE Transactions on Neural Networks.

[46]  Terrence J. Sejnowski,et al.  Nonlinear blind source separation using kernel feature spaces , 2001 .

[47]  Aapo Hyvärinen,et al.  Nonlinear independent component analysis: Existence and uniqueness results , 1999, Neural Networks.

[48]  Jean-Francois Cardoso,et al.  Blind signal separation: statistical principles , 1998, Proc. IEEE.

[49]  Terrence J. Sejnowski,et al.  Variational Learning of Clusters of Undercomplete Nonsymmetric Independent Components , 2003, J. Mach. Learn. Res..

[50]  Jürgen Schmidhuber,et al.  Feature Extraction Through LOCOCODE , 1999, Neural Computation.

[51]  Christian Jutten,et al.  Identifiability of post-nonlinear mixtures , 2005, IEEE Signal Processing Letters.

[52]  Hujun Yin,et al.  Image denoising using self-organizing map-based nonlinear independent component analysis , 2002, Neural Networks.

[53]  Juan K. Lin,et al.  Faithful Representation of Separable Distributions , 1997, Neural Computation.

[54]  Andrzej Cichocki,et al.  Adaptive Blind Signal and Image Processing - Learning Algorithms and Applications , 2002 .

[55]  Michael Herrmann,et al.  Perspectives and limitations of self-organizing maps , 1996 .

[56]  Christian Jutten,et al.  MINIMIZATION-PROJECTION (MP) APPROACH FOR BLIND SOURCE SEPARATION IN DIFFERENT MIXING MODELS , 2003 .

[57]  Antti Honkela,et al.  Variational learning and bits-back coding: an information-theoretic view to Bayesian learning , 2004, IEEE Transactions on Neural Networks.

[58]  Christian Jutten,et al.  Three easy ways for separating nonlinear mixtures? , 2004, Signal Process..

[59]  Juha Karhunen,et al.  Accelerating Cyclic Update Algorithms for Parameter Estimation by Pattern Searches , 2003, Neural Processing Letters.

[60]  Adrian M. Ionescu,et al.  High Performance Magnetic Field Smart Sensor Arrays with Source Separation , 1998 .

[61]  Pierre Comon,et al.  Independent component analysis, A new concept? , 1994, Signal Process..

[62]  Luís B. Almeida,et al.  Separating a Real-Life Nonlinear Mixture of Images , 2004, ICA.

[63]  Luís B. Almeida,et al.  Linear and nonlinear ICA based on mutual information - the MISEP method , 2004, Signal Process..

[64]  Christian Jutten,et al.  A geometric approach for separating post non-linear mixtures , 2002, 2002 11th European Signal Processing Conference.

[65]  A. Hyvärinen,et al.  Nonlinear Blind Source Separation by Self-Organizing Maps , 1996 .

[66]  Alberto Prieto,et al.  Separation of sources in a class of post-nonlinear mixtures , 1998, ESANN.

[67]  Eric Moulines,et al.  A blind source separation technique using second-order statistics , 1997, IEEE Trans. Signal Process..

[68]  Geoffrey E. Hinton,et al.  Keeping the neural networks simple by minimizing the description length of the weights , 1993, COLT '93.

[69]  Christian Jutten,et al.  Markovian Source Separation in Post-nonlinear Mixtures , 2004, ICA.

[70]  Dimitrios Hatzinakos,et al.  Blind identification of LTI-ZMNL-LTI nonlinear channel models , 1995, IEEE Trans. Signal Process..

[71]  Michael C. Mozer,et al.  An electric field approach to independent component analysis , 2000 .

[72]  D. Chakrabarti,et al.  A fast fixed - point algorithm for independent component analysis , 1997 .

[73]  C. Jutten,et al.  Improving semiconductor-based chemical sensor arrays using advanced algorithms for blind source separation , 2004, ISA/IEEE Sensors for Industry Conference, 2004. Proceedings the.

[74]  Christian Jutten,et al.  Parametric approach to blind deconvolution of nonlinear channels , 2002, ESANN.

[75]  Michel Verleysen,et al.  Non-linear ICA by Using Isometric Dimensionality Reduction , 2004, ICA.

[76]  R Hecht-Nielsen,et al.  Replicator neural networks for universal optimal source coding. , 1995, Science.

[77]  Christopher M. Bishop,et al.  GTM: The Generative Topographic Mapping , 1998, Neural Computation.

[78]  J. Príncipe,et al.  Entropy manipulation of arbitrary nonlinear mappings , 1997, Neural Networks for Signal Processing VII. Proceedings of the 1997 IEEE Signal Processing Society Workshop.

[79]  Michael I. Jordan,et al.  Kernel independent component analysis , 2003 .

[80]  Te-Won Lee,et al.  Blind source separation of nonlinear mixing models , 1997, Neural Networks for Signal Processing VII. Proceedings of the 1997 IEEE Signal Processing Society Workshop.

[81]  M. G. Middelhoek.  Hall Effect Devices: Second Edition; R.S. Popovic; Institute of Physics, Bristol, 2003, 419 pages, hardback, ISBN 0-7503-30855-9, £ 90.00 , 2004 .

[82]  Juha Karhunen,et al.  Nonlinear Independent Component Analysis Using Ensemble Learning: Experiments and Discussion , 2000 .

[83]  Dinh-Tuan Pham,et al.  Blind separation of instantaneous mixtures of nonstationary sources , 2001, IEEE Trans. Signal Process..

[84]  Javier R. Movellan,et al.  DIFFUSION NETWORKS , PRODUCT OF EXPERTS , AND FACTOR ANALYSIS , 2001 .

[85]  Antti Honkela,et al.  Using Kernel PCA for Initialisation of Variational Bayesian Nonlinear Blind Source Separation Method , 2004, ICA.

[86]  Harri Lappalainen,et al.  Ensemble learning for independent component analysis , 1999 .

[87]  Harri Valpola.  Nonlinear independent component analysis using ensemble learning: Theory , 2000 .

[88]  Fabian J. Theis,et al.  A New Concept for Separability Problems in Blind Source Separation , 2004, Neural Computation.

[89]  Christian Jutten,et al.  Differential of the mutual information , 2004, IEEE Signal Processing Letters.

[90]  Antti Honkela,et al.  Post-nonlinear Independent Component Analysis by Variational Bayesian Learning , 2004, ICA.

[91]  J. Aczél,et al.  Lectures on Functional Equations and Their Applications , 1968 .

[92]  Juha Karhunen,et al.  Hierarchical models of variance sources , 2004, Signal Process..

[93]  A. Honkela.  Approximating nonlinear transformations of probability distributions for nonlinear independent component analysis , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).

[94]  Fabian J. Theis,et al.  Separability of analytic postnonlinear blind source separation with bounded sources , 2004, ESANN.

[95]  C. Jutten,et al.  On the separability of nonlinear mixtures of temporally correlated sources , 2003, IEEE Signal Processing Letters.

[96]  Andrzej Cichocki,et al.  Robust learning algorithm for blind separation of signals , 1994 .

[97]  Anisse Taleb,et al.  A generic framework for blind source separation in structured nonlinear models , 2002, IEEE Trans. Signal Process..

[98]  Juha Karhunen,et al.  Missing Values in Hierarchical Nonlinear Factor Analysis , 2003 .

[99]  Ziyou Xiong,et al.  NONLINEAR INDEPENDENT COMPONENT ANALYSIS(ICA) USING POWER SERIES AND APPLICATION TO BLIND SOURCE SEPARATION , 2001 .

[100]  George K. Kokkinakis,et al.  Neural Network Based Blind Source Separation of Non-linear Mixtures , 2001, ICANN.

[101]  Juha Karhunen,et al.  Local Linear Independent Component Analysis Based on Clustering , 2000, Int. J. Neural Syst..

[102]  Christian Jutten,et al.  Source separation based processing for integrated Hall sensor arrays , 2002 .

[103]  Motoaki Kawanabe,et al.  Kernel-Based Nonlinear Blind Source Separation , 2003, Neural Computation.

[104]  Gilles Burel,et al.  Blind separation of sources: A nonlinear neural algorithm , 1992, Neural Networks.

[105]  Kiyotoshi Matsuoka,et al.  A neural net for blind separation of nonstationary signals , 1995, Neural Networks.

[106]  Richard M. Everson,et al.  Image processing methods using ICA mixture models , 2001 .

[107]  S. Haykin.  Unsupervised adaptive filtering, vol. 1: Blind source separation , 2000 .

[108]  Fabian J Theis,et al.  Postnonlinear blind source separation via linearization identification , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).

[109]  Dinh-Tuan Pham,et al.  Fast approximation of nonlinearities for improving inversion algorithms of PNL mixtures and Wiener systems , 2005, Signal Process..

[110]  J. Karhunen,et al.  Nonlinear Independent Factor Analysis by Hierarchical Models , 2003 .

[111]  Dinh Tuan Pham,et al.  BLIND SOURCE SEPARATION IN POST NONLINEAR MIXTURES , 2001 .

[112]  Fabian J. Theis,et al.  Maximum Entropy and Minimal Mutual Information in a Nonlinear Model , 2001 .

[113]  Lucas C. Parra,et al.  Statistical Independence and Novelty Detection with Information Preserving Nonlinear Maps , 1996, Neural Computation.

[114]  Luís B. Almeida,et al.  MISEP -- Linear and Nonlinear ICA Based on Mutual Information , 2003, J. Mach. Learn. Res..

[115]  Pei Ling Lai,et al.  Ica Using Kernel Canonical Correlation Analysis , 2000 .

[116]  Petteri Pajunen,et al.  Nonlinear Blind Source Separation by Self-Organizing Maps , 1996 .

[117]  Andrzej Cichocki,et al.  A New Learning Algorithm for Blind Signal Separation , 1995, NIPS.

[118]  Stephen J. Roberts,et al.  Hierarchy, priors and wavelets: structure and signal modelling using ICA , 2004, Signal Process..

[119]  M. J. Korenberg,et al.  The identification of nonlinear biological systems: LNL cascade models , 1986, Biological Cybernetics.

[120]  Juha Karhunen,et al.  An Unsupervised Ensemble Learning Method for Nonlinear Dynamic State-Space Models , 2002, Neural Computation.

[121]  C. Jutten,et al.  Bayesian versus constrained structure approaches for source separation in post-nonlinear mixtures , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).

[122]  Christian Jutten,et al.  Blind separation of sources, part I: An adaptive algorithm based on neuromimetic architecture , 1991, Signal Process..

[123]  F. Palmieri,et al.  Multi-Class Independent Component Analysis (MUCICA) for Rank-Deficient Distributions , 2000 .

[124]  Pierre Comon.  Independent component analysis - a new concept? signal processing , 1994 .

[125]  Massoud Babaiezadeh Malmiri.  On blind source separation in convolutive and nonlinear mixtures , 2002 .