Nonlinear Source Separation

Abstract The purpose of this lecture book is to present the state of the art in nonlinear blind source separation, in a form appropriate for students, researchers and developers. Source separation deals with the problem of recovering sources that are observed in a mixed condition. When we have little knowledge about the sources and about the mixture process, we speak of blind source separation. Linear blind source separation is a relatively well studied subject, however nonlinear blind source separation is still in a less advanced stage, but has seen several significant developments in the last few years. This publication reviews the main nonlinear separation methods, including the separation of post-nonlinear mixtures, and the MISEP, ensemble learning and kTDSEP methods for generic mixtures. These methods are studied with a significant depth. A historical overview is also presented, mentioning most of the relevant results, on nonlinear blind source separation, that have been presented over the years.

[1]  Juha Karhunen,et al.  Nonlinear Independent Component Analysis Using Ensemble Learning: Experiments and Discussion , 2000 .

[2]  J. Karhunen,et al.  Building Blocks for Hierarchical Latent Variable Models , 2001 .

[3]  A. Honkela.  Speeding up cyclic update schemes by pattern searches , 2002, Proceedings of the 9th International Conference on Neural Information Processing, 2002. ICONIP '02..

[4]  Luís B. Almeida,et al.  Linear and nonlinear ICA based on mutual information - the MISEP method , 2004, Signal Process..

[5]  Christian Jutten,et al.  A geometric approach for separating post non-linear mixtures , 2002, 2002 11th European Signal Processing Conference.

[6]  C. E. SHANNON,et al.  A mathematical theory of communication , 1948, MOCO.

[7]  E. Lorenz.  Deterministic nonperiodic flow , 1963 .

[8]  Motoaki Kawanabe,et al.  Blind Separation of Post-nonlinear Mixtures using Linearizing Transformations and Temporal Decorrelation , 2003, J. Mach. Learn. Res..

[9]  Andrzej Cichocki,et al.  Information-theoretic approach to blind separation of sources in non-linear mixture , 1998, Signal Process..

[10]  Eric Moulines,et al.  A blind source separation technique using second-order statistics , 1997, IEEE Trans. Signal Process..

[11]  Antti Honkela,et al.  Post-nonlinear Independent Component Analysis by Variational Bayesian Learning , 2004, ICA.

[12]  Antti Honkela,et al.  Bayesian Non-Linear Independent Component Analysis by Multi-Layer Perceptrons , 2000 .

[13]  Terrence J. Sejnowski,et al.  Independent Component Analysis Using an Extended Infomax Algorithm for Mixed Subgaussian and Supergaussian Sources , 1999, Neural Computation.

[14]  Luis B. Almeida,et al.  Faster Training in Nonlinear ICA using MISEP , 2002 .

[15]  Aapo Hyvärinen,et al.  Nonlinear independent component analysis: Existence and uniqueness results , 1999, Neural Networks.

[16]  Jürgen Schmidhuber,et al.  Feature Extraction Through LOCOCODE , 1999, Neural Computation.

[17]  H. Lappalainen,et al.  Multilayer perceptrons as nonlinear generative models for unsupervised learning: a Bayesian treatment , 1999 .

[18]  Hujun Yin,et al.  Image denoising using self-organizing map-based nonlinear independent component analysis , 2002, Neural Networks.

[19]  Juan K. Lin,et al.  Faithful Representation of Separable Distributions , 1997, Neural Computation.

[20]  John W. Fisher,et al.  A novel measure for independent component analysis (ICA) , 1998, Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP '98 (Cat. No.98CH36181).

[21]  Christian Jutten,et al.  Nonlinear source separation: the post-nonlinear mixtures , 1997, ESANN.

[22]  Juha Karhunen,et al.  Hierarchical models of variance sources , 2004, Signal Process..

[23]  Gunnar Rätsch,et al.  An introduction to kernel-based learning algorithms , 2001, IEEE Trans. Neural Networks.

[24]  L. B. Almeida,et al.  An Objective Function for Independence , 1996, Proceedings of International Conference on Neural Networks (ICNN'96).

[25]  Jacek M. Zurada,et al.  Nonlinear Blind Source Separation Using a Radial Basis Function Network , 2001 .

[26]  Terrence J. Sejnowski,et al.  An Information-Maximization Approach to Blind Separation and Blind Deconvolution , 1995, Neural Computation.

[27]  Erkki Oja,et al.  DETECTING PROCESS STATE CHANGES BY NONLINEAR BLIND SOURCE SEPARATION , 2003 .

[28]  Juha Karhunen,et al.  Nonlinear source separation using ensemble learning and MLP networks , 2000, Proceedings of the IEEE 2000 Adaptive Systems for Signal Processing, Communications, and Control Symposium (Cat. No.00EX373).

[29]  Gustavo Deco,et al.  Nonlinear higher-order statistical decorrelation by volume-conserving neural architectures , 1995, Neural Networks.

[30]  Harri Valpola,et al.  Separation of Nonlinear Image Mixtures by Denoising Source Separation , 2006, ICA.

[31]  Andreas Ziehe,et al.  TDSEP — an efficient algorithm for blind separation using time structure , 1998 .

[32]  Bin Yu,et al.  Model Selection and the Principle of Minimum Description Length , 2001 .

[33]  Te-Won Lee,et al.  Blind source separation of nonlinear mixing models , 1997, Neural Networks for Signal Processing VII. Proceedings of the 1997 IEEE Signal Processing Society Workshop.

[34]  Christian Jutten,et al.  Parametric approach to blind deconvolution of nonlinear channels , 2002, ESANN.

[35]  Vwani P. Roychowdhury,et al.  Independent component analysis based on nonparametric density estimation , 2004, IEEE Transactions on Neural Networks.

[36]  Antti Honkela,et al.  Using Kernel PCA for Initialisation of Variational Bayesian Nonlinear Blind Source Separation Method , 2004 .

[37]  Laurenz Wiskott,et al.  Independent Slow Feature Analysis and Nonlinear Blind Source Separation , 2004, ICA.

[38]  Schuster,et al.  Separation of a mixture of independent signals using time delayed correlations. , 1994, Physical review letters.

[39]  Luís B. Almeida,et al.  Separating a Real-Life Nonlinear Image Mixture , 2005, J. Mach. Learn. Res..

[40]  Juha Karhunen,et al.  Accelerating Cyclic Update Algorithms for Parameter Estimation by Pattern Searches , 2003, Neural Processing Letters.

[41]  S. Haykin,et al.  Making sense of a complex world [chaotic events modeling] , 1998, IEEE Signal Process. Mag..

[42]  J. Rissanen,et al.  Modeling By Shortest Data Description* , 1978, Autom..

[43]  Toshihisa Tanaka,et al.  Temporal Decorrelation as Preprocessing for Linear and Post-nonlinear ICA , 2004, ICA.

[44]  J. Karhunen,et al.  Nonlinear Independent Factor Analysis by Hierarchical Models , 2003 .

[45]  Dinh Tuan Pham,et al.  BLIND SOURCE SEPARATION IN POST NONLINEAR MIXTURES , 2001 .

[46]  Motoaki Kawanabe,et al.  Kernel-Based Nonlinear Blind Source Separation , 2003, Neural Computation.

[47]  Christian Jutten,et al.  Source separation in post-nonlinear mixtures , 1999, IEEE Trans. Signal Process..

[48]  Gilles Burel,et al.  Blind separation of sources: A nonlinear neural algorithm , 1992, Neural Networks.

[49]  Christian Jutten,et al.  MINIMIZATION-PROJECTION (MP) APPROACH FOR BLIND SOURCE SEPARATION IN DIFFERENT MIXING MODELS , 2003 .

[50]  Antti Honkela,et al.  Variational learning and bits-back coding: an information-theoretic view to Bayesian learning , 2004, IEEE Transactions on Neural Networks.

[51]  C. S. Wallace,et al.  Classification by Minimum-Message-Length Inference , 1991, ICCI.

[52]  C. Jutten,et al.  QUADRATIC DEPENDENCE MEASURE FOR NONLINEAR BLIND SOURCES SEPARATION , 2003 .

[53]  Yujie Zhang,et al.  Linear and nonlinear ICA based on mutual information , 2007, 2007 International Symposium on Intelligent Signal Processing and Communication Systems.

[54]  Juha Karhunen,et al.  A Maximum Likelihood Approach to Nonlinear Blind Source Separation , 1997, ICANN.

[55]  Harri Valpola,et al.  Denoising Source Separation , 2005, J. Mach. Learn. Res..

[56]  S. Hochreiter,et al.  Lococode Performs Nonlinear ICA Without Knowing The Number Of Sources , 1999 .

[57]  Bernhard Schölkopf,et al.  Nonlinear Component Analysis as a Kernel Eigenvalue Problem , 1998, Neural Computation.

[58]  Ziyou Xiong,et al.  NONLINEAR INDEPENDENT COMPONENT ANALYSIS(ICA) USING POWER SERIES AND APPLICATION TO BLIND SOURCE SEPARATION , 2001 .

[59]  Jürgen Schmidhuber,et al.  LOCOCODE versus PCA and ICA , 1998 .

[60]  R Hecht-Nielsen,et al.  Replicator neural networks for universal optimal source coding. , 1995, Science.

[61]  Juha Karhunen,et al.  An Unsupervised Ensemble Learning Method for Nonlinear Dynamic State-Space Models , 2002, Neural Computation.

[62]  Christian Jutten,et al.  Quasi-nonparametric blind inversion of Wiener systems , 2001, IEEE Trans. Signal Process..

[63]  Christian Jutten,et al.  Entropy Optimization - Application to Blind Source Separation , 1997, ICANN.

[64]  T. Sejnowski,et al.  Separation of post-nonlinear mixtures using ACE and temporal decorrelation , 2001 .

[65]  Michael C. Mozer,et al.  An electric field approach to independent component analysis , 2000 .

[66]  C. Jutten,et al.  On the separability of nonlinear mixtures of temporally correlated sources , 2003, IEEE Signal Processing Letters.

[67]  Nikos A. Vlassis,et al.  Efficient source adaptivity in independent component analysis , 2001, IEEE Trans. Neural Networks.

[68]  Kanter,et al.  Eigenvalues of covariance matrices: Application to neural-network learning. , 1991, Physical review letters.

[69]  Lucas C. Parra,et al.  Statistical Independence and Novelty Detection with Information Preserving Nonlinear Maps , 1996, Neural Computation.

[70]  Aapo Hyvärinen,et al.  Fast and robust fixed-point algorithms for independent component analysis , 1999, IEEE Trans. Neural Networks.

[71]  G. Darmois,et al.  Analyse générale des liaisons stochastiques: etude particulière de l'analyse factorielle linéaire , 1953 .

[72]  J. Príncipe,et al.  Entropy manipulation of arbitrary nonlinear mappings , 1997, Neural Networks for Signal Processing VII. Proceedings of the 1997 IEEE Signal Processing Society Workshop.

[73]  Michael I. Jordan,et al.  Kernel independent component analysis , 2003 .

[74]  Dinh-Tuan Pham,et al.  Fast approximation of nonlinearities for improving inversion algorithms of PNL mixtures and Wiener systems , 2005, Signal Process..

[75]  Terrence J. Sejnowski,et al.  Nonlinear blind source separation using kernel feature spaces , 2001 .

[76]  Fabian J. Theis,et al.  Nonlinear Geometric ICA , 2003 .

[77]  Christian Jutten,et al.  Identifiability of post-nonlinear mixtures , 2005, IEEE Signal Processing Letters.

[78]  Antti Honkela,et al.  Unsupervised Variational Bayesian Learning of Nonlinear Models , 2004, NIPS.

[79]  Luís B. Almeida,et al.  MISEP -- Linear and Nonlinear ICA Based on Mutual Information , 2003, J. Mach. Learn. Res..

[80]  Petteri Pajunen,et al.  Nonlinear Blind Source Separation by Self-Organizing Maps , 1996 .

[81]  Christian Jutten,et al.  A minimization-projection (MP) approach for blind separating convolutive mixtures , 2004, 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[82]  Juha Karhunen,et al.  Advances in blind source separation (BSS) and independent component analysis (ICA) for nonlinear mixtures , 2004, Int. J. Neural Syst..

[83]  Pierre Comon,et al.  Independent component analysis, A new concept? , 1994, Signal Process..

[84]  Jürgen Schmidhuber,et al.  Learning Factorial Codes by Predictability Minimization , 1992, Neural Computation.