A class of neural networks for independent component analysis

Independent component analysis (ICA) is a recently developed, useful extension of standard principal component analysis (PCA). The ICA model is utilized mainly in blind separation of unknown source signals from their linear mixtures. In this application only the source signals which correspond to the coefficients of the ICA expansion are of interest. In this paper, we propose neural structures related to multilayer feedforward networks for performing complete ICA. The basic ICA network consists of whitening, separation, and basis vector estimation layers. It can be used for both blind source separation and estimation of the basis vectors of ICA. We consider learning algorithms for each layer, and modify our previous nonlinear PCA type algorithms so that their separation capabilities are greatly improved. The proposed class of networks yields good results in test examples with both artificial and real-world data.

[1]  E. Oja.  Simplified neuron model as a principal component analyzer , 1982, Journal of mathematical biology.

[2]  E. Oja,et al.  On stochastic approximation of the eigenvectors and eigenvalues of the expectation of a random matrix , 1985 .

[3]  Christian Jutten,et al.  Detection de grandeurs primitives dans un message composite par une architecture de calcul neuromime , 1985 .

[4]  Robin Sibson,et al.  What is projection pursuit , 1987 .

[5]  C. Jutten.  Calcul neuromimétique et traitement du signal : analyse en composantes indépendantes , 1987 .

[6]  Christian Jutten,et al.  Independent components analysis versus principal components analysis , 1988 .

[7]  P. Comon.  Separation Of Stochastic Processes , 1989, Workshop on Higher-Order Spectral Analysis.

[8]  Kurt Hornik,et al.  Neural networks and principal component analysis: Learning from examples without local minima , 1989, Neural Networks.

[9]  Christian Jutten,et al.  Blind separation of sources, part I: An adaptive algorithm based on neuromimetic architecture , 1991, Signal Process..

[10]  John C. Platt,et al.  Networks for the Separation of Sources that Are Superimposed and Delayed , 1991, NIPS.

[11]  Charles W. Therrien,et al.  Discrete Random Signals and Statistical Signal Processing , 1992 .

[12]  Jukka Saarinen,et al.  Hardware Implementations of PCA Neural Networks , 1992 .

[13]  Gilles Burel,et al.  Blind separation of sources: A nonlinear neural algorithm , 1992, Neural Networks.

[14]  Erkki Oja,et al.  Principal components, minor components, and linear neural networks , 1992, Neural Networks.

[15]  Mark D. Plumbley.  A Hebbian/anti-Hebbian network which optimizes information capacity by orthonormalizing the principal subspace , 1993 .

[16]  Gilles Burel,et al.  Neural networks for array processing: from DOA estimation to blind separation of sources , 1993, Proceedings of IEEE Systems Man and Cybernetics Conference - SMC.

[17]  Andrzej Cichocki,et al.  Neural networks for optimization and signal processing , 1993 .

[18]  Mark D. Plumbley.  Efficient information transfer and anti-Hebbian neural networks , 1993, Neural Networks.

[19]  Mark D. Plumbley,et al.  Information Theory and Neural Networks , 1993 .

[20]  Eric Moreau,et al.  New self-adaptative algorithms for source separation based on contrast functions , 1993, [1993 Proceedings] IEEE Signal Processing Workshop on Higher-Order Statistics.

[21]  S. Hyakin,et al.  Neural Networks: A Comprehensive Foundation , 1994 .

[22]  J. Karhunen.  Optimization criteria and nonlinear PCA neural networks , 1994, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94).

[23]  Jean-Francois Cardoso,et al.  Adaptive Source Separation With Uniform Performance , 1994 .

[24]  Juha Karhunen,et al.  Representation and separation of signals using nonlinear PCA type learning , 1994, Neural Networks.

[25]  Pierre Comon,et al.  Independent component analysis, A new concept? , 1994, Signal Process..

[26]  Juha Karhunen,et al.  Stability of Oja's PCA Subspace Rule , 1994, Neural Computation.

[27]  E. Oja.  The Nonlinear PCA Learning Rule and Signal Separation - Mathematical Analysis , 1995 .

[28]  Terrence J. Sejnowski,et al.  An Information-Maximization Approach to Blind Separation and Blind Deconvolution , 1995, Neural Computation.

[29]  S. Klinke,et al.  Exploratory Projection Pursuit , 1995 .

[30]  Kurt Hornik,et al.  Learning in linear neural networks: a survey , 1995, IEEE Trans. Neural Networks.

[31]  Juha Karhunen,et al.  Generalizations of principal component analysis, optimization problems, and neural networks , 1995, Neural Networks.

[32]  Kiyotoshi Matsuoka,et al.  A neural net for blind separation of nonstationary signals , 1995, Neural Networks.

[33]  J. Karhunen,et al.  A bigradient optimization approach for robust PCA, MCA, and source separation , 1995, Proceedings of ICNN'95 - International Conference on Neural Networks.

[34]  Andrzej Cichocki,et al.  A New Learning Algorithm for Blind Signal Separation , 1995, NIPS.

[35]  G. Deco,et al.  An Information-Theoretic Approach to Neural Computing , 1997, Perspectives in Neural Computing.

[36]  Kari Torkkola,et al.  Blind separation of convolved sources based on information maximization , 1996, Neural Networks for Signal Processing VI. Proceedings of the 1996 IEEE Signal Processing Society Workshop.

[37]  Sun-Yuan Kung,et al.  Principal Component Neural Networks: Theory and Applications , 1996 .

[38]  Terrence J. Sejnowski,et al.  Edges are the Independent Components of Natural Scenes , 1996, NIPS.

[39]  Erkki Oja,et al.  Image Feature Extraction Using Independent Component Analysis , 1996 .

[40]  Juha Karhunen,et al.  A Unified Neural Bigradient Algorithm for robust PCA and MCA , 1996, Int. J. Neural Syst..

[41]  Jean-François Cardoso,et al.  Equivariant adaptive source separation , 1996, IEEE Trans. Signal Process..

[42]  Juha Karhunen,et al.  Neural approaches to independent component analysis and source separation , 1996, ESANN.

[43]  T J Sejnowski,et al.  Learning the higher-order structure of a natural sound. , 1996, Network.

[44]  Erkki Oja,et al.  One-unit Learning Rules for Independent Component Analysis , 1996, NIPS.

[45]  Aapo Hyvärinen,et al.  A family of fixed-point algorithms for independent component analysis , 1997, ICASSP.

[46]  Juha Karhunen,et al.  Blind source separation using least-squares type adaptive algorithms , 1997, 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[47]  Erkki Oja,et al.  Applications of neural blind separation to signal and image processing , 1997, 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[48]  Terrence J. Sejnowski,et al.  The “independent components” of natural scenes are edge filters , 1997, Vision Research.