A Fast Algorithm for Joint Diagonalization with Non-orthogonal Transformations and its Application to Blind Source Separation

A new efficient algorithm is presented for joint diagonalization of several matrices. The algorithm is based on the Frobenius-norm formulation of the joint diagonalization problem, and addresses diagonalization with a general, non-orthogonal transformation. The iterative scheme of the algorithm is based on a multiplicative update which ensures the invertibility of the diagonalizer. The algorithm's efficiency stems from the special approximation of the cost function resulting in a sparse, block-diagonal Hessian to be used in the computation of the quasi-Newton update step. Extensive numerical simulations illustrate the performance of the algorithm and provide a comparison to other leading diagonalization methods. The results of such comparison demonstrate that the proposed algorithm is a viable alternative to existing state-of-the-art joint diagonalization algorithms. The practical use of our algorithm is shown for blind source separation problems.

[1]  W. Gautschi,et al.  An algorithm for simultaneous orthogonal transformation of several positive definite symmetric matrices to nearly diagonal form , 1986 .

[2]  Arogyaswami Paulraj,et al.  Joint angle and delay estimation using shift-invariance techniques , 1998, IEEE Trans. Signal Process..

[3]  Tony Springall.  Common Principal Components and Related Multivariate Models , 1991 .

[4]  Dinh-Tuan Pham,et al.  Blind separation of instantaneous mixtures of nonstationary sources , 2001, IEEE Trans. Signal Process..

[5]  Charles R. Johnson,et al.  Matrix analysis , 1985, Statistical Inference for Engineers and Data Scientists.

[6]  Erkki Oja,et al.  Independent Component Analysis , 2001 .

[7]  J. J. Moré,et al.  Levenberg--Marquardt algorithm: implementation and theory , 1977 .

[8]  S. Haykin.  Unsupervised adaptive filtering, vol. 1: Blind source separation , 2000 .

[9]  Kiyotoshi Matsuoka,et al.  A neural net for blind separation of nonstationary signals , 1995, Neural Networks.

[10]  L. Parra,et al.  Convolutive blind source separation based on multiple decorrelation , 1998, Neural Networks for Signal Processing VIII. Proceedings of the 1998 IEEE Signal Processing Society Workshop (Cat. No.98TH8378).

[11]  Lang Tong,et al.  Indeterminacy and identifiability of blind identification , 1991 .

[12]  M. Joho,et al.  Joint diagonalization of correlation matrices by using Newton methods with application to blind signal separation , 2002, Sensor Array and Multichannel Signal Processing Workshop Proceedings, 2002.

[13]  M. R. Osborne.  Nonlinear least squares — the Levenberg algorithm revisited , 1976, The Journal of the Australian Mathematical Society. Series B. Applied Mathematics.

[14]  Kenneth Levenberg.  A METHOD FOR THE SOLUTION OF CERTAIN NON – LINEAR PROBLEMS IN LEAST SQUARES , 1944 .

[15]  Eric Moulines,et al.  A blind source separation technique using second-order statistics , 1997, IEEE Trans. Signal Process..

[16]  J. Cardoso.  On the Performance of Orthogonal Source Separation Algorithms , 1994 .

[17]  Ch. von der Malsburg,et al.  A neural cocktail-party processor , 1986, Biological Cybernetics.

[18]  Andreas Ziehe,et al.  Artifact Reduction in Magnetoneurography Based on Time-Delayed Second Order Correlations , 1998 .

[19]  Andrzej Cichocki,et al.  Nonholonomic Orthogonal Learning Algorithms for Blind Source Separation , 2000, Neural Computation.

[20]  Christian Jutten,et al.  Blind separation of sources, part I: An adaptive algorithm based on neuromimetic architecture , 1991, Signal Process..

[21]  Pierre Comon.  Independent component analysis - a new concept? signal processing , 1994 .

[22]  A.-J. van der Veen.  Joint diagonalization via subspace fitting techniques , 2001, 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.01CH37221).

[23]  Antoine Souloumiac,et al.  Jacobi Angles for Simultaneous Diagonalization , 1996, SIAM J. Matrix Anal. Appl..

[24]  J. Cardoso,et al.  Blind beamforming for non-gaussian signals , 1993 .

[25]  Andreas Ziehe,et al.  Combining time-delayed decorrelation and ICA: towards solving the cocktail party problem , 1998, Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP '98 (Cat. No.98CH36181).

[26]  B. Flury,et al.  An application of common principal component analysis to cranial morphometry of Microtus californicus and M. ochrogaster(Mammalia, Rodentia) , 1988 .

[27]  I. Duff,et al.  The state of the art in numerical analysis , 1997 .

[28]  H. Luetkepohl.  The Handbook of Matrices , 1996 .

[29]  Pierre Comon,et al.  Independent component analysis, A new concept? , 1994, Signal Process..

[30]  Seungjin Choi,et al.  Blind separation of second-order nonstationary and temporally colored sources , 2001, Proceedings of the 11th IEEE Signal Processing Workshop on Statistical Signal Processing (Cat. No.01TH8563).

[31]  Jean-François Cardoso,et al.  Equivariant adaptive source separation , 1996, IEEE Trans. Signal Process..

[32]  G. Golub,et al.  150 years old and still alive: eigenproblems , 1997 .

[33]  Eric Moreau,et al.  A generalization of joint-diagonalization criteria for source separation , 2001, IEEE Trans. Signal Process..

[34]  Motoaki Kawanabe,et al.  Kernel-Based Nonlinear Blind Source Separation , 2003, Neural Computation.

[35]  A. Ziehe,et al.  A LINEAR LEAST-SQUARES ALGORITHM FOR JOINT DIAGONALIZATION , 2003 .

[36]  Heinz Mathis,et al.  Joint diagonalization of correlation matrices by using gradient methods with application to blind signal separation , 2002, Sensor Array and Multichannel Signal Processing Workshop Proceedings, 2002.

[37]  Noboru Murata,et al.  Multiplicative Nonholonomic/Newton -like Algorithm , 2000, ArXiv.

[38]  Arie Yeredor,et al.  Non-orthogonal joint diagonalization in the least-squares sense with application in blind source separation , 2002, IEEE Trans. Signal Process..

[39]  A. Goldstein.  On Newton's method , 1965 .

[40]  Arie Yeredor,et al.  Blind source separation via the second characteristic function , 2000, Signal Process..

[41]  Schuster,et al.  Separation of a mixture of independent signals using time delayed correlations. , 1994, Physical review letters.

[42]  Tzyy-Ping Jung,et al.  Independent Component Analysis of Electroencephalographic Data , 1995, NIPS.

[43]  Andreas Ziehe,et al.  TDSEP — an efficient algorithm for blind separation using time structure , 1998 .

[44]  Wolfgang K. Härdle,et al.  The Dynamics of Implied Volatilities: A Common Principal Components Approach , 2003 .

[45]  Andreas Ziehe,et al.  An approach to blind source separation based on temporal structure of speech signals , 2001, Neurocomputing.

[46]  Andreas Ziehe,et al.  Independent component analysis of noninvasively recorded cortical magnetic DC-fields in humans , 2000, IEEE Transactions on Biomedical Engineering.

[47]  Jose C. Principe,et al.  Simultaneous Diagonalization in the Frequency Domain (SDIF) for Source Separation , 2000 .

[48]  Erkki Oja,et al.  Independent Component Analysis for Identification of Artifacts in Magnetoencephalographic Recordings , 1997, NIPS.

[49]  A. Bunse-Gerstner,et al.  Numerical Methods for Simultaneous Diagonalization , 1993, SIAM J. Matrix Anal. Appl..

[50]  Shun-ichi Amari,et al.  Adaptive blind signal processing-neural network approaches , 1998, Proc. IEEE.

[51]  Ed F. Deprettere,et al.  Azimuth and elevation computation in high resolution DOA estimation , 1992, IEEE Trans. Signal Process..

[52]  D. Marquardt.  An Algorithm for Least-Squares Estimation of Nonlinear Parameters , 1963 .

[53]  Jean-Franois Cardoso.  High-Order Contrasts for Independent Component Analysis , 1999, Neural Computation.

[54]  Dinh Tuan Pham,et al.  Joint Approximate Diagonalization of Positive Definite Hermitian Matrices , 2000, SIAM J. Matrix Anal. Appl..