Gradient based block coordinate descent algorithms for joint approximate diagonalization of matrices

In this paper, we propose a gradient based block coordinate descent (BCD-G) framework to solve the joint approximate diagonalization of matrices defined on the product of the complex Stiefel manifold and the special linear group. Instead of the cyclic fashion, we choose the block for optimization in a way based on the Riemannian gradient. To update the first block variable in the complex Stiefel manifold, we use the well-known line search descent method. To update the second block variable in the special linear group, based on four different kinds of elementary rotations, we construct two classes: Jacobi-GLU and Jacobi-GLQ, and then get two BCD-G algorithms: BCD-GLU and BCD-GLQ. We establish the weak convergence and global convergence of these two algorithms using the Łojasiewicz gradient inequality under the assumption that the iterates are bounded. In particular, the problem we focus on in this paper includes as special cases the well-known joint approximate diagonalization of Hermitian (or complex symmetric) matrices by invertible transformations in blind source separation, and our algorithms specialize as the gradient based Jacobi-type algorithms. All the algorithms and convergence results in this paper also apply to the real case.

[1]  Pierre Comon,et al.  Jacobi-type algorithm for low rank orthogonal approximation of symmetric tensors and its convergence analysis , 2019, ArXiv.

[2]  Reinhold Schneider,et al.  Convergence Results for Projected Line-Search Methods on Varieties of Low-Rank Matrices Via Łojasiewicz Inequality , 2014, SIAM J. Optim..

[3]  Shuzhong Zhang,et al.  Maximum Block Improvement and Polynomial Optimization , 2012, SIAM J. Optim..

[4]  Robert E. Mahony,et al.  Convergence of the Iterates of Descent Methods for Analytic Cost Functions , 2005, SIAM J. Optim..

[5]  Paul Van Dooren,et al.  Jacobi Algorithm for the Best Low Multilinear Rank Approximation of Symmetric Tensors , 2013, SIAM J. Matrix Anal. Appl..

[6]  Pierre Comon,et al.  Independent component analysis, A new concept? , 1994, Signal Process..

[7]  K. Schittkowski,et al.  NONLINEAR PROGRAMMING , 2022 .

[8]  Jonathan H. Manton.  Modified steepest descent and Newton algorithms for orthogonally constrained optimisation. Part II. The complex Grassmann manifold , 2001, Proceedings of the Sixth International Symposium on Signal Processing and its Applications (Cat.No.01EX467).

[9]  Hiroyuki Sato,et al.  A new, globally convergent Riemannian conjugate gradient method , 2013, 1302.0125.

[10]  A. Uschmajew,et al.  A new convergence proof for the higher-order power method and generalizations , 2014, 1407.4586.

[11]  Arie Yeredor,et al.  Joint Matrices Decompositions and Blind Source Separation: A survey of methods, identification, and applications , 2014, IEEE Signal Processing Magazine.

[12]  Antoine Souloumiac,et al.  Jacobi Angles for Simultaneous Diagonalization , 1996, SIAM J. Matrix Anal. Appl..

[13]  Bijan Afsari,et al.  Simple LU and QR Based Non-orthogonal Matrix Joint Diagonalization , 2006, ICA.

[14]  J. Cardoso,et al.  Blind beamforming for non-gaussian signals , 1993 .

[15]  Pierre Comon,et al.  Approximate tensor diagonalization by invertible transforms , 2009, 2009 17th European Signal Processing Conference.

[16]  B. Afsari.  Gradient Flow Based Matrix Joint Diagonalization for Independent Component Analysis , 2004 .

[17]  Eric Moreau,et al.  Jacobi like algorithm for non-orthogonal joint diagonalization of hermitian matrices , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[18]  S. Łojasiewicz.  Sur la géométrie semi- et sous- analytique , 1993 .

[19]  P. Tseng,et al.  On the convergence of the coordinate descent method for convex differentiable minimization , 1992 .

[20]  Z.-Q. Luo,et al.  Error bounds and convergence analysis of feasible descent methods: a general approach , 1993, Ann. Oper. Res..

[21]  P. Comon,et al.  On approximate diagonalization of third order symmetric tensors by orthogonal transformations , 2018, Linear Algebra and its Applications.

[22]  R. Carter.  Lie Groups , 1970, Nature.

[23]  Jonathan H. Manton,et al.  Modified steepest descent and Newton algorithms for orthogonally constrained optimisation. Part I. The complex Stiefel manifold , 2001, Proceedings of the Sixth International Symposium on Signal Processing and its Applications (Cat.No.01EX467).

[24]  Ke Wang,et al.  Complex Non-Orthogonal Joint Diagonalization Based on LU and LQ Decompositions , 2012, LVA/ICA.

[25]  Levent Tunçel,et al.  Optimization algorithms on matrix manifolds , 2009, Math. Comput..

[26]  Pierre Comon,et al.  Approximate Matrix and Tensor Diagonalization by Unitary Transformations: Convergence of Jacobi-Type Algorithms , 2019, SIAM J. Optim..

[27]  Stephen J. Wright.  Coordinate descent algorithms , 2015, Mathematical Programming.

[28]  Eric Moreau,et al.  Fast Jacobi like algorithms for joint diagonalization of complex symmetric matrices , 2013, 21st European Signal Processing Conference (EUSIPCO 2013).

[29]  Benedikt Wirth,et al.  Optimization Methods on Riemannian Manifolds and Their Application to Shape Space , 2012, SIAM J. Optim..

[30]  Visa Koivunen,et al.  Steepest Descent Algorithms for Optimization Under Unitary Matrix Constraint , 2008, IEEE Transactions on Signal Processing.

[31]  A. Baker.  Matrix Groups: An Introduction to Lie Group Theory , 2003 .

[32]  Pierre Comon,et al.  Globally convergent Jacobi-type algorithms for simultaneous orthogonal symmetric tensor diagonalization , 2017, SIAM J. Matrix Anal. Appl..

[33]  Seungjin Choi,et al.  Independent Component Analysis , 2009, Handbook of Natural Computing.

[34]  Marco Congedo,et al.  Approximate Joint Diagonalization with Riemannian Optimization on the General Linear Group , 2020, SIAM J. Matrix Anal. Appl..

[35]  B. A. D. H. Brandwood.  A complex gradient operator and its applica-tion in adaptive array theory , 1983 .

[36]  Wotao Yin,et al.  A Block Coordinate Descent Method for Regularized Multiconvex Optimization with Applications to Nonnegative Tensor Factorization and Completion , 2013, SIAM J. Imaging Sci..

[37]  B. Hall.  Lie Groups, Lie Algebras, and Representations: An Elementary Introduction , 2004 .

[38]  Pierre Comon,et al.  On the convergence of Jacobi-type algorithms for Independent Component Analysis , 2019, 2020 IEEE 11th Sensor Array and Multichannel Signal Processing Workshop (SAM).

[39]  André Uschmajew,et al.  On Convergence of the Maximum Block Improvement Method , 2015, SIAM J. Optim..

[40]  Eric Moreau,et al.  A Decoupled Jacobi-Like Algorithm for Non-Unitary Joint Diagonalization of Complex-Valued Matrices , 2014, IEEE Signal Processing Letters.

[41]  N. Levenberg,et al.  Function theory in several complex variables , 2001 .

[42]  Harold R. Parks,et al.  A Primer of Real Analytic Functions , 1992 .

[43]  Robert H. Halstead,et al.  Matrix Computations , 2011, Encyclopedia of Parallel Computing.

[44]  Antoine Souloumiac,et al.  Nonorthogonal Joint Diagonalization by Combining Givens and Hyperbolic Rotations , 2009, IEEE Transactions on Signal Processing.