Constrained Projection Approximation Algorithms for Principal Component Analysis

In this paper, we introduce a new error measure, integrated reconstruction error (IRE) and show that the minimization of IRE leads to principal eigenvectors (without rotational ambiguity) of the data covariance matrix. Then, we present iterative algorithms for the IRE minimization, where we use the projection approximation. The proposed algorithm is referred to as COnstrained Projection Approximation (COPA) algorithm and its limiting case is called COPAL. Numerical experiments demonstrate that these algorithms successfully find exact principal eigenvectors of the data covariance matrix.

[1]  Andrzej Cichocki,et al.  Neural networks for computing eigenvalues and eigenvectors , 1992, Biological Cybernetics.

[2]  R. Brockett,et al.  Dynamical systems that sort lists, diagonalize matrices and solve linear programming problems , 1988, Proceedings of the 27th IEEE Conference on Decision and Control.

[3]  Bin Yang,et al.  Projection approximation subspace tracking , 1995, IEEE Trans. Signal Process..

[4]  I. Jolliffe.  Principal Component Analysis , 2002 .

[5]  Juha Karhunen,et al.  Principal component neural networks — Theory and applications , 1998, Pattern Analysis and Applications.

[6]  Lei Xu,et al.  Least mean square error reconstruction principle for self-organizing neural-nets , 1993, Neural Networks.

[7]  Jong-Hoon Ahn,et al.  A Constrained EM Algorithm for Principal Component Analysis , 2003, Neural Computation.

[8]  R. Brockett.  Dynamical systems that sort lists, diagonalize matrices, and solve linear programming problems , 1991 .

[9]  Terence D. Sanger,et al.  Optimal unsupervised learning in a single-layer linear feedforward neural network , 1989, Neural Networks.

[10]  Jong-Hoon Ahn,et al.  A new way of PCA: integrated-squared-error and EM algorithms , 2004, 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[11]  Sam T. Roweis,et al.  EM Algorithms for PCA and SPCA , 1997, NIPS.

[12]  Kurt Hornik,et al.  Neural networks and principal component analysis: Learning from examples without local minima , 1989, Neural Networks.

[13]  Seungjin Choi.  On Variations of Power Iteration , 2005, ICANN.

[14]  R. Cox,et al.  Journal of the Royal Statistical Society B , 1972 .

[15]  Karim Abed-Meraim,et al.  A New Look at the Power Method for Fast Subspace Tracking , 1999, Digit. Signal Process..

[16]  Erkki Oja,et al.  Neural Networks, Principal Components, and Subspaces , 1989, Int. J. Neural Syst..

[17]  Michael E. Tipping,et al.  Probabilistic Principal Component Analysis , 1999 .

[18]  Y. Hua,et al.  Fast orthonormal PAST algorithm , 2000, IEEE Signal Processing Letters.