Iterative Projection Approximation Algorithms for PCA

In this paper we introduce a new error measure, integrated reconstruction error (IRE), the minimization of which leads to principal eigenvectors (without rotational ambiguity) of the data covariance matrix. Then we present iterative algorithms for the IRE minimization, through the projection approximation. The proposed algorithm is referred to as constrained projection approximation (COPA) algorithm and its limiting case is called COPAL. We also discuss regularized algorithms, referred to as R-COPA and R-COPAL. Numerical experiments demonstrate that these algorithms successfully find exact principal eigenvectors of the data covariance matrix

[1]  Andrzej Cichocki,et al.  Neural networks for computing eigenvalues and eigenvectors , 1992, Biological Cybernetics.

[2]  Seungjin Choi.  On Variations of Power Iteration , 2005, ICANN.

[3]  Karim Abed-Meraim,et al.  A New Look at the Power Method for Fast Subspace Tracking , 1999, Digit. Signal Process..

[4]  Bin Yang,et al.  Projection approximation subspace tracking , 1995, IEEE Trans. Signal Process..

[5]  Sun-Yuan Kung,et al.  Principal Component Neural Networks: Theory and Applications , 1996 .

[6]  Michael E. Tipping,et al.  Probabilistic Principal Component Analysis , 1999 .

[7]  Erkki Oja,et al.  Neural Networks, Principal Components, and Subspaces , 1989, Int. J. Neural Syst..

[8]  Y. Hua,et al.  Fast orthonormal PAST algorithm , 2000, IEEE Signal Processing Letters.

[9]  Sam T. Roweis,et al.  EM Algorithms for PCA and SPCA , 1997, NIPS.

[10]  Jong-Hoon Ahn,et al.  A new way of PCA: integrated-squared-error and EM algorithms , 2004, 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[11]  Terence D. Sanger,et al.  Optimal unsupervised learning in a single-layer linear feedforward neural network , 1989, Neural Networks.

[12]  R. Cox,et al.  Journal of the Royal Statistical Society B , 1972 .

[13]  Jong-Hoon Ahn,et al.  A Constrained EM Algorithm for Principal Component Analysis , 2003, Neural Computation.

[14]  R. Brockett.  Dynamical systems that sort lists, diagonalize matrices, and solve linear programming problems , 1991 .

[15]  Kurt Hornik,et al.  Neural networks and principal component analysis: Learning from examples without local minima , 1989, Neural Networks.