Learning principal directions: Integrated-squared-error minimization

A common derivation of principal component analysis (PCA) is based on the minimization of the squared-error between centered data and linear model, corresponding to the reconstruction error. In fact, minimizing the squared-error leads to principal subspace analysis where scaled and rotated principal axes of a set of observed data, are estimated. In this paper, we introduce and investigate an alternative error measure, integrated-squared error (ISE), the minimization of which determines the exact principal axes (without rotational ambiguity) of a set of observed data. We show that exact principal directions emerge from the minimization of ISE. We present a simple EM algorithm, 'EM-ePCA', which is similar to EM-PCA [S.T. Roweis, EM algorithms for PCA and SPCA, in: Advances in Neural Information Processing Systems, vol. 10, MIT Press, Cambridge, 1998, pp. 626-632.], but finds exact principal directions without rotational ambiguity. In addition, we revisit the generalized Hebbian algorithm (GHA) and show that it emerges from the ISE minimization in a single-layer linear feedforward neural network.

[1]  Juha Karhunen,et al.  Principal component neural networks — Theory and applications , 1998, Pattern Analysis and Applications.

[2]  Heng Tao Shen,et al.  Principal Component Analysis , 2009, Encyclopedia of Biometrics.

[3]  Erkki Oja,et al.  Neural Networks, Principal Components, and Subspaces , 1989, Int. J. Neural Syst..

[4]  Juha Karhunen,et al.  Generalizations of principal component analysis, optimization problems, and neural networks , 1995, Neural Networks.

[5]  Sam T. Roweis,et al.  EM Algorithms for PCA and SPCA , 1997, NIPS.

[6]  Jong-Hoon Ahn,et al.  A Constrained EM Algorithm for Principal Component Analysis , 2003, Neural Computation.

[7]  Seungjin Choi,et al.  Sequential EM learning for subspace analysis , 2002, Pattern Recognit. Lett..

[8]  Terence D. Sanger,et al.  Optimal unsupervised learning in a single-layer linear feedforward neural network , 1989, Neural Networks.

[9]  Michael E. Tipping,et al.  Probabilistic Principal Component Analysis , 1999 .

[10]  Christopher M. Bishop,et al.  Mixtures of Probabilistic Principal Component Analyzers , 1999, Neural Computation.

[11]  Jong-Hoon Ahn,et al.  A new way of PCA: integrated-squared-error and EM algorithms , 2004, 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[12]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[13]  D. Signorini,et al.  Neural networks , 1995, The Lancet.