Nonlinear Component Analysis as a Kernel Eigenvalue Problem

A new method for performing a nonlinear form of principal component analysis is proposed. By the use of integral operator kernel functions, one can efficiently compute principal components in high-dimensional feature spaces, related to input space by some nonlinear mapfor instance, the space of all possible five-pixel products in 16 16 images. We give the derivation of the method and present experimental results on polynomial feature extraction for pattern recognition.

[1]  K. Karhunen.  Zur Spektraltheorie stochastischer prozesse , 1946 .

[2]  R. Courant,et al.  Methods of Mathematical Physics , 1962 .

[3]  R. Courant,et al.  Methods of Mathematical Physics, Vol. I , 1954 .

[4]  M. Aizerman,et al.  Theoretical Foundations of the Potential Function Method in Pattern Recognition Learning , 1964 .

[5]  R. Courant.  Methods of mathematical physics, Volume I , 1965 .

[6]  Shun-ichi Amari,et al.  A Theory of Pattern Recognition , 1968 .

[7]  N. S. Barnett,et al.  Private communication , 1969 .

[8]  G. Reuter.  LINEAR OPERATORS PART II (SPECTRAL THEORY) , 1969 .

[9]  W. N. Wapnik,et al.  Theorie der Zeichenerkennung , 1979 .

[10]  E. Oja.  Simplified neuron model as a principal component analyzer , 1982, Journal of mathematical biology.

[11]  J. Friedman.  Exploratory Projection Pursuit , 1987 .

[12]  Lawrence D. Jackel,et al.  Backpropagation Applied to Handwritten Zip Code Recognition , 1989, Neural Computation.

[13]  Lawrence Sirovich,et al.  Application of the Karhunen-Loeve Procedure for the Characterization of Human Faces , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[14]  Christian Jutten,et al.  Blind separation of sources, part I: An adaptive algorithm based on neuromimetic architecture , 1991, Signal Process..

[15]  M. Turk,et al.  Eigenfaces for Recognition , 1991, Journal of Cognitive Neuroscience.

[16]  Léon Bottou,et al.  Local Learning Algorithms , 1992, Neural Computation.

[17]  Bernhard E. Boser,et al.  A training algorithm for optimal margin classifiers , 1992, COLT '92.

[18]  Yann LeCun,et al.  Efficient Pattern Recognition Using a New Transformation Distance , 1992, NIPS.

[19]  Stephen M. Omohundro,et al.  Surface Learning with Applications to Lipreading , 1993, NIPS.

[20]  R. Tibshirani,et al.  Flexible Discriminant Analysis by Optimal Scoring , 1994 .

[21]  Terrence J. Sejnowski,et al.  An Information-Maximization Approach to Blind Separation and Blind Deconvolution , 1995, Neural Computation.

[22]  S. Klinke,et al.  Exploratory Projection Pursuit , 1995 .

[23]  Bernhard Schölkopf,et al.  Extracting Support Data for a Given Task , 1995, KDD.

[24]  Christopher J. C. Burges,et al.  Simplified Support Vector Decision Rules , 1996, ICML.

[25]  Sun-Yuan Kung,et al.  Principal Component Neural Networks: Theory and Applications , 1996 .

[26]  Bernhard Schölkopf,et al.  Incorporating Invariances in Support Vector Learning Machines , 1996, ICANN.

[27]  Bernhard Schölkopf,et al.  Comparison of View-Based Object Recognition Algorithms Using Realistic 3D Models , 1996, ICANN.

[28]  Juyang Weng,et al.  Using Discriminant Eigenfeatures for Image Retrieval , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[29]  Tomaso A. Poggio,et al.  Image Synthesis from a Single Example Image , 1996, ECCV.

[30]  Bernhard Schölkopf,et al.  Improving the accuracy and speed of support vector learning machines , 1997, NIPS 1997.

[31]  Joachim M. Buhmann,et al.  Data clustering and learning , 1998 .

[32]  Marti A. Hearst.  Trends & Controversies: Support Vector Machines , 1998, IEEE Intell. Syst..

[33]  George W. Irwin,et al.  RBF principal manifolds for process monitoring , 1999, IEEE Trans. Neural Networks.

[34]  Vladimir Vapnik,et al.  An overview of statistical learning theory , 1999, IEEE Trans. Neural Networks.

[35]  Gunnar Rätsch,et al.  Input space versus feature space in kernel-based methods , 1999, IEEE Trans. Neural Networks.

[36]  E. Bagarinao,et al.  Reconstructing bifurcation diagrams from noisy time series using nonlinear autoregressive models. , 1999, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[37]  Anil K. Jain,et al.  Statistical Pattern Recognition: A Review , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[38]  Andreas Ziehe,et al.  Artifact Reduction in Magnetoneurography Based on Time-Delayed Second Order Correlations , 1998 .

[39]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[40]  Martin Brown,et al.  Linear spectral mixture models and support vector machines for remote sensing , 2000, IEEE Trans. Geosci. Remote. Sens..