Iterative kernel principal component analysis for image modeling

In recent years, kernel principal component analysis (KPCA) has been suggested for various image processing tasks requiring an image model such as, e.g., denoising or compression. The original form of KPCA, however, can be only applied to strongly restricted image classes due to the limited number of training examples that can be processed. We therefore propose a new iterative method for performing KPCA, the kernel Hebbian algorithm, which iteratively estimates the kernel principal components with only linear order memory complexity. In our experiments, we compute models for complex image classes such as faces and natural images which require a large number of training examples. The resulting image models are tested in single-frame super-resolution and denoising applications. The KPCA model is not specifically tailored to these tasks; in fact, the same model can be used in super-resolution with variable input resolution, or denoising with unknown noise characteristics, in spite of this, both super-resolution and denoising performance are comparable to existing methods.

[1]  O. Chapelle,et al.  Bounds on error expectation for SVM , 2000 .

[2]  Ingo Steinwart,et al.  On the Influence of the Kernel on the Consistency of Support Vector Machines , 2002, J. Mach. Learn. Res..

[3]  Gunnar Rätsch,et al.  Kernel PCA and De-Noising in Feature Spaces , 1998, NIPS.

[4]  Christopher J. C. Burges,et al.  Simplified Support Vector Decision Rules , 1996, ICML.

[5]  Erkki Oja,et al.  Image Feature Extraction Using Independent Component Analysis , 1996 .

[6]  Wei-Yong Yan,et al.  Global convergence of Oja's subspace algorithm for principal component extraction , 1998, IEEE Trans. Neural Networks.

[7]  Aleksandra Pizurica,et al.  Estimating the probability of the presence of a signal of interest in multiresolution single- and multiband image denoising , 2006, IEEE Transactions on Image Processing.

[8]  David Mumford,et al.  Occlusion Models for Natural Images: A Statistical Study of a Scale-Invariant Dead Leaves Model , 2004, International Journal of Computer Vision.

[9]  David J. Field,et al.  Emergence of simple-cell receptive field properties by learning a sparse code for natural images , 1996, Nature.

[10]  David J. Field,et al.  What Is the Goal of Sensory Coding? , 1994, Neural Computation.

[11]  D. Munson.  A note on Lena , 1996 .

[12]  Terence D. Sanger,et al.  Optimal unsupervised learning in a single-layer linear feedforward neural network , 1989, Neural Networks.

[13]  Song-Chun Zhu,et al.  Prior Learning and Gibbs Reaction-Diffusion , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[14]  Tomaso A. Poggio,et al.  Extensions of a Theory of Networks for Approximation and Learning , 1990, NIPS.

[15]  William T. Freeman,et al.  Example-Based Super-Resolution , 2002, IEEE Computer Graphics and Applications.

[16]  Uwe Helmke,et al.  Convergence analysis for principal component flows , 2001 .

[17]  Erkki Oja,et al.  Principal components, minor components, and linear neural networks , 1992, Neural Networks.

[18]  B. Scholkopf,et al.  Fisher discriminant analysis with kernels , 1999, Neural Networks for Signal Processing IX: Proceedings of the 1999 IEEE Signal Processing Society Workshop (Cat. No.98TH8468).

[19]  Ivor W. Tsang,et al.  The pre-image problem in kernel methods , 2003, IEEE Transactions on Neural Networks.

[20]  Christopher J. Taylor,et al.  Kernel Principal Component Analysis and the construction of non-linear Active Shape Models , 2001, BMVC.

[21]  Bernhard Schölkopf,et al.  Learning with kernels , 2001 .

[22]  Bernhard Schölkopf,et al.  Nonlinear Component Analysis as a Kernel Eigenvalue Problem , 1998, Neural Computation.

[23]  Leslie S. Smith,et al.  The principal components of natural images , 1992 .

[24]  Federico Girosi,et al.  An Equivalence Between Sparse Approximation and Support Vector Machines , 1998, Neural Computation.

[25]  David J. Kriegman,et al.  From Few to Many: Illumination Cone Models for Face Recognition under Variable Lighting and Pose , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[26]  Ivor W. Tsang,et al.  Finding the pre-images in kernel principal component analysis , 2002 .

[27]  I. Omiaj,et al.  Extensions of a Theory of Networks for Approximation and Learning : dimensionality reduction and clustering , 2022 .

[28]  Bernhard Schölkopf,et al.  Bounds on Error Expectation for SVM , 2000 .

[29]  Daniel L. Ruderman,et al.  Origins of scaling in natural images , 1996, Vision Research.

[30]  E. Oja.  Simplified neuron model as a principal component analyzer , 1982, Journal of mathematical biology.

[31]  Bernhard Schölkopf,et al.  A kernel view of the dimensionality reduction of manifolds , 2004, ICML.

[32]  David Salesin,et al.  Image Analogies , 2001, SIGGRAPH.

[33]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[34]  B. Scholkopf,et al.  Implicit estimation of wiener series , 2004, Proceedings of the 2004 14th IEEE Signal Processing Society Workshop Machine Learning for Signal Processing, 2004..

[35]  Anuj Srivastava,et al.  Probability Models for Clutter in Natural Images , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[36]  P. P. Vaidyanathan,et al.  Role of principal component filter banks in noise reduction , 1999, Optics & Photonics.

[37]  Eero P. Simoncelli,et al.  Image compression via joint statistical characterization in the wavelet domain , 1999, IEEE Trans. Image Process..

[38]  Takeo Kanade,et al.  Limits on super-resolution and how to break them , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[39]  William T. Freeman,et al.  Learning Low-Level Vision , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[40]  Shaogang Gong,et al.  A Multi-View Nonlinear Active Shape Model Using Kernel PCA , 1999, BMVC.

[41]  Eero P. Simoncelli.  Bayesian Denoising of Visual Images in the Wavelet Domain , 1999 .

[42]  Terrence J. Sejnowski,et al.  The “independent components” of natural scenes are edge filters , 1997, Vision Research.

[43]  B. Schölkopf,et al.  Kernel Hebbian Algorithm for Iterative Kernel Principal Component Analysis , 2003 .

[44]  James T. Kwok,et al.  Eigenvoice Speaker Adaptation via Composite Kernel Principal Component Analysis , 2003, NIPS 2003.

[45]  Bernhard Schölkopf,et al.  A Compression Approach to Support Vector Model Selection , 2004, J. Mach. Learn. Res..

[46]  R. Keys.  Cubic convolution interpolation for digital image processing , 1981 .

[47]  Richard G. Baraniuk,et al.  Multiple basis wavelet denoising using Besov projections , 1999, Proceedings 1999 International Conference on Image Processing (Cat. 99CH36348).

[48]  R. Baddeley,et al.  A statistical analysis of natural images matches psychophysically derived orientation tuning curves , 1991, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[49]  Jitendra Malik,et al.  Contour and Texture Analysis for Image Segmentation , 2001, International Journal of Computer Vision.

[50]  S. Hyakin,et al.  Neural Networks: A Comprehensive Foundation , 1994 .

[51]  Olivier Chapelle,et al.  Model Selection for Support Vector Machines , 1999, NIPS.

[52]  Ralf Herbrich,et al.  Learning Kernel Classifiers: Theory and Algorithms , 2001 .

[53]  N. Cristianini,et al.  On Kernel-Target Alignment , 2001, NIPS.

[54]  Alexander J. Smola,et al.  Learning with kernels , 1998 .

[55]  D. Signorini,et al.  Neural networks , 1995, The Lancet.

[56]  Ho-Young Jung,et al.  Speech feature extraction using independent component analysis , 2000, 2000 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.00CH37100).