Sparse Online Greedy Support Vector Regression

We present a novel algorithm for sparse online greedy kernel-based nonlinear regression. This algorithm improves current approaches to kernel-based regression in two aspects. First, it operates online - at each time step it observes a single new input sample, performs an update and discards it. Second, the solution maintained is extremely sparse. This is achieved by an explicit greedy sparsification process that admits into the kernel representation a new input sample only if its feature space image is linearly independent of the images of previously admitted samples. We show that the algorithm implements a form of gradient ascent and demonstrate its scaling and noise tolerance properties on three benchmark regression problems.

[1]  Tom Downs,et al.  Exact Simplification of Support Vector Solutions , 2002, J. Mach. Learn. Res..

[2]  Liva Ralaivola,et al.  Incremental Support Vector Machine Learning: A Local Approach , 2001, ICANN.

[3]  Alexander J. Smola,et al.  Learning with kernels , 1998 .

[4]  G. Wahba Spline models for observational data , 1990 .

[5]  Dustin Boswell,et al.  Introduction to Support Vector Machines , 2002 .

[6]  Huan Liu,et al.  Handling concept drifts in incremental learning with support vector machines , 1999, KDD '99.

[7]  Ralf Herbrich,et al.  Learning Kernel Classifiers: Theory and Algorithms , 2001 .

[8]  Nello Cristianini,et al.  An Introduction to Support Vector Machines and Other Kernel-based Learning Methods , 2000 .

[9]  Sayan Mukherjee,et al.  Choosing Multiple Parameters for Support Vector Machines , 2002, Machine Learning.

[10]  Gert Cauwenberghs,et al.  Incremental and Decremental Support Vector Machine Learning , 2000, NIPS.

[11]  Sethu Vijayakumar,et al.  Sequential Support Vector Classifiers and Regression , 1999, IIA/SOCO.

[12]  Christopher K. I. Williams Learning Kernel Classifiers , 2003 .

[13]  John C. Platt,et al.  Fast training of support vector machines using sequential minimal optimization, advances in kernel methods , 1999 .

[14]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[15]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[16]  Glenn Fung,et al.  Incremental Support Vector Machine Classification , 2002, SDM.

[17]  Stefan Rüping,et al.  Incremental Learning with Support Vector Machines , 2001, ICDM.

[18]  Samy Bengio,et al.  SVMTorch: Support Vector Machines for Large-Scale Regression Problems , 2001, J. Mach. Learn. Res..

[19]  Bernhard Schölkopf,et al.  Improving the Accuracy and Speed of Support Vector Machines , 1996, NIPS.