Benchmarking Support Vector Machines

Support Vector Machines (SVMs) are rarely benchmarked against other classification or regression methods. We compare a popular SVM implementation (libsvm) to 16 classification methods and 9 regression methods-all accessible through the software R-by the means of standard performance measures (classification error and mean squared error) which are also analyzed by the means of bias-variance decompositions. SVMs showed mostly good performances both on classification and regression tasks, but other methods proved to be very competitive.

[1]  Hélène Paugam-Moisy,et al.  A new multi-class SVM based on a uniform convergence result , 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium.

[2]  Brian D. Ripley,et al.  Pattern Recognition and Neural Networks , 1996 .

[3]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[4]  David R. Musicant,et al.  Robust Linear and Support Vector Regression , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[5]  Wei-Yin Loh,et al.  A Comparison of Prediction Accuracy, Complexity, and Training Time of Thirty-Three Old and New Classification Algorithms , 2000, Machine Learning.

[6]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[7]  Marti A. Hearst.  Trends & Controversies: Support Vector Machines , 1998, IEEE Intell. Syst..

[8]  Leo Breiman,et al.  Classification and Regression Trees , 1984 .

[9]  Hava T. Siegelmann,et al.  Support Vector Clustering , 2002, J. Mach. Learn. Res..

[10]  William N. Venables,et al.  Modern Applied Statistics with S-Plus. , 1996 .

[11]  Thomas G. Dietterich,et al.  Error-Correcting Output Coding Corrects Bias and Variance , 1995, ICML.

[12]  David J. Spiegelhalter,et al.  Machine Learning, Neural and Statistical Classification , 2009 .

[13]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[14]  Torsten Hothorn,et al.  Double-Bagging: Combining Classifiers by Bootstrap Aggregation , 2002, Pattern Recognit..

[15]  K. Bennett,et al.  Optimization Approaches to Semi-Supervised Learning , 2001 .

[16]  Ralf Herbrich,et al.  Bayes Point Machines: Estimating the Bayes Point in Kernel Space , 1999 .

[17]  Jason Weston,et al.  Multi-Class Support Vector Machines , 1998 .

[18]  Davide Anguita,et al.  Evaluating the Generalization Ability of Support Vector Machines through the Bootstrap , 2000, Neural Processing Letters.

[19]  J. Friedman.  Multivariate adaptive regression splines , 1990 .

[20]  Johan A. K. Suykens,et al.  Benchmarking Least Squares Support Vector Machine Classifiers , 2004, Machine Learning.

[21]  Ron Kohavi,et al.  Bias Plus Variance Decomposition for Zero-One Loss Functions , 1996, ICML.

[22]  J. Friedman.  Stochastic gradient boosting , 2002 .

[23]  J. Chambers.  Programming with Data: A Guide to the S Language , 1998 .

[24]  Pascal Vincent,et al.  A Neural Support Vector Network architecture with adaptive kernels , 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium.

[25]  Michael E. Tipping.  The Relevance Vector Machine , 1999, NIPS.

[26]  Lutz Prechelt,et al.  Some notes on neural learning algorithm benchmarking , 1995, Neurocomputing.

[27]  J. Friedman.  Greedy function approximation: A gradient boosting machine. , 2001 .

[28]  Ross Ihaka,et al.  Gentleman R: R: A language for data analysis and graphics , 1996 .

[29]  Gunnar Rätsch,et al.  Soft Margins for AdaBoost , 2001, Machine Learning.

[30]  Trevor Hastie,et al.  The Elements of Statistical Learning , 2001 .

[31]  Radford M. Neal.  Assessing Relevance determination methods using DELVE , 1998 .

[32]  Liva Ralaivola,et al.  Incremental Support Vector Machine Learning: A Local Approach , 2001, ICANN.

[33]  Eddy Mayoraz.  Multiclass Classification with Pairwise Coupled Neural Networks or Support Vector Machines , 2001, ICANN.

[34]  Peter Auer,et al.  Reducing Communication for Distributed Learning in Neural Networks , 2002, ICANN.