On Speeding Up Support Vector Machines: Proximity Graphs Versus Random Sampling for Pre-Selection Condensation

Support vector machines (SVMs) are considered to be the best machine learning algorithms for minimizing the predictive probability of misclassification. However, their drawback is that for large data sets the computation of the optimal decision boundary is a time consuming function of the size of the training set. Hence several methods have been proposed to speed up the SVM algorithm. Here three methods used to speed up the computation of the SVM classifiers are compared experimentally using a musical genre classification problem. The simplest method pre-selects a random sample of the data before the application of the SVM algorithm. Two additional methods use proximity graphs to pre-select data that are near the decision boundary. One uses k-Nearest Neighbor graphs and the other Relative Neighborhood Graphs to accomplish the task. Keywords—Machine learning, data mining, support vector machines, proximity graphs, relative-neighborhood graphs, k-nearestneighbor graphs, random sampling, training data condensation.

[1]  S. Sathiya Keerthi,et al.  Improvements to Platt's SMO Algorithm for SVM Classifier Design , 2001, Neural Computation.

[2]  Steven J. Simske,et al.  Training Set Compression by Incremental Clustering , 2011 .

[3]  Godfried T. Toussaint,et al.  Geometric Decision Rules for Instance-Based Learning Problems , 2005, PReMI.

[4]  Fabrizio Angiulli,et al.  Fast Nearest Neighbor Condensation for Large Data Sets Classification , 2007, IEEE Transactions on Knowledge and Data Engineering.

[5]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[6]  Narendra Ahuja,et al.  A geometric approach to train support vector machines , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[7]  Edward Y. Chang,et al.  Concept boundary detection for speeding up SVMs , 2006, ICML '06.

[8]  C. G. Hilborn,et al.  The Condensed Nearest Neighbor Rule , 1967 .

[9]  Godfried T. Toussaint,et al.  Proximity-Graph Instance-Based Learning, Support Vector Machines, and High Dimensionality: An Empirical Comparison , 2012, MLDM.

[10]  Xinyu Guo,et al.  Pruning Support Vector Machines Without Altering Performances , 2008, IEEE Transactions on Neural Networks.

[11]  W..  Relative Neighborhood Graphs and Their Relatives , 2004 .

[12]  Yuh-Jye Lee,et al.  RSVM: Reduced Support Vector Machines , 2001, SDM.

[13]  Godfried T. Toussaint,et al.  The relative neighbourhood graph of a finite planar set , 1980, Pattern Recognit..

[14]  Daniel H. Huson,et al.  SplitsTree-a program for analyzing and visualizing evolutionary data , 1997 .

[15]  Frank Weber,et al.  Optimal Reduced-Set Vectors for Support Vector Machines with a Quadratic Kernel , 2004, Neural Computation.

[16]  Wentao Mao,et al.  Pre-extracting method for SVM classification based on the non-parametric K-NN rule , 2008, 2008 19th International Conference on Pattern Recognition.

[17]  Ivor W. Tsang,et al.  Core Vector Machines: Fast SVM Training on Very Large Data Sets , 2005, J. Mach. Learn. Res..

[18]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[19]  Dennis L. Wilson,et al.  Asymptotic Properties of Nearest Neighbor Rules Using Edited Data , 1972, IEEE Trans. Syst. Man Cybern..

[20]  Antônio de Pádua Braga,et al.  SVM-KM: speeding SVMs learning with a priori cluster selection and k-means , 2000, Proceedings. Vol.1. Sixth Brazilian Symposium on Neural Networks.

[21]  S. Halgamuge,et al.  Reducing the Number of Training Samples for Fast Support Vector Machine Classification , 2004 .

[22]  Peter E. Hart,et al.  Nearest neighbor pattern classification , 1967, IEEE Trans. Inf. Theory.

[23]  John C. Platt,et al.  Fast training of support vector machines using sequential minimal optimization, advances in kernel methods , 1999 .

[24]  Irwin King,et al.  A study of the relationship between support vector machine and Gabriel graph , 2002, Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN'02 (Cat. No.02CH37290).

[25]  Tom Downs,et al.  Exact Simplification of Support Vector Solutions , 2002, J. Mach. Learn. Res..

[26]  E. Mayr.  Cladistic analysis or cladistic classification , 2009 .

[27]  Belur V. Dasarathy,et al.  Tandem fusion of nearest neighbor editing and condensing algorithms - data dimensionality effects , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[28]  Emilia Gómez,et al.  Comparative Analysis of Music Recordings from Western and Non-Western traditions by Automatic Tonal Feature Extraction , 2008 .

[29]  Leon N. Cooper,et al.  Training Data Selection for Support Vector Machines , 2005, ICNC.

[30]  Godfried T. Toussaint,et al.  Geometric proximity graphs for improving nearest neighbor methods in instance-based learning and data mining , 2005, Int. J. Comput. Geom. Appl..

[31]  Cheng-Lin Liu,et al.  Fast multi-class sample reduction for speeding up support vector machines , 2011, 2011 IEEE International Workshop on Machine Learning for Signal Processing.

[32]  S. Sathiya Keerthi,et al.  Building Support Vector Machines with Reduced Classifier Complexity , 2006, J. Mach. Learn. Res..

[33]  O Gascuel,et al.  BIONJ: an improved version of the NJ algorithm based on a simple model of sequence data. , 1997, Molecular biology and evolution.

[34]  Chunguang Zhou,et al.  A BOUNDARY METHOD TO SPEED UP TRAINING SUPPORT VECTOR MACHINES , 2006 .

[35]  Katya Scheinberg,et al.  Efficient SVM Training Using Low-Rank Kernel Representations , 2002, J. Mach. Learn. Res..

[36]  Annabella Astorino,et al.  Scaling Up Support Vector Machines Using Nearest Neighbor Condensation , 2010, IEEE Transactions on Neural Networks.

[37]  Sayan Mukherjee,et al.  Choosing Multiple Parameters for Support Vector Machines , 2002, Machine Learning.

[38]  Sergios Theodoridis,et al.  A geometric approach to Support Vector Machine (SVM) classification , 2006, IEEE Transactions on Neural Networks.

[39]  Thorsten Joachims,et al.  Making large scale SVM learning practical , 1998 .

[40]  Bernhard Schölkopf,et al.  Improving the Accuracy and Speed of Support Vector Machines , 1996, NIPS.

[41]  Irwin King,et al.  Locating support vectors via /spl beta/-skeleton technique , 2002, Proceedings of the 9th International Conference on Neural Information Processing, 2002. ICONIP '02..

[42]  Leon N. Cooper,et al.  Selecting Data for Fast Support Vector Machines Training , 2007, Trends in Neural Computation.