Comparison of Neural and Statistical Classifiers - Theory and Practice

Pattern classi cation using neural networks and statistical methods is discussed. We rst give a tutorial overview that groups popular classi ers according to their underlying mathematical principles into several distinct categories. Starting from the Bayes classi er, one division is whether the classi er is explicitly estimating class conditional densities, or directly estimating the posterior probabilities by regression. Another criterion is the exibility of the architecture in the sense of how rich the discriminant function family is. Still one dimension is neural vs. nonneural learning: neural learning is characterized by simple local computations in a number of real or virtual processing elements. Based on these comparisons, a number of classi cation methods were selected for a case study that uses handwritten digit data. An e ort was made to get fair estimates of their true classi cation performance, thus training set cross-validation was extensively used to design the various classi ers. The classi cation errors were estimated with an independent testing set. The performance of a number of most typical neural and statistical classi ers was compared. Also, four methods of our own were used in the comparisons: the Reduced Kernel Discriminant Analysis (RKDA), the Learning kNearest Neighbor Classi er, the Averaged Learning Subspace Method (ALSM), and a modi ed version of Kernel Discriminant Analysis. Also, committee classi ers and classi cation with rejection were considered. In these experiments, the Local Linear Regression (LLR) method, although computationally prohibitively heavy, was the best classi er from the point of view of classi cation accuracy, with the Averaged Learning Subspace Method (ALSM) following close behind. For methods having both a learning and a non-learning version, error correcting learning seemed to give an advantage.