A parameter independent fuzzy weighted k-Nearest neighbor classifier

Abstract Unlike the canonical k-Nearest Neighbor classifier (kNN) which treat the neighbors equally, the Fuzzy k-Nearest Neighbor (FkNN) classifier imposes a weight on each of the k nearest neighbors based on their distances from the query point, by using a fuzzy membership function. FkNN though improves the performance of kNN, requires optimizing additional data dependent parameters other than k. Furthermore, FkNN does not consider the effect of those representative features of a data point which may be noisy, redundant, and may not contain useful information to distinctly identify a specific class. We attempt to address both of these issues in the current study by proposing a Parameter Independent Fuzzy class-specific Feature Weighted k-Nearest Neighbor (PIFW-kNN) classifier. PIFW-kNN formulates the issues of choosing a suitable value of k and a set of class dependent optimum weights for the features as a single-objective continuous non-convex optimization problem. We solve this problem by using a very competitive variant of Differential Evolution (DE), called Success-History based Adaptive DE (SHADE). We perform extensive experiments to demonstrate the improved accuracy of PIFW-kNN compared to the other state-of-the-art classifiers.

[1]  P. N. Suganthan,et al.  Differential Evolution: A Survey of the State-of-the-Art , 2011, IEEE Transactions on Evolutionary Computation.

[2]  Enrique Vidal,et al.  Learning weighted metrics to minimize nearest-neighbor classification error , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  José Cristóbal Riquelme Santos,et al.  On the evolutionary optimization of k-NN by label-dependent feature weighting , 2012, Pattern Recognit. Lett..

[4]  Bo Tang,et al.  ENN: Extended Nearest Neighbor Method for Pattern Recognition [Research Frontier] , 2015, IEEE Computational Intelligence Magazine.

[5]  Leon N. Cooper,et al.  Improving nearest neighbor rule with a simple adaptive distance measure , 2007, Pattern Recognit. Lett..

[6]  Dimitrios Gunopulos,et al.  Locally Adaptive Metric Nearest-Neighbor Classification , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[7]  Songbo Tan,et al.  Neighbor-weighted K-nearest neighbor for unbalanced text corpus , 2005, Expert Syst. Appl..

[8]  David G. Stork,et al.  Pattern Classification , 1973 .

[9]  Gautam Bhattacharya,et al.  An affinity-based new local distance function and similarity measure for kNN algorithm , 2012, Pattern Recognit. Lett..

[10]  Douglas A. Wolfe,et al.  Nonparametric Statistical Methods , 1973 .

[11]  Ponnuthurai N. Suganthan,et al.  Recent advances in differential evolution - An updated survey , 2016, Swarm Evol. Comput..

[12]  B. Park,et al.  Choice of neighbor order in nearest-neighbor classification , 2008, 0810.5276.

[13]  Kilian Q. Weinberger,et al.  Distance Metric Learning for Large Margin Nearest Neighbor Classification , 2005, NIPS.

[14]  R. Samworth.  Optimal weighted nearest neighbour classifiers , 2011, 1101.5783.

[15]  Francisco Herrera,et al.  Evolutionary fuzzy k-nearest neighbors algorithm using interval-valued fuzzy sets , 2016, Inf. Sci..

[16]  Anil K. Jain,et al.  Dimensionality reduction using genetic algorithms , 2000, IEEE Trans. Evol. Comput..

[17]  D. Quade.  Using Weighted Rankings in the Analysis of Complete Blocks with Additive Block Effects , 1979 .

[18]  Fabrizio Angiulli,et al.  Nearest Neighbor-Based Classification of Uncertain Data , 2013, TKDD.

[19]  Alex S. Fukunaga,et al.  Success-history based parameter adaptation for Differential Evolution , 2013, 2013 IEEE Congress on Evolutionary Computation.

[20]  Rainer Storn,et al.  Differential Evolution – A Simple and Efficient Heuristic for global Optimization over Continuous Spaces , 1997, J. Glob. Optim..

[21]  J. L. Hodges,et al.  Discriminatory Analysis - Nonparametric Discrimination: Consistency Properties , 1989 .

[22]  Francisco Herrera,et al.  A practical tutorial on the use of nonparametric statistical tests as a methodology for comparing evolutionary and swarm intelligence algorithms , 2011, Swarm Evol. Comput..

[23]  Arthur C. Sanderson,et al.  JADE: Adaptive Differential Evolution With Optional External Archive , 2009, IEEE Transactions on Evolutionary Computation.

[24]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[25]  James M. Keller,et al.  A fuzzy K-nearest neighbor algorithm , 1985, IEEE Transactions on Systems, Man, and Cybernetics.

[26]  Francisco Herrera,et al.  Fuzzy nearest neighbor algorithms: Taxonomy, experimental analysis and prospects , 2014, Inf. Sci..

[27]  Subhabrata Chakraborti,et al.  Nonparametric Statistical Inference , 2011, International Encyclopedia of Statistical Science.

[28]  Peter E. Hart,et al.  Nearest neighbor pattern classification , 1967, IEEE Trans. Inf. Theory.

[29]  Larry J. Eshelman,et al.  The CHC Adaptive Search Algorithm: How to Have Safe Search When Engaging in Nontraditional Genetic Recombination , 1990, FOGA.

[30]  Chengjun Liu,et al.  A Novel Locally Linear KNN Method With Applications to Visual Recognition , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[31]  Anil K. Ghosh,et al.  On optimum choice of k , 2006, Comput. Stat. Data Anal..

[32]  Jing J. Liang,et al.  Problem Definitions and Evaluation Criteria for the CEC 2005 Special Session on Real-Parameter Optimization , 2005 .

[33]  Sankha Subhra Mullick,et al.  A switched parameter differential evolution with optional blending crossover for scalable numerical optimization , 2017, Appl. Soft Comput..