Locally Adaptive Metric Nearest-Neighbor Classification

Nearest-neighbor classification assumes locally constant class conditional probabilities. This assumption becomes invalid in high dimensions with finite samples due to the curse of dimensionality. Severe bias can be introduced under these conditions when using the nearest-neighbor rule. We propose a locally adaptive nearest-neighbor classification method to try to minimize bias. We use a chi-squared distance analysis to compute a flexible metric for producing neighborhoods that are highly adaptive to query locations. Neighborhoods are elongated along less relevant feature dimensions and constricted along most influential ones. As a result, the class conditional probabilities are smoother in the modified neighborhoods, whereby better classification performance can be achieved. The efficacy of our method is validated and compared against other techniques using both simulated and real-world data.

[1]  Keinosuke Fukunaga,et al.  The optimal distance measure for nearest neighbor classification , 1981, IEEE Trans. Inf. Theory.

[2]  Léon Bottou,et al.  Local Learning Algorithms , 1992, Neural Computation.

[3]  Thomas G. Dietterich What is machine learning? , 2020, Archives of Disease in Childhood.

[4]  R. Bellman,et al.  V. Adaptive Control Processes , 1964 .

[5]  W. Cleveland,et al.  Locally Weighted Regression: An Approach to Regression Analysis by Local Fitting , 1988 .

[6]  Andrew W. Moore,et al.  Locally Weighted Learning , 1997, Artificial Intelligence Review.

[7]  Alberto Maria Segre,et al.  Programs for Machine Learning , 1994 .

[8]  G. McLachlan Discriminant Analysis and Statistical Pattern Recognition , 1992 .

[9]  David J. Hand,et al.  The multi-class metric problem in nearest neighbour discrimination rules , 1990, Pattern Recognit..

[10]  Y. Chien,et al.  Pattern classification and scene analysis , 1974 .

[11]  Robert Tibshirani,et al.  Discriminant Adaptive Nearest Neighbor Classification , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[12]  David W. Aha,et al.  Lazy Learning , 1997, Springer Netherlands.

[13]  David G. Lowe,et al.  Similarity Metric Learning for a Variable-Kernel Classifier , 1995, Neural Computation.

[14]  Peter E. Hart,et al.  Pattern classification and scene analysis , 1974, A Wiley-Interscience publication.

[15]  Jerome H. Friedman,et al.  Flexible Metric Nearest Neighbor Classification , 1994 .

[16]  C. J. Stone,et al.  Consistent Nonparametric Regression , 1977 .

[17]  Peter E. Hart,et al.  Nearest neighbor pattern classification , 1967, IEEE Trans. Inf. Theory.