Automated feature weighting in clustering with separable distances and inner product induced norms - A theoretical generalization

The feature weighting schemes for crisp and fuzzy clustering are generalized.Alternating optimization heuristics are derived for hard and fuzzy clustering.Existence of the minimizer of the proposed optimization tasks is proved.Prominent weighting schemes shown as special cases of the proposed generalization. For decades practitioners have been using the separable distance and inner product induced norms as the distance measures for k-means, Fuzzy C-Means (FCM), hard and fuzzy k-modes clustering algorithms. In this paper, we introduce a novel concept of automated feature weighting for general clustering algorithms (including both hard and fuzzy clustering) to amplify the effect of the discriminating features, which play a key role in identifying the naturally occurring groups in data with minimal computational overheads. We derive a Lloyd heuristic and an alternating optimization algorithm for solving the hard and the fuzzy clustering problems respectively. We also investigate the mathematical nature of the problems in sufficient details to guarantee the existence and feasibility of a solution at each iteration of the aforementioned algorithms. We show that majority of the automated feature weighting schemes existing in the literature turn out to be the special cases of this proposed generalization. A brief discussion on practical utility of the proposed generalization is also presented along with indication of the future applications of this new approach.

[1]  J. C. Dunn,et al.  A Fuzzy Relative of the ISODATA Process and Its Use in Detecting Compact Well-Separated Clusters , 1973 .

[2]  Marc Teboulle,et al.  A Unified Continuous Optimization Framework for Center-Based Clustering Methods , 2007, J. Mach. Learn. Res..

[3]  Vladimir Makarenkov,et al.  Optimal Variable Weighting for Ultrametric and Additive Trees and K-means Partitioning: Methods and Software , 2001, J. Classif..

[4]  Hong Jia,et al.  Feature Weighted Kernel Clustering with Application to Medical Data Analysis , 2013, Brain and Health Informatics.

[5]  Adrian S. Lewis,et al.  Convex Analysis And Nonlinear Optimization , 2000 .

[6]  Vladimir J. Lumelsky,et al.  A combined algorithm for weighting the variables and clustering in the clustering problem , 1982, Pattern Recognit..

[7]  Michael K. Ng,et al.  A fuzzy k-modes algorithm for clustering categorical data , 1999, IEEE Trans. Fuzzy Syst..

[8]  J. MacQueen Some methods for classification and analysis of multivariate observations , 1967 .

[9]  Der-Bang Wu,et al.  Fuzzy C-mean algorithm based on “complete” Mahalanobis distances , 2008, 2008 International Conference on Machine Learning and Cybernetics.

[10]  Michael K. Ng,et al.  Automated variable weighting in k-means type clustering , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  Joshua Zhexue Huang,et al.  Extensions to the k-Means Algorithm for Clustering Large Data Sets with Categorical Values , 1998, Data Mining and Knowledge Discovery.

[12]  J. Bezdek,et al.  FCM: The fuzzy c-means clustering algorithm , 1984 .

[13]  Hichem Frigui,et al.  Unsupervised learning of prototypes and attribute weights , 2004, Pattern Recognit..

[14]  James C. Bezdek,et al.  Convergence of Alternating Optimization , 2003, Neural Parallel Sci. Comput..

[15]  F. Klawonn,et al.  Fuzzy clustering with weighting of data variables , 2000 .

[16]  Anil K. Jain Data clustering: 50 years beyond K-means , 2010, Pattern Recognit. Lett..

[17]  S. P. Lloyd,et al.  Least squares quantization in PCM , 1982, IEEE Trans. Inf. Theory.

[18]  Jeng-Ming Yih,et al.  Fuzzy C-means algorithm based on standard mahalanobis distances , 2009 .

[19]  Peter H. A. Sneath,et al.  Numerical Taxonomy: The Principles and Practice of Numerical Classification , 1973 .

[20]  Francisco de A. T. de Carvalho,et al.  Kernel-based hard clustering methods in the feature space with automatic variable weighting , 2014, Pattern Recognit..

[21]  Shokri Z. Selim,et al.  K-Means-Type Algorithms: A Generalized Convergence Theorem and Characterization of Local Optimality , 1984, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  Yunming Ye,et al.  TW-k-means: Automated two-level variable weighting clustering algorithm for multiview data , 2013, IEEE Transactions on Knowledge and Data Engineering.

[23]  G. Soete OVWTRE: A program for optimal variable weighting for ultrametric and additive tree fitting , 1988 .

[24]  Anil K. Jain Data clustering: 50 years beyond K-means , 2008, Pattern Recognit. Lett..

[25]  J. Carroll,et al.  Synthesized clustering: A method for amalgamating alternative clustering bases with differential weighting of variables , 1984 .

[26]  Lotfi A. Zadeh,et al.  Fuzzy Sets , 1996, Inf. Control..

[27]  Michael K. Ng,et al.  An optimization algorithm for clustering using weighted dissimilarity measures , 2004, Pattern Recognit..

[28]  Abdolhossein Sarrafzadeh,et al.  Fuzzy C-means based on Automated Variable Feature Weighting , 2013 .

[29]  Renato Cordeiro de Amorim,et al.  Minkowski metric, feature weighting and anomalous cluster initializing in K-Means clustering , 2012, Pattern Recognit..

[30]  James C. Bezdek,et al.  Pattern Recognition with Fuzzy Objective Function Algorithms , 1981, Advanced Applications in Pattern Recognition.

[31]  Jiye Liang,et al.  A novel attribute weighting algorithm for clustering high-dimensional categorical data , 2011, Pattern Recognit..