Feature-weighted clustering with inner product induced norm based dissimilarity measures: an optimization perspective

The performance of a clustering algorithm can be improved by assigning appropriate weights to different features of the data. Such feature weighting is likely to reduce the effect of noise and irrelevant features while enhancing the effect of the discriminative features simultaneously. For the clustering purpose, feature-weighted dissimilarity measures are so far limited to Euclidean, Mahalanobis, and exponential distances. In this article, we introduce a novel feature weighting scheme for the general class of inner product induced norm (IPIN) based weighted dissimilarity measures. This class has a wide range and includes the three above-mentioned distances as special cases. We develop the general algorithms to solve the hard (k-means) and fuzzy (fuzzy c-means) partitional clustering problems and undertake in-depth analyses of the convergence of the algorithms as well. In addition, we address issues like feasibility and uniqueness of the solutions of these problems in sufficient details. The novelty of the article lies in the introduction of a general feature weighting scheme for the generalized class of IPIN-based dissimilarity measures and a complete convergence analysis for the automated feature-weighted clustering algorithms using such measures.

[1]  A. Asuncion,et al.  UCI Machine Learning Repository, University of California, Irvine, School of Information and Computer Sciences , 2007 .

[2]  Jeng-Ming Yih,et al.  Fuzzy C-means algorithm based on standard mahalanobis distances , 2009 .

[3]  Marc Teboulle,et al.  Clustering with Entropy-Like k-Means Algorithms , 2006, Grouping Multidimensional Data.

[4]  Hsiang-Chuan Liu,et al.  Fuzzy C-Means Algorithm Based on Common Mahalanobis Distances , 2009, J. Multiple Valued Log. Soft Comput..

[5]  Hui Xiong,et al.  A Generalization of Distance Functions for Fuzzy $c$ -Means Clustering With Centroids of Arithmetic Means , 2012, IEEE Transactions on Fuzzy Systems.

[6]  Pierpaolo D'Urso,et al.  Exponential distance-based fuzzy clustering for interval-valued data , 2017, Fuzzy Optim. Decis. Mak..

[7]  S. P. Lloyd,et al.  Least squares quantization in PCM , 1982, IEEE Trans. Inf. Theory.

[8]  Swagatam Das,et al.  Geometric divergence based fuzzy clustering with strong resilience to noise features , 2016, Pattern Recognit. Lett..

[9]  Jongwoo Kim,et al.  A note on the Gustafson-Kessel and adaptive fuzzy clustering algorithms , 1999, IEEE Trans. Fuzzy Syst..

[10]  Michael R. Anderberg,et al.  Cluster Analysis for Applications , 1973 .

[11]  F. Klawonn,et al.  Fuzzy clustering with weighting of data variables , 2000 .

[12]  Gene H. Golub,et al.  Matrix computations , 1983 .

[13]  Anil K. Jain,et al.  A self-organizing network for hyperellipsoidal clustering (HEC) , 1996, IEEE Trans. Neural Networks.

[14]  Marc Teboulle,et al.  A Unified Continuous Optimization Framework for Center-Based Clustering Methods , 2007, J. Mach. Learn. Res..

[15]  Vladimir J. Lumelsky,et al.  A combined algorithm for weighting the variables and clustering in the clustering problem , 1982, Pattern Recognit..

[16]  Abdolhossein Sarrafzadeh,et al.  Fuzzy C-means based on Automated Variable Feature Weighting , 2013 .

[17]  Ka Yee Yeung,et al.  Details of the Adjusted Rand index and Clustering algorithms Supplement to the paper “ An empirical study on Principal Component Analysis for clustering gene expression data ” ( to appear in Bioinformatics ) , 2001 .

[18]  Rajesh N. Davé,et al.  Validating fuzzy partitions obtained through c-shells clustering , 1996, Pattern Recognit. Lett..

[19]  J. MacQueen.  Some methods for classification and analysis of multivariate observations , 1967 .

[20]  Jian Yu,et al.  Analysis of Parameter Selection for Gustafson–Kessel Fuzzy Clustering Using Jacobian Matrix , 2015, IEEE Transactions on Fuzzy Systems.

[21]  Frank Klawonn,et al.  What Is Fuzzy about Fuzzy Clustering? Understanding and Improving the Concept of the Fuzzifier , 2003, IDA.

[22]  Miin-Shen Yang,et al.  Exponential-Distance Weighted K-Means Algorithm with Spatial Constraints for Color Image Segmentation , 2011, 2011 International Conference on Multimedia and Signal Processing.

[23]  Swagatam Das,et al.  Automated feature weighting in clustering with separable distances and inner product induced norms - A theoretical generalization , 2015, Pattern Recognit. Lett..

[24]  Inderjit S. Dhillon,et al.  Clustering with Bregman Divergences , 2005, J. Mach. Learn. Res..

[25]  J. C. Dunn,et al.  A Fuzzy Relative of the ISODATA Process and Its Use in Detecting Compact Well-Separated Clusters , 1973 .

[26]  Stefan Friedrich,et al.  Topology , 2019, Arch. Formal Proofs.

[27]  Sanghamitra Bandyopadhyay,et al.  Classification and learning using genetic algorithms - applications in bioinformatics and web intelligence , 2007, Natural computing series.

[28]  Ujjwal Maulik,et al.  Genetic clustering for automatic evolution of clusters and application to image classification , 2002, Pattern Recognit..

[29]  W. Scott Spangler,et al.  Feature Weighting in k-Means Clustering , 2003, Machine Learning.

[30]  Michael K. Ng,et al.  Automated variable weighting in k-means type clustering , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31]  Swagatam Das,et al.  Optimizing cluster structures with inner product induced norm based dissimilarity measures: Theoretical development and convergence analysis , 2016, Inf. Sci..

[32]  Anil K. Jain,et al.  Data clustering: a review , 1999, CSUR.

[33]  Vladimir Makarenkov,et al.  Optimal Variable Weighting for Ultrametric and Additive Trees and K-means Partitioning: Methods and Software , 2001, J. Classif..

[34]  J. Carroll,et al.  Synthesized clustering: A method for amalgamating alternative clustering bases with differential weighting of variables , 1984 .

[35]  Swagatam Das,et al.  Categorical fuzzy k-modes clustering with automated feature weight learning , 2015, Neurocomputing.

[36]  J. Bezdek.  Cluster Validity with Fuzzy Sets , 1973 .

[37]  G. Soete.  OVWTRE: A program for optimal variable weighting for ultrametric and additive tree fitting , 1988 .

[38]  Tunc Geveci,et al.  Advanced Calculus , 2014, Nature.

[39]  Tian-Wei Sheu,et al.  A New Fuzzy Possibility Clustering Algorithms Based on Unsupervised Mahalanobis Distances , 2007, 2007 International Conference on Machine Learning and Cybernetics.

[40]  Pavel Berkhin,et al.  A Survey of Clustering Data Mining Techniques , 2006, Grouping Multidimensional Data.

[41]  Ka Yee Yeung,et al.  Principal component analysis for clustering gene expression data , 2001, Bioinform..

[42]  E. M. L. Beale,et al.  Nonlinear Programming: A Unified Approach. , 1970 .

[43]  Gerardo Beni,et al.  A Validity Measure for Fuzzy Clustering , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[44]  James C. Bezdek,et al.  Convergence of Alternating Optimization , 2003, Neural Parallel Sci. Comput..

[45]  L. Hubert,et al.  Comparing partitions , 1985 .

[46]  Shokri Z. Selim,et al.  K-Means-Type Algorithms: A Generalized Convergence Theorem and Characterization of Local Optimality , 1984, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[47]  R. Sokal,et al.  Numerical Taxonomy: The Principles and Practice of Numerical Classification. , 1975 .

[48]  Isak Gath,et al.  Unsupervised Optimal Fuzzy Clustering , 1989, IEEE Trans. Pattern Anal. Mach. Intell..

[49]  Hsiang-Chuan Liu,et al.  Fuzzy C-Mean Algorithm Based on Mahalanobis Distances and Better Initial Values , 2007 .

[50]  Donald Gustafson,et al.  Fuzzy clustering with a fuzzy covariance matrix , 1978, 1978 IEEE Conference on Decision and Control including the 17th Symposium on Adaptive Processes.

[51]  James C. Bezdek,et al.  Pattern Recognition with Fuzzy Objective Function Algorithms , 1981, Advanced Applications in Pattern Recognition.

[52]  Matthias Wölfel,et al.  Feature weighted mahalanobis distance: Improved robustness for Gaussian classifiers , 2005, 2005 13th European Signal Processing Conference.