Entropy-based sliced inverse regression

The importance of dimension reduction has been increasing according to the growth of the size of available data in many fields. An appropriate dimension reduction method of raw data helps to reduce computational time and to expose the intrinsic structure of complex data. Sliced inverse regression is a well-known dimension reduction method for regression, which assumes an elliptical distribution for the explanatory variable, and ingeniously reduces the problem of dimension reduction to a simple eigenvalue problem. Sliced inverse regression is based on the strong assumptions on the data distribution and the form of regression function, and there are a number of methods to relax or remove these assumptions to extend the applicability of the inverse regression method. However, each method is known to have its drawbacks either theoretically or empirically. To alleviate drawbacks in the existing methods, a dimension reduction method for regression based on the notion of conditional entropy minimization is proposed. Using entropy as a measure of dispersion of data, a low dimensional subspace is estimated without assuming any specific distribution nor any regression function. The proposed method is shown to perform comparable or superior to the conventional methods through experiments using artificial and real-world datasets.

[1]  Radford M. Neal.  Pattern Recognition and Machine Learning , 2007, Technometrics.

[2]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[3]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[4]  Shaoli Wang,et al.  On Directional Regression for Dimension Reduction , 2007 .

[5]  Hideitsu Hino,et al.  A Conditional Entropy Minimization Criterion for Dimensionality Reduction and Multiple Kernel Learning , 2010, Neural Computation.

[6]  Shotaro Akaho,et al.  Learning algorithms utilizing quasi-geodesic flows on the Stiefel manifold , 2005, Neurocomputing.

[7]  Ian T. Jolliffe,et al.  Principal Component Analysis , 2002, International Encyclopedia of Statistical Science.

[8]  R. Fisher.  THE USE OF MULTIPLE MEASUREMENTS IN TAXONOMIC PROBLEMS , 1936 .

[9]  E. Oja,et al.  Independent Component Analysis , 2001 .

[10]  R. Dennis Cook,et al.  K-Means Inverse Regression , 2004, Technometrics.

[11]  Michael I. Jordan,et al.  Kernel dimension reduction in regression , 2009, 0908.1854.

[12]  Luca Scrucca,et al.  Model-based SIR for dimension reduction , 2011, Comput. Stat. Data Anal..

[13]  Robert Tibshirani,et al.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd Edition , 2001, Springer Series in Statistics.

[14]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[15]  Henry Horng-Shing Lu,et al.  SUPERVISED MOTION SEGMENTATION BY SPATIAL-FREQUENTIAL ANALYSIS AND DYNAMIC SLICED INVERSE REGRESSION , 2004 .

[16]  R. Cook.  Regression Graphics , 1994 .

[17]  Ker-Chau Li.  Sliced inverse regression for dimension reduction (with discussion) , 1991 .

[18]  Elaine B. Martin,et al.  On principal component analysis in L 1 , 2002 .

[19]  S. Weisberg,et al.  Comments on "Sliced inverse regression for dimension reduction" by K. C. Li , 1991 .

[20]  A. Tsybakov,et al.  Sliced Inverse Regression for Dimension Reduction - Comment , 1991 .

[21]  Jacob Goldberger,et al.  ICA based on a Smooth Estimation of the Differential Entropy , 2008, NIPS.

[22]  Henry Horng-Shing Lu,et al.  Iterative sliced inverse regression for segmentation of ultrasound and MR images , 2007, Pattern Recognit..

[23]  R. Cook,et al.  Sufficient Dimension Reduction via Inverse Regression , 2005 .

[24]  Joel Owen,et al.  On the Class of Elliptical Distributions and Their Applications to the Theory of Portfolio Choice , 1983 .

[25]  S T Roweis,et al.  Nonlinear dimensionality reduction by locally linear embedding. , 2000, Science.

[26]  Simone G. O. Fiori,et al.  Formulation and integration of learning differential equations on the stiefel manifold , 2005, IEEE Transactions on Neural Networks.

[27]  Masashi Sugiyama,et al.  Sufficient Dimension Reduction via Squared-Loss Mutual Information Estimation , 2010, Neural Computation.

[28]  Ker-Chau Li,et al.  Sliced Inverse Regression for Dimension Reduction , 1991 .

[29]  J. Tenenbaum,et al.  A global geometric framework for nonlinear dimensionality reduction. , 2000, Science.

[30]  Chun-Houh Chen,et al.  CAN SIR BE AS POPULAR AS MULTIPLE LINEAR REGRESSION , 2003 .