Why significant variables aren’t automatically good predictors

Significance A recent puzzle in the big data scientific literature is that an increase in explanatory variables found to be significantly correlated with an outcome variable does not necessarily lead to improvements in prediction. This problem occurs in both simple and complex data. We offer explanations and statistical insights into why higher significance does not automatically imply stronger predictivity and why variables with strong predictivity sometimes fail to be significant. We suggest shifting the research agenda toward searching for a criterion to locate highly predictive variables rather than highly significant variables. We offer an alternative approach, the partition retention method, which was effective in reducing prediction error from 30% to 8% on a long-studied breast cancer data set. Thus far, genome-wide association studies (GWAS) have been disappointing in the inability of investigators to use the results of identified, statistically significant variants in complex diseases to make predictions useful for personalized medicine. Why are significant variables not leading to good prediction of outcomes? We point out that this problem is prevalent in simple as well as complex data, in the sciences as well as the social sciences. We offer a brief explanation and some statistical insights on why higher significance cannot automatically imply stronger predictivity and illustrate through simulations and a real breast cancer example. We also demonstrate that highly predictive variables do not necessarily appear as highly significant, thus evading the researcher using significance-based methods. We point out that what makes variables good for prediction versus significance depends on different properties of the underlying distributions. If prediction is the goal, we must lay aside significance as the only selection standard. We suggest that progress in prediction requires efforts toward a new research agenda of searching for a novel criterion to retrieve highly predictive variables rather than highly significant variables. We offer an alternative approach that was not designed for significance, the partition retention method, which was very effective predicting on a long-studied breast cancer data set, by reducing the classification error rate from 30% to 8%.

[1]  Yudong D. He,et al.  Gene expression profiling predicts clinical outcome of breast cancer , 2002, Nature.

[2]  Isabelle Guyon,et al.  An Introduction to Variable and Feature Selection , 2003, J. Mach. Learn. Res..

[3]  Yuichi Mori,et al.  Handbook of Computational Statistics , 2004 .

[4]  I. Welch,et al.  A Comprehensive Look at the Empirical Performance of Equity Premium Prediction II , 2004, SSRN Electronic Journal.

[5]  Tian Zheng,et al.  A demonstration and findings of a statistical approach through reanalysis of inflammatory bowel disease data. , 2004, Proceedings of the National Academy of Sciences of the United States of America.

[6]  Tian Zheng,et al.  Backward Genotype-Trait Association (BGTA)-Based Dissection of Complex Traits in Case-Control Designs , 2006, Human Heredity.

[7]  Pedro Larrañaga,et al.  Bioinformatics Advance Access published August 24, 2007 A review of feature selection techniques in bioinformatics , 2022 .

[8]  Pär Stattin,et al.  Cumulative association of five genetic variants with prostate cancer. , 2008, The New England journal of medicine.

[9]  Cornelia M van Duijn,et al.  Genome-based prediction of common diseases: advances and prospects. , 2008, Human molecular genetics.

[10]  Herman Chernoff,et al.  Discovering interactions among BRCA1 and other candidate genes associated with sporadic breast cancer , 2008, Proceedings of the National Academy of Sciences.

[11]  Daniel E. Weeks,et al.  Interpretation of Genetic Association Studies: Markers with Replicated Highly Significant Odds Ratios May Be Poor Classifiers , 2009, PLoS genetics.

[12]  E. Dougherty,et al.  Performance of feature-selection methods in the classification of high-dimension data , 2009, Pattern Recognit..

[13]  Bruce D. Phelps A Comprehensive Look at the Empirical Performance of Equity Premium Prediction , 2009 .

[14]  D. Clayton Prediction and Interaction in Complex Disease Genetics: Experience in Type 1 Diabetes , 2009, PLoS genetics.

[15]  Herman Chernoff,et al.  Discovering influential variables: A method of partitions , 2009, 1009.5744.

[16]  Kristin M. Bakke,et al.  The perils of policy by p-value: Predicting civil conflicts , 2010 .

[17]  Daniel Gianola,et al.  Predicting genetic predisposition in humans: the promise of whole-genome markers , 2010, Nature Reviews Genetics.

[18]  Verónica Bolón-Canedo,et al.  A review of feature selection methods on synthetic data , 2013, Knowledge and Information Systems.

[19]  Tian Zheng,et al.  Interaction-based feature selection and classification for high-dimensional biological data , 2012, Bioinform..

[20]  Predicting the influence of common variants , 2013, Nature Genetics.

[21]  P. Almgren,et al.  Chromosome 9p21 genetic variation explains 13% of cardiovascular disease incidence but does not improve risk prediction , 2013, Journal of internal medicine.