A unified treatment of multiple testing with prior knowledge using the p-filter

A significant literature studies ways of employing prior knowledge to improve power and precision of multiple testing procedures. Some common forms of prior knowledge may include (a) \emph{a priori} beliefs about which hypotheses are null, modeled by non-uniform prior weights; (b) differing importances of hypotheses, modeled by differing penalties for false discoveries; (c) multiple arbitrary partitions of the hypotheses into known (possibly overlapping) groups, indicating (dis)similarity of hypotheses; and (d) knowledge of independence, positive dependence or arbitrary dependence between hypotheses or groups, allowing for more aggressive or conservative procedures. We unify a number of existing procedures, generalize the conditions under which they are known to work, and simplify their proofs of FDR control. Then, we present a unified algorithmic framework for global null testing and false discovery rate (FDR) control that allows the scientist to incorporate all four types of prior knowledge (a)--(d) simultaneously, recovering a wide variety of common algorithms as special cases.

[1]  E. Suchman,et al.  The American soldier: Adjustment during army life. (Studies in social psychology in World War II), Vol. 1 , 1949 .

[2]  E. Lehmann Some Concepts of Dependence , 1966 .

[3]  P. Seeger A Note on a Method for the Analysis of Significances en masse , 1968 .

[4]  K. Gabriel,et al.  SIMULTANEOUS TEST PROCEDURES-SOME THEORY OF MULTIPLE COMPARISONS' , 1969 .

[5]  S. Karlin,et al.  Classes of orderings of measures and related correlation inequalities. I. Multivariate totally positive distributions , 1980 .

[6]  G. Hommel Tests of the overall hypothesis for arbitrary dependence structures , 1983 .

[7]  R. Simes,et al.  An improved Bonferroni procedure for multiple tests of significance , 1986 .

[8]  E. Sonnemann,et al.  Vollständigkeitssätze für Multiple Testprobleme , 1988 .

[9]  Y. Benjamini,et al.  More powerful procedures for multiple significance testing. , 1990, Statistics in medicine.

[10]  Henry Braun,et al.  The Collected Works of John W. Tukey, Volume VIII: Multiple Comparisons, 1948-1983 , 1994 .

[11]  An extended Simes' test , 1994 .

[12]  Y. Benjamini,et al.  Controlling the false discovery rate: a practical and powerful approach to multiple testing , 1995 .

[13]  Y. Benjamini,et al.  Multiple Hypotheses Testing with Weights , 1997 .

[14]  Y. Benjamini,et al.  A step-down multiple hypotheses testing procedure that controls the false discovery rate under independence , 1999 .

[15]  Y. Benjamini,et al.  On the Adaptive Control of the False Discovery Rate in Multiple Testing With Independent Statistics , 2000 .

[16]  Y. Benjamini,et al.  THE CONTROL OF THE FALSE DISCOVERY RATE IN MULTIPLE TESTING UNDER DEPENDENCY , 2001 .

[17]  John D. Storey A direct approach to false discovery rates , 2002 .

[18]  John D. Storey,et al.  Strong control, conservative point estimation and simultaneous conservative consistency of false discovery rates: a unified approach , 2004 .

[19]  Y. Benjamini,et al.  Adaptive linear step-up procedures that control the false discovery rate , 2006 .

[20]  Azeem M. Shaikh,et al.  On stepdown control of the false discovery proportion , 2006 .

[21]  L. Wasserman,et al.  False discovery control with p-value weighting , 2006 .

[22]  G. Blanchard,et al.  Two simple sufficient conditions for FDR control , 2008, 0802.1406.

[23]  S. Sarkar Two-stage stepup procedures controlling FDR , 2008 .

[24]  Preface to the Translation of Eckart Sonnemann's 1982 Paper on General Solutions to Multiple Testing Problems , 2008 .

[25]  D. Yekutieli Hierarchical False Discovery Rate–Controlling Methodology , 2008 .

[26]  S. Sarkar On Methods Controlling the False Discovery Rate 1 , 2009 .

[27]  Y. Benjamini,et al.  An adaptive step-down procedure with proven FDR control under independence , 2009, 0903.5373.

[28]  Azeem M. Shaikh,et al.  Consonance and the closure method in multiple testing , 2009 .

[29]  Gilles Blanchard,et al.  Adaptive False Discovery Rate Control under Independence and Dependence , 2009, J. Mach. Learn. Res..

[30]  Harrison H. Zhou,et al.  False Discovery Rate Control With Groups , 2010, Journal of the American Statistical Association.

[31]  V. Vovk,et al.  Combining P-Values Via Averaging , 2012, Biometrika.

[32]  Yoav Benjamini,et al.  Selective inference on multiple families of hypotheses , 2014 .

[33]  Christine B. Peterson,et al.  Many Phenotypes Without Many False Discoveries: Error Controlling Strategies for Multitrait Association Studies , 2015, Genetic epidemiology.

[34]  Christine B. Peterson,et al.  Controlling the Rate of GWAS False Discoveries , 2016, Genetics.

[35]  Adel Javanmard,et al.  Online Rules for Control of False Discovery Rate and False Discovery Exceedance , 2016, ArXiv.

[36]  Martin J. Wainwright,et al.  DAGGER: A sequential algorithm for FDR control on DAGs , 2017, ArXiv.

[37]  Patrick Rubin-Delanchy,et al.  Choosing between methods of combining p-values , 2017, 1707.06897.

[38]  Yoav Benjamini,et al.  Testing hypotheses on a tree: new error rates and controlling strategies , 2017, 1705.07529.

[39]  Martin J. Wainwright,et al.  QuTE: Decentralized multiple testing on sensor networks with false discovery rate control , 2017, 2017 IEEE 56th Annual Conference on Decision and Control (CDC).

[40]  Martin J. Wainwright,et al.  Online control of the false discovery rate with decaying memory , 2017, NIPS.

[41]  Aaditya Ramdas,et al.  The p‐filter: multilayer false discovery rate control for grouped hypotheses , 2017 .

[42]  Aaditya Ramdas,et al.  Towards "simultaneous selective inference": post-hoc bounds on the false discovery proportion , 2018, 1803.06790.

[43]  Martin J. Wainwright,et al.  SAFFRON: an adaptive algorithm for online control of the false discovery rate , 2018, ICML.

[44]  Controlling FDR while highlighting distinct discoveries , 2018 .

[45]  Chiara Sabatti,et al.  MULTILAYER KNOCKOFF FILTER: CONTROLLED VARIABLE SELECTION AT MULTIPLE RESOLUTIONS. , 2017, The annals of applied statistics.

[46]  Michael I. Jordan,et al.  A sequential algorithm for false discovery rate control on directed acyclic graphs , 2019, Biometrika.