A Nonparametric Sequential Test for Online Randomized Experiments

We propose a nonparametric sequential test that aims to address two practical problems pertinent to online randomized experiments: (i) how to do a hypothesis test for complex metrics; (ii) how to prevent type 1 error inflation under continuous monitoring. The proposed test does not require knowledge of the underlying probability distribution generating the data. We use the bootstrap to estimate the likelihood for blocks of data followed by mixture sequential probability ratio test. We validate this procedure on data from a major online e-commerce website. We show that the proposed test controls type 1 error at any time, has good power, is robust to misspecification in the distribution generating the data, and allows quick inference in online randomized experiments.

[1]  Martin Goodson.  MOST WINNING A / B TEST RESULTS ARE ILLUSORY , 2014 .

[2]  E. Mammen.  The Bootstrap and Edgeworth Expansion , 1997 .

[3]  T. Lai.  SEQUENTIAL ANALYSIS: SOME CLASSICAL PROBLEMS AND NEW CHALLENGES , 2001 .

[4]  References , 1971 .

[5]  Ron Kohavi,et al.  Seven pitfalls to avoid when running controlled experiments on the web , 2009, KDD.

[6]  E. Lehmann.  Elements of large-sample theory , 1998 .

[7]  Ron Kohavi,et al.  Controlled experiments on the web: survey and practical guide , 2009, Data Mining and Knowledge Discovery.

[8]  Moshe Pollak,et al.  Optimality and Almost Optimality of Mixture Stopping Rules , 1978 .

[9]  M. Kenward,et al.  An Introduction to the Bootstrap , 2007 .

[10]  Leif D. Nelson,et al.  False-Positive Psychology , 2011, Psychological science.

[11]  S. Pocock.  Group sequential methods in the design and analysis of clinical trials , 1977 .

[12]  Peter Hall,et al.  On the bootstrap and likelihood-based confidence regions , 1987 .

[13]  H. Robbins.  Statistical Methods Related to the Law of the Iterated Logarithm , 1970 .

[14]  Geert M. P. van Kempen,et al.  Mean and variance of ratio estimators used in fluorescence ratio imaging. , 2000 .

[15]  R. Khan,et al.  Sequential Tests of Statistical Hypotheses. , 1972 .

[16]  Craig MacDonald,et al.  Sequential Testing for Early Stopping of Online Experiments , 2015, SIGIR.

[17]  L. Pekelis,et al.  Always Valid Inference: Bringing Sequential Analysis to A/B Testing , 2015, 1512.04922.

[18]  M. Kulldorff,et al.  A Maximized Sequential Probability Ratio Test for Drug and Vaccine Safety Surveillance , 2011 .

[19]  L. Pekelis,et al.  The New Stats Engine , 2015 .

[20]  B. Efron.  Bayes and likelihood calculations from confidence intervals , 1993 .