Assessment of heterogeneous treatment effect estimation accuracy via matching

We study the assessment of the accuracy of heterogeneous treatment effect (HTE) estimation, where the HTE is not directly observable so standard computation of prediction errors is not applicable. To tackle the difficulty, we propose an assessment approach by constructing pseudo-observations of the HTE based on matching. Our contributions are three-fold: first, we introduce a novel matching distance derived from proximity scores in random forests; second, we formulate the matching problem as an average minimum-cost flow problem and provide an efficient algorithm; third, we propose a match-then-split principle for the assessment with cross-validation. We demonstrate the efficacy of the assessment approach on synthetic data and data generated from a real dataset.

[1]  D. Green,et al.  Modeling Heterogeneous Treatment Effects in Survey Experiments with Bayesian Additive Regression Trees , 2012 .

[2]  Vivian C. Wong,et al.  Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within‐study comparisons , 2008 .

[3]  James Bennett,et al.  The Netflix Prize , 2007 .

[4]  Yen-Liang Chen,et al.  The minimal average cost flow problem , 1995 .

[5]  Trevor Hastie,et al.  Some methods for heterogeneous treatment effect estimation in high dimensions , 2017, Statistics in medicine.

[6]  D. Rubin,et al.  Causal Inference for Statistics, Social, and Biomedical Sciences: A General Method for Estimating Sampling Variances for Standard Estimators for Average Causal Effects , 2015 .

[7]  D. Rubin,et al.  Reducing Bias in Observational Studies Using Subclassification on the Propensity Score , 1984 .

[8]  Janet S Twyman,et al.  Handbook on Personalized Learning for States, Districts, and Schools. , 2016 .

[9]  T. Speed,et al.  On the Application of Probability Theory to Agricultural Experiments. Essay on Principles. Section 9 , 1990 .

[10]  Oswald Veblen,et al.  On Matrices Whose Elements Are Integers , 1921 .

[11]  Alexander Schrijver,et al.  Theory of linear and integer programming , 1986, Wiley-Interscience series in discrete mathematics and optimization.

[12]  D. Rubin,et al.  The central role of the propensity score in observational studies for causal effects , 1983 .

[13]  L. Lesko,et al.  Personalized Medicine: Elusive Dream or Imminent Reality? , 2007, Clinical pharmacology and therapeutics.

[14]  Yen S. Low,et al.  Comparing high-dimensional confounder control methods for rapid cohort studies from electronic health records , 2015, Journal of comparative effectiveness research.

[15]  Trevor Hastie,et al.  Synth-Validation: Selecting the Best Causal Inference Method for a Given Dataset , 2017, 1711.00083.

[16]  Stefan Wager,et al.  Estimation and Inference of Heterogeneous Treatment Effects using Random Forests , 2015, Journal of the American Statistical Association.

[17]  Donald B. Rubin,et al.  MULTIVARIATE MATCHING METHODS THAT ARE EQUAL PERCENT BIAS REDUCING, I: SOME EXAMPLES , 1974 .

[18]  D. Rubin.  [On the Application of Probability Theory to Agricultural Experiments. Essay on Principles. Section 9.] Comment: Neyman (1923) and Causal Inference in Experiments and Observational Studies , 1990 .

[19]  D. Rubin.  Estimating causal effects of treatments in randomized and nonrandomized studies. , 1974 .

[20]  R. Lalonde.  Evaluating the Econometric Evaluations of Training Programs with Experimental Data , 1984 .

[21]  Marc Ratkovic,et al.  Estimating treatment effect heterogeneity in randomized program evaluation , 2013, 1305.5682.

[22]  D. Rubin.  Bias Reduction Using Mahalanobis-Metric Matching , 1980 .

[23]  Paul R. Rosenbaum,et al.  Modern Algorithms for Matching in Observational Studies , 2020, Annual Review of Statistics and Its Application.

[24]  Ashkan Ertefaie,et al.  Variable Selection in Causal Inference using a Simultaneous Penalization Method , 2015, 1511.08501.

[25]  Pieter Abbeel,et al.  Transfer Learning for Estimating Causal Effects using Neural Networks , 2018, ArXiv.

[26]  P. Rosenbaum.  A Characterization of Optimal Designs for Observational Studies , 1991 .

[27]  Susan Athey,et al.  Recursive partitioning for heterogeneous causal effects , 2015, Proceedings of the National Academy of Sciences.

[28]  Jennifer L. Hill,et al.  Bayesian Nonparametric Modeling for Causal Inference , 2011 .

[29]  A. Agresti.  An introduction to categorical data analysis , 1990 .

[30]  B. Hansen.  The prognostic analogue of the propensity score , 2008 .

[31]  P. Rosenbaum.  Design of Observational Studies , 2009, Springer Series in Statistics.

[32]  M. Kendall.  Theoretical Statistics , 1956, Nature.

[33]  Paul R. Rosenbaum,et al.  Optimal Matching for Observational Studies , 1989 .

[34]  Sören R. Künzel,et al.  Metalearners for estimating heterogeneous treatment effects using machine learning , 2017, Proceedings of the National Academy of Sciences.

[35]  Elizabeth A Stuart,et al.  Matching methods for causal inference: A review and a look forward. , 2010, Statistical science : a review journal of the Institute of Mathematical Statistics.

[36]  D. Rubin.  Matched Sampling for Causal Effects: Matching to Remove Bias in Observational Studies , 1973 .

[37]  B. Hansen.  Full Matching in an Observational Study of Coaching for the SAT , 2004 .