Improvements on Cross-Validation: The 632+ Bootstrap Method

Abstract A training set of data has been used to construct a rule for predicting future responses. What is the error rate of this rule? This is an important question both for comparing models and for assessing a final selected model. The traditional answer to this question is given by cross-validation. The cross-validation estimate of prediction error is nearly unbiased but can be highly variable. Here we discuss bootstrap estimates of prediction error, which can be thought of as smoothed versions of cross-validation. We show that a particular bootstrap method, the .632+ rule, substantially outperforms cross-validation in a catalog of 24 simulation experiments. Besides providing point estimates, we also consider estimating the variability of an error rate estimate. All of the results here are nonparametric and apply to any possible prediction rule; however, we study only classification problems with 0–1 loss in detail. Our simulations include “smooth” prediction rules like Fisher's linear discriminant fun...

[1]  S. M. Perlmutter,et al.  Training sequence size and vector quantizer performance , 1991, [1991] Conference Record of the Twenty-Fifth Asilomar Conference on Signals, Systems & Computers.

[2]  M. Stone.  An Asymptotic Equivalence of Choice of Model by Cross‐Validation and Akaike's Criterion , 1977 .

[3]  G. McLachlan.  Discriminant Analysis and Statistical Pattern Recognition , 1992 .

[4]  B. Efron.  Bootstrap Methods: Another Look at the Jackknife , 1979 .

[5]  Ping Zhang.  Model Selection Via Multifold Cross Validation , 1993 .

[6]  M. R. Chernick,et al.  Application of bootstrap and other resampling techniques: Evaluation of classifier performance , 1985, Pattern Recognit. Lett..

[7]  C. L. Mallows.  Some comments on C_p , 1973 .

[8]  M. Stone.  Cross‐Validatory Choice and Assessment of Statistical Predictions , 1976 .

[9]  Leo Breiman,et al.  Classification and Regression Trees , 1984 .

[10]  Anil K. Jain,et al.  Bootstrap Techniques for Error Estimation , 1987, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  R. Tibshirani,et al.  An introduction to the bootstrap , 1993 .

[12]  B. Efron.  How Biased is the Apparent Error Rate of a Prediction Rule , 1986 .

[13]  Jerome H. Friedman,et al.  Flexible Metric Nearest Neighbor Classification , 1994 .

[14]  P. Zhang,et al.  Assessing prediction error in non-parametric regression , 1995 .

[15]  David M. Allen,et al.  The Relationship Between Variable Selection and Data Agumentation and a Method for Prediction , 1974 .

[16]  L. Breiman,et al.  Submodel selection and evaluation in regression. The X-random case , 1992 .

[17]  M. R. Chernick,et al.  Correction note to 'application of bootstrap and other resampling techniques: Evaluation of classifier performance' , 1986, Pattern Recognit. Lett..

[18]  B. Efron,et al.  A Leisurely Look at the Bootstrap, the Jackknife, and , 1983 .

[19]  B. Efron.  Estimating the Error Rate of a Prediction Rule: Improvement on Cross-Validation , 1983 .

[20]  Seymour Geisser,et al.  The Predictive Sample Reuse Method with Applications , 1975 .

[21]  Edna Schechtman,et al.  Efficient bootstrap simulation , 1986 .

[22]  B. Efron.  Jackknife‐After‐Bootstrap Standard Errors and Influence Functions , 1992 .