Estimation of prediction error with known covariate shift

In supervised learning, the estimation of prediction error on unlabeled test data 1 is an important task. Existing methods are usually built on the assumption that 2 the training and test data are sampled from the same distribution, which is often 3 violated in practice. As a result, traditional estimators like cross-validation (CV) 4 will be biased and this may result in poor model selection. In this paper, we 5 assume that we have a test dataset in which the feature values are available but 6 not the outcome labels, and focus on a particular form of distributional shift of 7 covariate shift. We propose an alternative method based on parametric bootstrap of 8 the target of conditional error Err X [2]. Empirically our method outperforms CV 9 for both simulation and real data example across different modeling tasks, and is 10 comparable to state-of-the-art methods for image classification. 11