Active Regression by Stratification

We propose a new active learning algorithm for parametric linear regression with random design. We provide finite sample convergence guarantees for general distributions in the misspecified model. This is the first active learner for this setting that provably can improve over passive learning. Unlike other learning settings (such as classification), in regression the passive learning rate of O(1/∊) cannot in general be improved upon. Nonetheless, the so-called 'constant' in the rate of convergence, which is characterized by a distribution-dependent risk, can be improved in many cases. For a given distribution, achieving the optimal risk requires prior knowledge of the distribution. Following the stratification technique advocated in Monte-Carlo function integration, our active learner approaches the optimal risk using piecewise constant approximations.

[1]  G. Casella,et al.  Estimating a Bounded Normal Mean , 1981 .

[2]  David A. Cohn,et al.  Active Learning with Statistical Models , 1996, NIPS.

[3]  D. Wiens.  Minimax Robust Designs and Weights for Approximately Specified Regression Models With Heteroscedastic Errors , 1998 .

[4]  D. Wiens.  Robust weights and designs for biased regression models: Least squares and generalized M-estimation , 2000 .

[5]  T. Kanamori.  Statistical Asymptotic Theory of Active Learning , 2002 .

[6]  Adam Krzyzak,et al.  A Distribution-Free Theory of Nonparametric Regression , 2002, Springer series in statistics.

[7]  Paul Glasserman,et al.  Monte Carlo Methods in Financial Engineering , 2003 .

[8]  Hidetoshi Shimodaira,et al.  Active learning algorithm using the maximum weighted log-likelihood estimator , 2003 .

[9]  David A. Cohn,et al.  Improving generalization with active learning , 1994, Machine Learning.

[10]  Masashi Sugiyama,et al.  Active Learning in Approximately Linear Regression Based on Conditional Expectation of Generalization Error , 2006, J. Mach. Learn. Res..

[11]  Sanjoy Dasgupta,et al.  A General Agnostic Active Learning Algorithm , 2007, ISAIM.

[12]  Ross D. King,et al.  Active Learning for Regression Based on Query by Committee , 2007, IDEAL.

[13]  S. Efromovich.  Sequential Design and Estimation in Heteroscedastic Nonparametric Regression , 2007 .

[14]  Shinichi Nakajima,et al.  Pool-based active learning in approximate linear regression , 2009, Machine Learning.

[15]  John Langford,et al.  Agnostic active learning , 2006, J. Comput. Syst. Sci..

[16]  Alexander G. Gray,et al.  UPAL: Unbiased Pool Based Active Learning , 2011, AISTATS.

[17]  Rémi Munos,et al.  Minimax Number of Strata for Online Stratified Sampling Given Noisy Samples , 2012, ALT.

[18]  Sham M. Kakade,et al.  Random Design Analysis of Ridge Regression , 2012, COLT.

[19]  Jun Zhou,et al.  Maximizing Expected Model Change for Active Learning in Regression , 2013, 2013 IEEE 13th International Conference on Data Mining.

[20]  Deanna Needell,et al.  Stochastic gradient descent and the randomized Kaczmarz algorithm , 2013, ArXiv.

[21]  Deanna Needell,et al.  Stochastic gradient descent, weighted sampling, and the randomized Kaczmarz algorithm , 2013, NIPS.

[22]  Daniel J. Hsu,et al.  Heavy-tailed regression with a generalized median-of-means , 2014, ICML.