An Upper Bound of the Bias of Nadaraya-Watson Kernel Regression under Lipschitz Assumptions

The Nadaraya-Watson kernel estimator is among the most popular nonparameteric regression technique thanks to its simplicity. Its asymptotic bias has been studied by Rosenblatt in 1969 and has been reported in several related literature. However, given its asymptotic nature, it gives no access to a hard bound. The increasing popularity of predictive tools for automated decision-making surges the need for hard (non-probabilistic) guarantees. To alleviate this issue, we propose an upper bound of the bias which holds for finite bandwidths using Lipschitz assumptions and mitigating some of the prerequisites of Rosenblatt’s analysis. Our bound has potential applications in fields like surgical robots or self-driving cars, where some hard guarantees on the prediction-error are needed.

[1]  M. Rosenblatt.  Remarks on Some Nonparametric Estimates of a Density Function , 1956 .

[2]  E. Parzen.  On Estimation of a Probability Density Function and Mode , 1962 .

[3]  E. Nadaraya.  On Estimating Regression , 1964 .

[4]  G. S. Watson,et al.  Smooth regression analysis , 1964 .

[5]  W. Härdle,et al.  Asymptotic nonequivalence of some bandwidth selectors in nonparametric regression , 1985 .

[6]  H. Müller,et al.  Convolution type estimators for nonparametric regression , 1988 .

[7]  T. Gasser,et al.  Choice of bandwidth for kernel regression when residuals are correlated , 1992 .

[8]  Jianqing Fan,et al.  Variable Bandwidth and Local Linear Regression Smoothers , 1992 .

[9]  Jianqing Fan.  Design-adaptive Nonparametric Regression , 1992 .

[10]  Ravi Bansal,et al.  Nonparametric estimation of structural models for high-frequency currency market data , 1995 .

[11]  B. P. Zhang,et al.  Estimation of the Lipschitz constant of a function , 1996, J. Glob. Optim..

[12]  B. Ray,et al.  Bandwidth selection for kernel regression with long-range dependent errors , 1997 .

[13]  Adam Krzyzak,et al.  A Distribution-Free Theory of Nonparametric Regression , 2002, Springer series in statistics.

[14]  Pierre Geurts,et al.  Tree-Based Batch Mode Reinforcement Learning , 2005, J. Mach. Learn. Res..

[15]  David J. Fleet,et al.  Gaussian Process Dynamical Models , 2005, NIPS.

[16]  L. Wasserman.  All of Nonparametric Statistics , 2005 .

[17]  Jan Peters,et al.  Using model knowledge for learning inverse dynamics , 2010, 2010 IEEE International Conference on Robotics and Automation.

[18]  Oliver Kroemer,et al.  A Non-Parametric Approach to Dynamic Programming , 2011, NIPS.

[19]  Carl E. Rasmussen,et al.  PILCO: A Model-Based and Data-Efficient Approach to Policy Search , 2011, ICML.

[20]  Oliver Kroemer,et al.  A kernel-based approach to direct action perception , 2012, 2012 IEEE International Conference on Robotics and Automation.

[21]  Anja Schindler,et al.  A Review and Comparison of Bandwidth Selection Methods for Kernel Regression , 2014 .

[22]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[23]  Seyed-Mohsen Moosavi-Dezfooli,et al.  DeepFool: A Simple and Accurate Method to Fool Deep Neural Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Wojciech Zaremba,et al.  OpenAI Gym , 2016, ArXiv.

[25]  Moses Charikar,et al.  Hashing-Based-Estimators for Kernel Density in High Dimensions , 2017, 2017 IEEE 58th Annual Symposium on Foundations of Computer Science (FOCS).

[26]  Dirk Ormoneit,et al.  Kernel-Based Reinforcement Learning , 2017, Encyclopedia of Machine Learning and Data Mining.

[27]  Piotr Indyk,et al.  Space and Time Efficient Kernel Density Estimation in High Dimensions , 2019, NeurIPS.

[28]  Jan Peters,et al.  A Nonparametric Off-Policy Policy Gradient , 2020, AISTATS.