Intuitively, we expect that averaging — or bagging — different regressors with low correlation should smooth their behavior and be somewhat similar to regularization. In this note we make this intuition precise. Using an almost classical definition of stability, we prove that a certain form of averaging provides generalization bounds with a rate of convergence of the same order as Tikhonov regularization — similar to fashionable RKHSbased learning algorithms. This report describes research done within the Center for Biological & Computational Learning which is part of the McGovern Institute, the Department of Brain & Cognitive Sciences and the Artificial Intelligence Laboratory at the Massachusetts Institute of Technology. This research was sponsored by grants from: Office of Naval Research (DARPA) under contract No. N00014-001-0907, National Science Foundation (ITR) under contract No. IIS-0085836, National Science Foundation (KDI) under contract No. DMS-9872936, and National Science Foundation under contract No. IIS-9800032 Additional support was provided by: Central Research Institute of Electric Power Industry, Eastman Kodak Company, DaimlerChrysler AG, Compaq, Honda R&D Co., Ltd., Komatsu Ltd., NEC Fund, Siemens Corporate Research, Inc., and The Whitaker Foundation.
[1]
A. N. Tikhonov,et al.
Solutions of ill-posed problems
,
1977
.
[2]
F. Girosi,et al.
Networks for approximation and learning
,
1990,
Proc. IEEE.
[3]
Tomaso A. Poggio,et al.
Extensions of a Theory of Networks for Approximation and Learning
,
1990,
NIPS.
[4]
Spectral properties and knot removal for interpolation by pure radial sums
,
1995
.
[5]
Tomaso A. Poggio,et al.
Regularization Theory and Neural Networks Architectures
,
1995,
Neural Computation.
[6]
Dana Ron,et al.
Algorithmic Stability and Sanity-Check Bounds for Leave-one-Out Cross-Validation
,
1997,
COLT.
[7]
Vladimir Vapnik,et al.
Statistical learning theory
,
1998
.
[8]
Tomaso A. Poggio,et al.
Regularization Networks and Support Vector Machines
,
2000,
Adv. Comput. Math..
[9]
Martin D. Buhmann,et al.
A new class of radial basis functions with compact support
,
2001,
Math. Comput..
[10]
Partha Niyogi,et al.
Almost-everywhere Algorithmic Stability and Generalization Error
,
2002,
UAI.
[11]
André Elisseeff,et al.
Stability and Generalization
,
2002,
J. Mach. Learn. Res..