Local Rademacher complexities and oracle inequalities in risk minimization

where X1, . . . ,Xn are drawn i.i.d. from a probability measure P on X and F is a class of real-valued functions defined on X. The study of bounds on the expectation P f̂ arises in many applied areas, including the analysis of randomized optimization methods involving Monte Carlo estimates of integrals. Motivated by prediction problems that arise in machine learning and nonparametric statistics, the paper makes an important contribution to the study of these bounds, and to the development of model selection methods that exploit the bounds. The broad approach taken in this paper, and in much earlier work, is to show that the empirical structure (i.e., the collection of sample averages, Pnf ) is close to the real structure (i.e., the collection of expectations, Pf ). If they are close in the additive sense that ‖Pn − P‖F decreases at some rate, then it is clear that P f̂ approaches inff ∈F Pf at that rate. As the paper recalls, there is a tight relationship between the Rademacher process indexed by coordinate projections of the class F and this additive notion of closeness of empirical and real structures. Also, it can be advantageous to consider these properties only locally, that is, in the set F(δ) ⊂ F of near-minimizers of Pf . In particular, if the variance of elements of F(δ) goes to zero with δ, then faster rates are possible through the study of these local properties. An alternative, developed in the paper, is closeness in the multiplicative sense that for 0 < ε < 1, for all functions f in F that have expectations not too small,