We consider the online binary classification problem, where we are given m classifiers. At each stage, the classifiers map the input to the probability that the input belongs to the positive class. An online classification meta-algorithm is an algorithm that combines the outputs of the classifiers in order to attain a certain goal, without having prior knowledge on the form and statistics of the input, and without prior knowledge on the performance of the given classifiers. In this paper, we use sensitivity and specificity as the performance metrics of the meta-algorithm. In particular, our goal is to design an algorithm that satisfies the following two properties (asymptotically): (i) its average false positive rate (fp-rate) is under some given threshold; and (ii) its average true positive rate (tp-rate) is not worse than the tp-rate of the best convex combination of the m given classifiers that satisfies fp-rate constraint, in hindsight. We show that this problem is in fact a special case of the regret minimization problem with constraints, and therefore the above goal is not attainable. Hence, we pose a relaxed goal and propose a corresponding practical online learning meta-algorithm that attains it. In the case of two classifiers, we show that this algorithm takes a very simple form. To our best knowledge, this is the first algorithm that addresses the problem of the average tp-rate maximization under average fp-rate constraints in the online setting.
[1]
D. Blackwell.
Controlled Random Walks
,
2010
.
[2]
Tom Fawcett,et al.
An introduction to ROC analysis
,
2006,
Pattern Recognit. Lett..
[3]
Gábor Lugosi,et al.
Prediction, learning, and games
,
2006
.
[4]
Martin Zinkevich,et al.
Online Convex Programming and Generalized Infinitesimal Gradient Ascent
,
2003,
ICML.
[5]
Philip Wolfe,et al.
Contributions to the theory of games
,
1953
.
[6]
D. Blackwell.
An analog of the minimax theorem for vector payoffs.
,
1956
.
[7]
Y. Freund,et al.
Adaptive game playing using multiplicative weights
,
1999
.
[8]
John N. Tsitsiklis,et al.
Online Learning with Sample Path Constraints
,
2009,
J. Mach. Learn. Res..
[9]
Nahum Shimkin,et al.
Stochastic Games with Average Cost Constraints
,
1994
.
[10]
James Hannan,et al.
4. APPROXIMATION TO RAYES RISK IN REPEATED PLAY
,
1958
.
[11]
Koby Crammer,et al.
Online Passive-Aggressive Algorithms
,
2003,
J. Mach. Learn. Res..
[12]
Manfred K. Warmuth,et al.
The weighted majority algorithm
,
1989,
30th Annual Symposium on Foundations of Computer Science.
[13]
Elad Hazan,et al.
Logarithmic regret algorithms for online convex optimization
,
2006,
Machine Learning.
[14]
Yoram Singer,et al.
Online Classification for Complex Problems Using Simultaneous Projections
,
2006,
NIPS.