Mathematical Programming in Machine Learning and Data Mining

The field of Machine Learning (ML) and Data Mining (DM) is focused around the following problem: Given a data domain D we want to approximate an unknown function y(x) on the given data set X ⊂ D (for which the values of y(x) may or may not be known) by a function f from a given class F so that the approximation generalizes in the best possible way on all of the (unseen) data x ∈ D. The approximating function f might take real values, as in the case of regression; binary values, as in the case of classification; or integer values, as in some cases of ranking; or this function might be a mapping between ordered subsets of data points and ordered subsets of real, integer or binary values, as in the case of structured object prediction. The quality of approximation by f can be measured by various objective functions. For instance in the case of support vector machine (SVM)[4] classification the quality of the approximating function is estimated by a weighted sum of a regularization term h(f) and the hinge loss term ∑ x∈X max{1−y(x)f(x), 0}. Hence, many of the machine learning problems can be posed as an optimization problem where optimization is performed over a given class F for a chosen objective. The connection between optimization and machine learning (although always present) became especially evident with the popularity of the SVMs [4], [24], and the kernel methods in general [18]. SVM classification problem is formulated as a convex quadratic program.