Bounding the Vapnik-Chervonenkis Dimension of Concept Classes Parameterized by Real Numbers

The Vapnik-Chervonenkis (V-C) dimension is an important combinatorial tool in the analysis of learning problems in the PAC framework. For polynomial learnability, we seek upper bounds on the V-C dimension that are polynomial in the syntactic complexity of concepts. Such upper bounds are automatic for discrete concept classes, but hitherto little has been known about what general conditions guarantee polynomial bounds on V-C dimension for classes in which concepts and examples are represented by tuples of real numbers. In this paper, we show that for two general kinds of concept class the V-C dimension is polynomially bounded in the number of real numbers used to define a problem instance. One is classes where the criterion for membership of an instance in a concept can be expressed as a formula (in the first-order theory of the reals) with fixed quantification depth and exponentially-bounded length, whose atomic predicates are polynomial inequalities of exponentially-bounded degree, The other is classes where containment of an instance in a concept is testable in polynomial time, assuming we may compute standard arithmetic operations on reals exactly in constant time. Our results show that in the continuous case, as in the discrete, the real barrier to efficient learning in the Occam sense is complexity-theoretic and not information-theoretic. We present examples to show how these results apply to concept classes defined by geometrical figures and neural nets, and derive polynomial bounds on the V-C dimension for these classes.

[1]  J. Milnor On the Betti numbers of real varieties , 1964 .

[2]  H. Warren Lower bounds for approximation by nonlinear manifolds , 1968 .

[3]  Vladimir Vapnik,et al.  Chervonenkis: On the uniform convergence of relative frequencies of events to their probabilities , 1971 .

[4]  R. Dudley Central Limit Theorems for Empirical Measures , 1978 .

[5]  Richard M. Dudley,et al.  Some special vapnik-chervonenkis classes , 1981, Discret. Math..

[6]  J. Michael Steele,et al.  Lower Bounds for Algebraic Decision Trees , 1982, J. Algorithms.

[7]  Michael Ben-Or,et al.  Lower bounds for algebraic computation trees , 1983, STOC.

[8]  Mikhail J. Atallah,et al.  A Linear Time Algorithm for the Hausdorff Distance Between Convex Polygons , 1983, Inf. Process. Lett..

[9]  Leslie G. Valiant,et al.  A theory of the learnable , 1984, STOC '84.

[10]  David Haussler,et al.  Occam's Razor , 1987, Inf. Process. Lett..

[11]  David Haussler,et al.  Predicting (0, 1)-functions on randomly drawn points , 1988, [Proceedings 1988] 29th Annual Symposium on Foundations of Computer Science.

[12]  Nathan Linial,et al.  Results on learnability and the Vapnik-Chervonenkis dimension , 1988, [Proceedings 1988] 29th Annual Symposium on Foundations of Computer Science.

[13]  Leslie G. Valiant,et al.  A general lower bound on the number of examples needed for learning , 1988, COLT '88.

[14]  David Haussler,et al.  Predicting {0,1}-functions on randomly drawn points , 1988, COLT '88.

[15]  Nathan Linial,et al.  Results on learnability and the Vapnick-Chervonenkis dimension , 1988, COLT '88.

[16]  David Haussler,et al.  What Size Net Gives Valid Generalization? , 1989, Neural Computation.

[17]  J. Yukich,et al.  Some new Vapnik-Chervonenkis classes , 1989 .

[18]  David Haussler,et al.  Learnability and the Vapnik-Chervonenkis dimension , 1989, JACM.

[19]  Helmut Alt,et al.  Approximate matching of polygonal shapes , 1995, SCG '91.

[20]  Helmut Alt,et al.  Approximate Matching of Polygonal Shapes (Extended Abstract) , 1991, SCG.

[21]  Balas K. Natarajan,et al.  Machine Learning: A Theoretical Approach , 1992 .

[22]  Martin Anthony,et al.  Computational learning theory: an introduction , 1992 .

[23]  Michael C. Laskowski,et al.  Vapnik-Chervonenkis classes of definable sets , 1992 .

[24]  Paul W. Goldberg,et al.  PAC-learning geometrical figures , 1992 .

[25]  Eduardo D. Sontag,et al.  Feedforward Nets for Interpolation and Classification , 1992, J. Comput. Syst. Sci..

[26]  James Renegar,et al.  On the Computational Complexity and Geometry of the First-Order Theory of the Reals, Part I: Introduction. Preliminaries. The Geometry of Semi-Algebraic Sets. The Decision Problem for the Existential Theory of the Reals , 1992, J. Symb. Comput..

[27]  Shai Ben-David,et al.  Localization vs. Identification of Semi-Algebraic Sets , 1993, COLT.

[28]  Wolfgang Maass,et al.  Bounds for the computational power and learning complexity of analog neural nets , 1993, SIAM J. Comput..

[29]  Leslie G. Valiant,et al.  A View of Computational Learning Theory , 1993 .

[30]  Eduardo D. Sontag,et al.  Finiteness results for sigmoidal “neural” networks , 1993, STOC.