On Invariance and Selectivity in Representation Learning

AbstractWe discuss data representation which can be learned automaticallyfrom data, are invariant to transformations, and at the same time selec-tive, in the sense that two points have the same representation only if theyare one the transformation of the other. The mathematical results heresharpen some of the key claims of i-theory { a recent theory of feedforwardprocessing in sensory cortex. [3, 4, 5].Keywords:Invariance, Machine LearningThe paper is submitted to Information and Inference Journal. 1 Introduction This paper considers the problem of learning "good" data representation whichcan lower the need of labeled data (sample complexity) in machine learning(ML). Indeed, while current ML systems have achieved impressive results in avariety of tasks, an obvious bottleneck appears to be the huge amount of labeleddata needed. This paper builds on the idea that data representation, which arelearned in an unsupervised manner, can be key to solve the problem. Classicalstatistical learning theory focuses on supervised learning and postulates that asuitable hypothesis space is given. In turn, under very general conditions, thelatter can be seen to be equivalent to a data representation. In other words,data representation and how to select and learn it, is classically not consideredto be part of the learning problem, but rather as a prior information. In practicead hoc solutions are often empirically found for each problem.The study in this paper is a step towards developing a theory of learningdata representation. Our starting point is the intuition that, since many learning