What If... the Background: Dclns (deep Convolutional Learning Networks) Are Doing Very Well

Over the last 3 years and increasingly so in the last few months, I have seen supervised DCLNs — feedforward and recurrent — do more and more of everything quite well. They seem to learn good representations for a growing number of speech and text problems (for a review by the pioneers in the field see LeCun, Bengio, Hinton, 2015). More interestingly, it is increasingly clear, as I will discuss later, that instead of being trained on millions of labeled examples they can be trained in implicitly supervised ways. This breakthrough in machine learning triggers a few dreams. What if we have now the basic answer to how to develop brain-like intelligence and its basic building blocks? Why it may be true There are several reasons to have been skeptical of neural networks old claims. But I think that I see now possible answers to all of them. I list the corresponding questions here, together with answers (which are, in part, conjectures) ranked in terms of increasing (personal) interest. • What is the tradeoff of nature vs. nurture (for neural networks)? I think that a version of the Baldwin (1896) effect (rediscovered by Hinton and Nowlan, 1987) provides a good framework for an answer. If an organism inherits the machinery that can learn a task (important for survival) from examples provided by the environment, then evolution only needs to discover the machinery and compile it into the genes. It does not need to discover and compile the full, detailed solution to the task — and never will, because of the lack of sufficient evolutionary pressure. This argument suggests that evolution determines the architecture of the network, for instance it may determine slightly different deep learning architectures for different sensory tasks with respect to number of layers, connectivity and pooling parameters — say in visual cortex vs. auditory cortex. Thus the tradeoff between nature and nurture, which applies

[1]  R. S-A. Gatsaeva,et al.  On the representation of continuous functions of several variables as superpositions of continuous functions of one variable and addition , 2018 .

[2]  Lorenzo Rosasco,et al.  Deep Convolutional Networks are Hierarchical Kernel Machines , 2015, ArXiv.

[3]  Nitish Srivastava Unsupervised Learning of Visual Representations using Videos , 2015 .

[4]  Shai Ben-David,et al.  Understanding Machine Learning - From Theory to Algorithms , 2014 .

[5]  T. Poggio THE COMPUTATIONAL MAGIC OF THE VENTRAL STREAM: TOWARDS A THEORY , 2011 .

[6]  Thomas Serre,et al.  A quantitative theory of immediate visual recognition. , 2007, Progress in brain research.

[7]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[8]  T. Poggio,et al.  The Mathematics of Learning: Dealing with Data , 2005, 2005 International Conference on Neural Networks and Brain.

[9]  T. Poggio,et al.  On the representation of multi-input systems: Computational properties of polynomial algorithms , 1980, Biological Cybernetics.

[10]  Tomaso Poggio,et al.  Incorporating prior information in machine learning by creating virtual examples , 1998, Proc. IEEE.

[11]  Tomaso Poggio,et al.  Image Representations for Visual Learning , 1996, Science.

[12]  T Poggio,et al.  Fast perceptual learning in visual hyperacuity. , 1991, Science.

[13]  T. Poggio A theory of how the brain might work. , 1990, Cold Spring Harbor symposia on quantitative biology.

[14]  Geoffrey E. Hinton,et al.  How Learning Can Guide Evolution , 1996, Complex Syst..

[15]  J. Baldwin A New Factor in Evolution , 1896, The American Naturalist.