A simple algorithm that discovers efficient perceptual codes

We describe the \wake-sleep" algorithm that allows a multilayer, unsupervised, neural network to build a hierarchy of representations of sensory input. The network has bottom-up \recognition" connections that are used to convert sensory input into underlying representations. Unlike most arti cial neural networks, it also has top-down \generative" connections that can be used to reconstruct the sensory input from the representations. In the \wake" phase of the learning algorithm, the network is driven by the bottom-up recognition connections and the top-down generative connections are trained to be better at reconstructing the sensory input from the representation chosen by the recognition process. In the \sleep" phase, the network is driven top-down by the generative connections to produce a fantasized representation and a fantasized sensory input. The recognition connections are then trained to be better at recovering the fantasized representation from the fantasized sensory input. In both phases, the synaptic learning rule is simple and local. The combined e ect of the two phases is to create representations of the sensory input that are e cient in the following sense: On average, it takes more bits to describe each sensory input vector directly than to rst describe the representation of the sensory input chosen by the recognition process and then describe the di erence between the sensory input and its reconstruction from the chosen representation.

[1]  M. V. Rossum,et al.  In Neural Computation , 2022 .

[2]  P. Fldik Forming sparse representations by local anti-Hebbian learning , 1990 .

[3]  Geoffrey E. Hinton,et al.  Autoencoders, Minimum Description Length and Helmholtz Free Energy , 1993, NIPS.

[4]  Geoffrey E. Hinton,et al.  The Helmholtz Machine , 1995, Neural Computation.

[5]  R. Zemel,et al.  Learning sparse multiple cause models , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[6]  Geoffrey E. Hinton,et al.  The "wake-sleep" algorithm for unsupervised neural networks. , 1995, Science.

[7]  S. Nelson,et al.  An emergent model of orientation selectivity in cat visual cortical simple cells , 1995, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[8]  Brendan J. Frey,et al.  Does the Wake-sleep Algorithm Produce Good Density Estimators? , 1995, NIPS.

[9]  Eric Saund,et al.  A Multiple Cause Mixture Model for Unsupervised Learning , 1995, Neural Computation.

[10]  Geoffrey E. Hinton,et al.  Varieties of Helmholtz Machine , 1996, Neural Networks.

[11]  Brendan J. Frey,et al.  Free energy coding , 1996, Proceedings of Data Compression Conference - DCC '96.

[12]  Jorma Rissanen,et al.  Stochastic Complexity in Statistical Inquiry , 1989, World Scientific Series in Computer Science.