The wake-sleep algorithm (Hinton, Dayan, Frey and Neal 1995) is a relatively efficient method of fitting a multilayer stochastic generative model to high-dimensional data. In addition to the top-down connections in the generative model, it makes use of bottom-up connections for approximating the probability distribution over the hidden units given the data, and it trains these bottom-up connections using a simple delta rule. We use a variety of synthetic and real data sets to compare the performance of the wake-sleep algorithm with Monte Carlo and mean field methods for fitting the same generative model and also compare it with other models that are less powerful but easier to fit.
[1]
D. Rubin,et al.
Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper
,
1977
.
[2]
Radford M. Neal.
Connectionist Learning of Belief Networks
,
1992,
Artif. Intell..
[3]
Geoffrey E. Hinton,et al.
The Helmholtz Machine
,
1995,
Neural Computation.
[4]
Geoffrey E. Hinton,et al.
The "wake-sleep" algorithm for unsupervised neural networks.
,
1995,
Science.
[5]
Michael I. Jordan,et al.
Mean Field Theory for Sigmoid Belief Networks
,
1996,
J. Artif. Intell. Res..