论文信息 - Sampling Hidden Parameters from Oracle Distribution

Sampling Hidden Parameters from Oracle Distribution

A new sampling learning method for neural networks is proposed. Derived from an integral representation of neural networks, an oracle probability distribution of hidden parameters is introduced. In general rigorous sampling from the oracle distribution holds numerical difficulty, a linear-time sampling algorithm is also developed. Numerical experiments showed that when hidden parameters were initialized by the oracle distribution, following backpropagation converged faster to better parameters than when parameters were initialized by a normal distribution.

Noboru Murata | Sho Sonoda | Sho Sonoda | Noboru Murata

[1] Thierry Denoeux,et al. Initializing back propagation networks with prototypes , 1993, Neural Networks.

[2] Arnaud Doucet,et al. Sequential Monte Carlo Methods to Train Neural Network Models , 2000, Neural Computation.

[3] Yoshua Bengio,et al. Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[4] Noboru Murata,et al. An Integral Representation of Functions Using Three-layered Networks and Their Approximation Bounds , 1996, Neural Networks.

[5] Vra Krková. Complexity estimates based on integral transforms induced by computational units , 2012 .

[6] Pascal Vincent,et al. Representation Learning: A Review and New Perspectives , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[7] David A. Sprecher,et al. A Numerical Implementation of Kolmogorov's Superpositions II , 1996, Neural Networks.

[8] Klaus-Robert Müller,et al. Efficient BackProp , 2012, Neural Networks: Tricks of the Trade.