论文信息 - Bayesian Backpropagation Over I-O Functions Rather Than Weights

Bayesian Backpropagation Over I-O Functions Rather Than Weights

The conventional Bayesian justification for backprop is that it finds the MAP weight vector. As this paper shows, to find the MAP i-o function instead, one must add a correction term to backprop. That term biases one towards i-o functions with small description lengths, and in particular favors (some kinds of) feature-selection, pruning, and weight-sharing. This can be viewed as an {\it a priori} argument in favor of those techniques.

David H. Wolpert | D. Wolpert

[1] A. F. Smith,et al. Statistical analysis of finite mixture distributions , 1986 .

[2] J. Rissanen. Stochastic Complexity and Modeling , 1986 .

[3] J. Berger. Statistical Decision Theory and Bayesian Analysis , 1988 .

[4] John Scott Bridle,et al. Probabilistic Interpretation of Feedforward Classification Network Outputs, with Relationships to Statistical Pattern Recognition , 1989, NATO Neurocomputing.

[5] Yann LeCun,et al. Transforming Neural-Net Output Levels to Probability Distributions , 1990, NIPS.

[6] D. Rumelhart,et al. Predicting sunspots and exchange rates with connectionist networks , 1991 .

[7] Wray L. Buntine,et al. Bayesian Back-Propagation , 1991, Complex Syst..

[8] David H. Wolpert,et al. On the Connection between In-sample Testing and Generalization Error , 1992, Complex Syst..

[9] David H. Wolpert,et al. On the Use of Evidence in Neural Networks , 1992, NIPS.

[10] David J. C. MacKay,et al. Bayesian Interpolation , 1992, Neural Computation.

[11] Babak Hassibi,et al. Second Order Derivatives for Network Pruning: Optimal Brain Surgeon , 1992, NIPS.

[12] Radford M. Neal. Bayesian Learning via Stochastic Dynamics , 1992, NIPS.

[13] Geoffrey E. Hinton,et al. Simplifying Neural Networks by Soft Weight-Sharing , 1992, Neural Computation.

[14] C. Fefferman. Reconstructing a neural net from its output , 1994 .

[15] David H. Wolpert,et al. Filter likelihoods and exhaustive learning , 1994, COLT 1994.