论文信息 - Slope Centering: Making Shortcut Weights Effective

Slope Centering: Making Shortcut Weights Effective

Shortcut connections are a popular architectural feature of multi-layer perceptrons. It is generally assumed that by implementing a linear sub-mapping, shortcuts assist the learning process in the remainder of the network. Here we find that this is not always the case: shortcut weights may also act as distractors that slow down convergence and can lead to inferior solutions. This problem can be addressed with slope centering, a particular form of gradient factor centering. By removing the linear component of the error signal at a hidden node, slope centering effectively decouples that node from the shortcuts that bypass it. This eliminates the possibility of destructive interference from shortcut weights, and thus ensures that the benefits of shortcut connections are fully realized.

N. Schraudolph

[1] B. Widrow,et al. Stationary and nonstationary learning characteristics of the LMS adaptive filter , 1976, Proceedings of the IEEE.

[2] Alan S. Lapedes,et al. A self-optimizing, nonsymmetrical neural net for content addressable memory and pattern recognition , 1986 .

[3] Roberto Battiti,et al. Accelerated Backpropagation Learning: Two Optimization Methods , 1989, Complex Syst..

[4] Kanter,et al. Eigenvalues of covariance matrices: Application to neural-network learning. , 1991, Physical review letters.

[5] Francesco Palmieri,et al. Optimal filtering algorithms for fast learning in feedforward neural networks , 1992, Neural Networks.

[6] Roberto Battiti,et al. First- and Second-Order Methods for Learning: Between Steepest Descent and Newton's Method , 1992, Neural Computation.

[7] Michael Finke,et al. Estimating A-Posteriori Probabilities using Stochastic Network Models , 1993 .

[8] Terrence J. Sejnowski,et al. Tempering Backpropagation Networks: Not All Weights are Created Equal , 1995, NIPS.

[9] Ralph Neuneier,et al. How to Train Neural Networks , 1996, Neural Networks: Tricks of the Trade.

[10] Jürgen Schmidhuber,et al. Unsupervised Coding with LOCOCODE , 1997, ICANN.

[11] J. van Leeuwen,et al. Neural Networks: Tricks of the Trade , 2002, Lecture Notes in Computer Science.