In this work we study a network architecture that is able to overcome some of the limitations of conventional second-order acceleration methods of the backpropagation algorithm. This architecture is based on the combination of conventional backpropagation layers with unsupervised layers that perform a simple data orthogonalization. A distributed learning algorithm for these unsupervised layers is briefly reviewed, and a simple learning strategy for the combined architecture is described. The performance of the suggested architecture is evaluated using the well-known “two-spirals” problem, showing speed gains of at least one order of magnitude over conventional acceleration techniques. Moreover, it is shown that rather small feed-forward networks are able to solve this simple classification task within an acceptable learning time.
[1]
Robert W. Chang,et al.
A new equalizer structure for fast start-up digital communication
,
1971
.
[2]
K. Lang,et al.
Learning to tell two spirals apart
,
1988
.
[3]
Yann LeCun,et al.
Improving the convergence of back-propagation learning with second-order methods
,
1989
.
[4]
Luís B. Almeida,et al.
Acceleration Techniques for the Backpropagation Algorithm
,
1990,
EURASIP Workshop.
[5]
Sophocles J. Orfanidis,et al.
GramSchmidt Neural Nets
,
1990,
Neural Computation.
[6]
Robert A. Jacobs,et al.
Increased rates of convergence through learning rate adaptation
,
1987,
Neural Networks.