An adaptive least squares algorithm for the efficient training of artificial neural networks

A novel learning algorithm is developed for the training of multilayer feedforward neural networks, based on a modification of the Marquardt-Levenberg least-squares optimization method. The algorithm updates the input weights of each neuron in the network in an effective parallel way. An adaptive distributed selection of the convergence rate parameter is presented, using suitable optimization strategies. The algorithm has better convergence properties than the conventional backpropagation learning technique. Its performance is illustrated, using examples from digital image halftoning and logical operations such as the XOR function. >

[1]  Kenneth Levenberg.  A METHOD FOR THE SOLUTION OF CERTAIN NON – LINEAR PROBLEMS IN LEAST SQUARES , 1944 .

[2]  D. Marquardt.  An Algorithm for Least-Squares Estimation of Nonlinear Parameters , 1963 .

[3]  John F. Jarvis,et al.  A survey of techniques for the display of continuous tone pictures on bilevel displays , 1976 .

[4]  J. Cadzow.  Recursive digital filter synthesis via gradient based algorithms , 1976 .

[5]  M. R. Osborne.  Nonlinear least squares — the Levenberg algorithm revisited , 1976, The Journal of the Australian Mathematical Society. Series B. Applied Mathematics.

[6]  R. Fletcher,et al.  A modified Newton method for minimization , 1977 .

[7]  Jorge J. Moré,et al.  The Levenberg-Marquardt algo-rithm: Implementation and theory , 1977 .

[8]  George Carayannis,et al.  A fast sequential algorithm for least-squares filtering and prediction , 1983 .

[9]  John E. Dennis,et al.  Numerical methods for unconstrained optimization and nonlinear equations , 1983, Prentice Hall series in computational mathematics.

[10]  A. D. Raza,et al.  Augmenting computer networks , 1984 .

[11]  Ralph Tindell,et al.  Circulants and their connectivities , 1984, J. Graph Theory.

[12]  Jhing-Fa Wang,et al.  Reliable circulant networks with minimum transmission delay , 1985 .

[13]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[14]  Paul J. Werbos,et al.  Building and Understanding Adaptive Systems: A Statistical/Numerical Approach to Factory Automation and Brain Research , 1987, IEEE Transactions on Systems, Man, and Cybernetics.

[15]  Terrence J. Sejnowski,et al.  Parallel Networks that Learn to Pronounce English Text , 1987, Complex Syst..

[16]  Raymond L. Watrous.  Learning Algorithms for Connectionist Networks: Applied Gradient Methods of Nonlinear Optimization , 1988 .

[17]  P. J. Werbos,et al.  Backpropagation: past and future , 1988, IEEE 1988 International Conference on Neural Networks.

[18]  Jean-Claude Bermond,et al.  Large fault-tolerant interconnection networks , 1989, Graphs Comb..

[19]  Stefanos D. Kollias,et al.  A fast multichannel approach to adaptive image estimation , 1989, IEEE Trans. Acoust. Speech Signal Process..