Adaptive neural networks and their applications

Fundamental developments in feedforward artificial neural networks from the past 30 years are reviewed. the central theme of this article is a description of the history, origination, operating characteristics, and basic theory of several supervised neural network training algorithms including the Perceptron rule, the LMS algorithm, three Madaline rules, and the backpropagation technique. These methods were developed independently, but with the perspective of history they can all be related to each other. the concept which underlies these algorithms is the “minimal disturbance principle,” which suggests that during training it is advisable to inject new information into a network in a manner which disturbs stored information to the smallest extent possible. In the utilization of present‐day rule‐based expert systems, decision rules must always be known for the application of interest. Sometimes there are no rules, however. the rules are either not explicit or they simply do not exist. For such applications, trainable expert systems might be usable. Rather than working with decision rules, an adaptive expert system might observe the decisions made by a human expert. Looking over the expert's shoulders, an adaptive system can learn to make similar decisions to those of the human. Trainable expert systems have been used in the laboratory for real‐time control of a “broom‐balancing system.” © 1993 John Wiley & Sons, Inc.

[1]  W. Thomas Miller,et al.  Sensor-based control of robotic manipulators using a general learning algorithm , 1987, IEEE J. Robotics Autom..

[2]  G. TEMPLE,et al.  Relaxation Methods in Engineering Science , 1942, Nature.

[3]  B Kosko,et al.  Adaptive bidirectional associative memories. , 1987, Applied optics.

[4]  Pineda,et al.  Generalization of back-propagation to recurrent neural networks. , 1987, Physical review letters.

[5]  David Haussler,et al.  What Size Net Gives Valid Generalization? , 1989, Neural Computation.

[6]  H. W. Bode,et al.  A Simplified Derivation of Linear Least Square Smoothing and Prediction Theory , 1950, Proceedings of the IRE.

[7]  J. S. Koford,et al.  Real‐Time Adaptive Speech‐Recognition System , 1963 .

[8]  K. Senne,et al.  Performance advantage of complex LMS for controlling narrow-band adaptive arrays , 1981 .

[9]  J. Orbach.  Principles of Neurodynamics. Perceptrons and the Theory of Brain Mechanisms. , 1962 .

[10]  Richard S. Sutton,et al.  Neuronlike adaptive elements that can solve difficult learning control problems , 1983, IEEE Transactions on Systems, Man, and Cybernetics.

[11]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[12]  Barak A. Pearlmutter.  Learning State Space Trajectories in Recurrent Neural Networks , 1989, Neural Computation.

[13]  Donald F. Specht,et al.  Generation of Polynomial Discriminant Functions for Pattern Recognition , 1967, IEEE Trans. Electron. Comput..

[14]  Geoffrey E. Hinton,et al.  Learning and relearning in Boltzmann machines , 1986 .

[15]  Bernard Widrow,et al.  The least mean fourth (LMF) adaptive algorithm and its family , 1984, IEEE Trans. Inf. Theory.

[16]  Teuvo Kohonen,et al.  Self-Organization and Associative Memory , 1988 .

[17]  Yann LeCun,et al.  A theoretical framework for back-propagation , 1988 .

[18]  Colin Giles,et al.  Learning, invariance, and generalization in high-order neural networks. , 1987, Applied optics.

[19]  Robert A. Jacobs,et al.  Increased rates of convergence through learning rate adaptation , 1987, Neural Networks.

[20]  Karl Steinbuch,et al.  Learning Matrices and Their Applications , 1963, IEEE Trans. Electron. Comput..

[21]  C. Lee Giles,et al.  Encoding Geometric Invariances in Higher-Order Neural Networks , 1987, NIPS.

[22]  Terrence J. Sejnowski,et al.  Parallel Networks that Learn to Pronounce English Text , 1987, Complex Syst..

[23]  A. G. Ivakhnenko,et al.  Polynomial Theory of Complex Systems , 1971, IEEE Trans. Syst. Man Cybern..

[24]  Bernard Widrow,et al.  Neural nets for adaptive filtering and adaptive pattern recognition , 1988, Computer.

[25]  B. Widrow,et al.  Adaptive noise cancelling: Principles and applications , 1975 .

[26]  Rodney Gerard Winter,et al.  Madaline Rule II : a new method for training networks of Adalines , 1989 .

[27]  S. Tam,et al.  An electrically trainable artificial neural network (ETANN) with 10240 'floating gate' synapses , 1990, International 1989 Joint Conference on Neural Networks.

[28]  Stephen Grossberg,et al.  A massively parallel architecture for a self-organizing neural pattern recognition machine , 1988, Comput. Vis. Graph. Image Process..

[29]  D F Specht,et al.  Vectorcardiographic diagnosis using the polynomial discriminant method of pattern recognition. , 1967, IEEE transactions on bio-medical engineering.

[30]  Marvin Minsky,et al.  Perceptrons: An Introduction to Computational Geometry , 1969 .

[31]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[32]  H. D. Block.  The perceptron: a model for brain functioning. I , 1962 .

[33]  Shun-ichi Amari,et al.  A Theory of Adaptive Pattern Classifiers , 1967, IEEE Trans. Electron. Comput..

[34]  P. M. Shea,et al.  Detection of explosives in checked airline baggage using an artificial neural system , 1989, International 1989 Joint Conference on Neural Networks.

[35]  Kumpati S. Narendra,et al.  Identification and control of dynamical systems using neural networks , 1990, IEEE Trans. Neural Networks.

[36]  Stephen Grossberg,et al.  Art 2: Self-Organization Of Stable Category Recognition Codes For Analog Input Patterns , 1988, Other Conferences.

[37]  Roman Bek,et al.  Discourse on one way in which a quantum-mechanics language on the classical logical base can be built up , 1978, Kybernetika.

[38]  James S. Albus,et al.  New Approach to Manipulator Control: The Cerebellar Model Articulation Controller (CMAC)1 , 1975 .

[39]  Richard P. Lippmann,et al.  An introduction to computing with neural nets , 1987 .

[40]  P. Werbos,et al.  Beyond Regression : "New Tools for Prediction and Analysis in the Behavioral Sciences , 1974 .

[41]  Ronald J. Williams,et al.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks , 1989, Neural Computation.

[42]  B. Widrow,et al.  The truck backer-upper: an example of self-learning in neural networks , 1989, International 1989 Joint Conference on Neural Networks.

[43]  F. K. Becker,et al.  Automatic equalization for digital communication , 1965 .

[44]  Alberto L. Sangiovanni-Vincentelli,et al.  Efficient Parallel Learning Algorithms for Neural Networks , 1988, NIPS.

[45]  D. O. Hebb,et al.  The organization of behavior , 1988 .

[46]  Richard Fozzard,et al.  A Connectionist Expert System that Actually Works , 1988, NIPS.

[47]  R W Lucky,et al.  Principles of data communication , 1968 .

[48]  Terrence J. Sejnowski,et al.  NETtalk: a parallel network that learns to read aloud , 1988 .

[49]  B. Widrow,et al.  Adaptive antenna systems , 1967 .

[50]  Filson Henry Glanz,et al.  Statistical extrapolation in certain adaptive pattern-recognition systems , 1965 .

[51]  J J Hopfield,et al.  Neurons with graded response have collective computational properties like those of two-state neurons. , 1984, Proceedings of the National Academy of Sciences of the United States of America.

[52]  J. Shynk,et al.  The LMS algorithm with momentum updating , 1988, 1988., IEEE International Symposium on Circuits and Systems.

[53]  Lawrence W. Stark,et al.  Computer pattern recognition techniques: electrocardiographic diagnosis , 1962, CACM.

[54]  Bernard Widrow,et al.  30 years of adaptive neural networks: perceptron, Madaline, and backpropagation , 1990, Proc. IEEE.

[55]  T. Kohonen.  Self-organized formation of topographically correct feature maps , 1982 .

[56]  Shuji Doshita,et al.  The Automatic Speech Recognition System for Conversational Sound , 1963, IEEE Trans. Electron. Comput..

[57]  Bernard Widrow,et al.  Improving the learning speed of 2-layer neural networks by choosing initial values of the adaptive weights , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[58]  Bernard Widrow,et al.  Adaptive switching circuits , 1988 .

[59]  James A. Anderson,et al.  Neurocomputing: Foundations of Research , 1988 .

[60]  Bernard Widrow,et al.  Punish/Reward: Learning with a Critic in Adaptive Threshold Systems , 1973, IEEE Trans. Syst. Man Cybern..

[61]  M. M. Sondhi,et al.  An adaptive echo canceller , 1967 .

[62]  P. Anandan,et al.  Pattern-recognizing stochastic learning automata , 1985, IEEE Transactions on Systems, Man, and Cybernetics.