Appendix G: Thirty Years of Adaptive Neural Networks: Perceptron, Madaline, and Backpropagation

This chapter contains sections titled: Introduction Fundamental Concepts Adaptation ??????-?????? The Minimal Disturbance Principle Error Correction Rules ??????-?????? Single Threshold Element Error Correction Rules ??????-?????? Multi-Element Networks Steepest-Descent Rules ??????-?????? Single Threshold Element Steepest-Descent Rules ??????-?????? Multi-Element Networks Summary Acknowledgments Bibliography

[1]  G. TEMPLE,et al.  Relaxation Methods in Engineering Science , 1942, Nature.

[2]  Richard S. Sutton,et al.  The Truck Backer-Upper: An Example of Self-Learning in Neural Networks , 1995 .

[3]  B Kosko,et al.  Adaptive bidirectional associative memories. , 1987, Applied optics.

[4]  B. Irie,et al.  Capabilities of three-layered perceptrons , 1988, IEEE 1988 International Conference on Neural Networks.

[5]  D F Specht,et al.  Vectorcardiographic diagnosis using the polynomial discriminant method of pattern recognition. , 1967, IEEE transactions on bio-medical engineering.

[6]  Yoh-Han Pao,et al.  Functional link nets: removing hidden layers , 1989 .

[7]  M. M. Sondhi,et al.  An adaptive echo canceller , 1967 .

[8]  Filson Henry Glanz,et al.  Statistical extrapolation in certain adaptive pattern-recognition systems , 1965 .

[9]  R. E. Kalman,et al.  Optimum Seeking Methods. , 1964 .

[10]  Thomas Kailath,et al.  A view of three decades of linear filtering theory , 1974, IEEE Trans. Inf. Theory.

[11]  David Haussler,et al.  What Size Net Gives Valid Generalization? , 1989, Neural Computation.

[12]  W. Thomas Miller,et al.  Sensor-based control of robotic manipulators using a general learning algorithm , 1987, IEEE J. Robotics Autom..

[13]  Barak A. Pearlmutter.  Learning State Space Trajectories in Recurrent Neural Networks , 1989, Neural Computation.

[14]  Charles M. Newman,et al.  Memory capacity in neural network models: Rigorous lower bounds , 1988, Neural Networks.

[15]  D. Casasent,et al.  Position, rotation, and scale invariant optical correlation. , 1976, Applied optics.

[16]  C. Malsburg.  Self-organization of orientation sensitive cells in the striate cortex , 2004, Kybernetik.

[17]  Bernard Widrow,et al.  Punish/Reward: Learning with a Critic in Adaptive Threshold Systems , 1973, IEEE Trans. Syst. Man Cybern..

[18]  S. Grossberg,et al.  Adaptive pattern classification and universal recoding: I. Parallel development and coding of neural feature detectors , 1976, Biological Cybernetics.

[19]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1989, Math. Control. Signals Syst..

[20]  J J Hopfield,et al.  Neurons with graded response have collective computational properties like those of two-state neurons. , 1984, Proceedings of the National Academy of Sciences of the United States of America.

[21]  Carver Mead,et al.  Analog VLSI and neural systems , 1989 .

[22]  R W Lucky,et al.  Principles of data communication , 1968 .

[23]  Alireza Khotanzad,et al.  Rotation invariant pattern recognition using Zernike moments , 1988, [1988 Proceedings] 9th International Conference on Pattern Recognition.

[24]  A PearlmutterBarak.  Learning state space trajectories in recurrent neural networks , 1989 .

[25]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[26]  Donald F. Specht,et al.  Generation of Polynomial Discriminant Functions for Pattern Recognition , 1967, IEEE Trans. Electron. Comput..

[27]  Geoffrey E. Hinton,et al.  Learning and relearning in Boltzmann machines , 1986 .

[28]  F. K. Becker,et al.  Automatic equalization for digital communication , 1965 .

[29]  Jr. William Louis Reber.  Artificial neural system design: rotation and scale invariant pattern recognition , 1987 .

[30]  Norbert Wiener,et al.  Extrapolation, Interpolation, and Smoothing of Stationary Time Series, with Engineering Applications , 1949 .

[31]  Eduardo D. Sontag,et al.  Backpropagation Can Give Rise to Spurious Local Minima Even for Networks without Hidden Layers , 1989, Complex Syst..

[32]  Bernard Widrow,et al.  The least mean fourth (LMF) adaptive algorithm and its family , 1984, IEEE Trans. Inf. Theory.

[33]  Luís B. Almeida,et al.  A learning rule for asynchronous perceptrons with feedback in a combinatorial environment , 1990 .

[34]  Bernard Widrow,et al.  Neural nets for adaptive filtering and adaptive pattern recognition , 1988, Computer.

[35]  B. Widrow,et al.  Adaptive noise cancelling: Principles and applications , 1975 .

[36]  Rodney Gerard Winter,et al.  Madaline Rule II : a new method for training networks of Adalines , 1989 .

[37]  S. Tam,et al.  An electrically trainable artificial neural network (ETANN) with 10240 'floating gate' synapses , 1990, International 1989 Joint Conference on Neural Networks.

[38]  Y S Abu-Mostafa,et al.  Neural networks for computing , 1987 .

[39]  Louise Hay,et al.  THE NUMBER OF ORTHANTS IN N-SPACE INTERSECTED BY AN S-DIMENSIONAL SUBSPACE , 1960 .

[40]  Esther Levin,et al.  Accelerated Learning in Layered Neural Networks , 1988, Complex Syst..

[41]  D. Hammerstrom,et al.  Neural networks at work , 1993, IEEE Spectrum.

[42]  Stephen Grossberg,et al.  A massively parallel architecture for a self-organizing neural pattern recognition machine , 1988, Comput. Vis. Graph. Image Process..

[43]  Terrence J. Sejnowski,et al.  NETtalk: a parallel network that learns to read aloud , 1988 .

[44]  S. Grossberg,et al.  ART 2: self-organization of stable category recognition codes for analog input patterns. , 1987, Applied optics.

[45]  Christoph von der Malsburg,et al.  Pattern recognition by labeled graph matching , 1988, Neural Networks.

[46]  Teuvo Kohonen,et al.  Self-Organization and Associative Memory , 1988 .

[47]  Richard D. Braatz,et al.  On the "Identification and control of dynamical systems using neural networks" , 1997, IEEE Trans. Neural Networks.

[48]  Lawrence W. Stark,et al.  Computer pattern recognition techniques: electrocardiographic diagnosis , 1962, CACM.

[49]  Yaser S. Abu-Mostafa,et al.  Information capacity of the Hopfield model , 1985, IEEE Trans. Inf. Theory.

[50]  Richard P. Lippmann,et al.  An introduction to computing with neural nets , 1987 .

[51]  P. Werbos,et al.  Beyond Regression : "New Tools for Prediction and Analysis in the Behavioral Sciences , 1974 .

[52]  Eric B. Baum,et al.  Supervised Learning of Probability Distributions by Neural Networks , 1987, NIPS.

[53]  S. Venkatesh.  Epsilon capacity of neural networks , 1987 .

[54]  S. Thomas Alexander,et al.  Adaptive Signal Processing , 1986, Texts and Monographs in Computer Science.

[55]  Geoffrey E. Hinton,et al.  Phoneme recognition using time-delay neural networks , 1989, IEEE Trans. Acoust. Speech Signal Process..

[56]  Bernard Widrow,et al.  Improving the learning speed of 2-layer neural networks by choosing initial values of the adaptive weights , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[57]  Ronald J. Williams,et al.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks , 1989, Neural Computation.

[58]  Kunihiko Fukushima,et al.  Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position , 1980, Biological Cybernetics.

[59]  J. Shynk,et al.  The LMS algorithm with momentum updating , 1988, 1988., IEEE International Symposium on Circuits and Systems.

[60]  H. W. Bode,et al.  A Simplified Derivation of Linear Least Square Smoothing and Prediction Theory , 1950, Proceedings of the IRE.

[61]  Stephen Grossberg,et al.  Adaptive pattern classification and universal recoding: II. Feedback, expectation, olfaction, illusions , 1976, Biological Cybernetics.

[62]  Terrence J. Sejnowski,et al.  Parallel Networks that Learn to Pronounce English Text , 1987, Complex Syst..

[63]  R. Lippmann,et al.  An introduction to computing with neural nets , 1987, IEEE ASSP Magazine.

[64]  C. Lee Giles,et al.  Encoding Geometric Invariances in Higher-Order Neural Networks , 1987, NIPS.

[65]  Frank Rosenblatt,et al.  PRINCIPLES OF NEURODYNAMICS. PERCEPTRONS AND THE THEORY OF BRAIN MECHANISMS , 1963 .

[66]  Stephen Grossberg,et al.  ART 3: Hierarchical search using chemical transmitters in self-organizing pattern recognition architectures , 1990, Neural Networks.

[67]  Alberto L. Sangiovanni-Vincentelli,et al.  Efficient Parallel Learning Algorithms for Neural Networks , 1988, NIPS.

[68]  Richard Fozzard,et al.  A Connectionist Expert System that Actually Works , 1988, NIPS.

[69]  Robert A. Jacobs,et al.  Increased rates of convergence through learning rate adaptation , 1987, Neural Networks.

[70]  J. S. Koford,et al.  Real‐Time Adaptive Speech‐Recognition System , 1963 .

[71]  Kunihiko Fukushima,et al.  Cognitron: A self-organizing multilayered neural network , 1975, Biological Cybernetics.

[72]  K. Senne,et al.  Performance advantage of complex LMS for controlling narrow-band adaptive arrays , 1981 .

[73]  Richard S. Sutton,et al.  Neuronlike adaptive elements that can solve difficult learning control problems , 1983, IEEE Transactions on Systems, Man, and Cybernetics.

[74]  B. Widrow,et al.  Adaptive inverse control , 1987, Proceedings of 8th IEEE International Symposium on Intelligent Control.

[75]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[76]  H. D. Block.  The perceptron: a model for brain functioning. I , 1962 .

[77]  P. M. Shea,et al.  Detection of explosives in checked airline baggage using an artificial neural system , 1989, International 1989 Joint Conference on Neural Networks.

[78]  Eduardo D. Sontag,et al.  Backpropagation separates when perceptrons do , 1989, International 1989 Joint Conference on Neural Networks.

[79]  B. Widrow,et al.  Adaptive antenna systems , 1967 .

[80]  Teuvo Kohonen,et al.  Self-organized formation of topologically correct feature maps , 2004, Biological Cybernetics.

[81]  Joseph D. Greenfield.  Practical Digital Design Using Ics , 1977 .

[82]  Colin Giles,et al.  Learning, invariance, and generalization in high-order neural networks. , 1987, Applied optics.

[83]  B. Moore,et al.  ART1 and pattern clustering , 1989 .

[84]  Yann Le Cun,et al.  A Theoretical Framework for Back-Propagation , 1988 .

[85]  J. J. Hopfield,et al.  “Neural” computation of decisions in optimization problems , 1985, Biological Cybernetics.

[86]  H. White,et al.  Universal approximation using feedforward networks with non-sigmoid hidden layer activation functions , 1989, International 1989 Joint Conference on Neural Networks.