Neural network learning algorithms

The earliest network models of associative memory were based on correlations between input and output patterns of activity in linear processing units. These models have several features that make them attractive: The synaptic strengths are computed from information available locally at each synapse in a single trial; the information is distributed in a large number of connection strengths, the recall of stored information is associative, and the network can generalize to new input patterns that are similar to stored patterns. There are also severe limitations with this class of linear associative matrix models, including interference between stored items, especially between ones that are related, and inability to make decisions that are contingent on several inputs. New neural network models and neural network learning algorithms have been introduced recently that overcome some of the shortcomings of the associative matrix models of memory. These learning algorithms require many training examples to create the internal representations needed to perform a difficult task and generalize properly. They share some properties with human skill acquisition.

[1]  Bernard Widrow,et al.  Adaptive switching circuits , 1988 .

[2]  A. Tversky,et al.  Subjective Probability: A Judgment of Representativeness , 1972 .

[3]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[4]  Terrence J. Sejnowski,et al.  NETtalk: a parallel network that learns to read aloud , 1988 .

[5]  Geoffrey E. Hinton,et al.  A Learning Algorithm for Boltzmann Machines , 1985, Cogn. Sci..

[6]  G. Bower,et al.  From conditioning to category learning: an adaptive network model. , 1988 .

[7]  Azriel Rosenfeld,et al.  Blob Detection by Relaxation , 1981, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Richard F. Thompson,et al.  Modeling the Neural Substrates of Associative Learning and Memory: A Computational Approach , 1987 .

[9]  W. F. Prokasy,et al.  Classical conditioning II: Current research and theory. , 1972 .

[10]  Geoffrey E. Hinton,et al.  OPTIMAL PERCEPTUAL INFERENCE , 1983 .

[11]  L. Finkel,et al.  Interaction of synaptic modification rules within populations of neurons. , 1985, Proceedings of the National Academy of Sciences of the United States of America.

[12]  Teuvo Kohonen,et al.  Self-Organization and Associative Memory , 1988 .

[13]  James L. McClelland,et al.  Psychological and biological models , 1986 .

[14]  Roger K. Moore.  Computer Speech and Language , 1986 .

[15]  Terrence J. Sejnowski,et al.  A Parallel Network that Learns to Play Backgammon , 1989, Artif. Intell..

[16]  Terrence J. Sejnowski,et al.  Learned classification of sonar targets using a massively parallel network , 1988, IEEE Trans. Acoust. Speech Signal Process..

[17]  David Zipser,et al.  Feature Discovery by Competive Learning , 1986, Cogn. Sci..

[18]  Jerome A. Feldman,et al.  Neural Representation of Conceptual Knowledge. , 1986 .

[19]  Klaus Schulten,et al.  Stochastic spin models for pattern recognition , 1987 .

[20]  W. Pitts,et al.  A Logical Calculus of the Ideas Immanent in Nervous Activity (1943) , 2021, Ideas That Created the Future.

[21]  Donald Geman,et al.  Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images , 1984, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  Teuvo Kohonen,et al.  Correlation Matrix Memories , 1972, IEEE Transactions on Computers.

[23]  A. Gamba,et al.  An outline of a mathematical theory of PAPA , 1961 .

[24]  D Marr,et al.  Cooperative computation of stereo disparity. , 1976, Science.

[25]  T. D. Harrison,et al.  Boltzmann machines for speech recognition , 1986 .

[26]  Barak A. Pearlmutter,et al.  G-maximization: An unsupervised learning procedure for discovering regularities , 1987 .

[27]  Françoise Fogelman-Soulié,et al.  Disordered Systems and Biological Organization , 1986, NATO ASI Series.

[28]  Geoffrey E. Hinton,et al.  Learning symmetry groups with hidden units: beyond the perceptron , 1986 .

[29]  R Linsker,et al.  From basic network principles to neural architecture: emergence of orientation columns. , 1986, Proceedings of the National Academy of Sciences of the United States of America.

[30]  Geoffrey E. Hinton,et al.  Learning and relearning in Boltzmann machines , 1986 .

[31]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[32]  E. Bienenstock,et al.  Theory for the development of neuron selectivity: orientation specificity and binocular interaction in visual cortex , 1982, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[33]  A. A. Mullin,et al.  Principles of neurodynamics , 1962 .

[34]  Elie Bienenstock,et al.  A neural network for the retrieval of superimposed connection patterns , 1987 .

[35]  Geoffrey E. Hinton,et al.  Parallel Models of Associative Memory , 1989 .

[36]  A. Harry Klopf,et al.  A drive-reinforcement model of single neuron function , 1987 .

[38]  J. Hopfield,et al.  Computing with neural circuits: a model. , 1986, Science.

[39]  Jerome A. Feldman,et al.  Connectionist Models and Their Properties , 1982, Cogn. Sci..

[40]  Geoffrey E. Hinton,et al.  Parallel visual computation , 1983, Nature.

[41]  Paul Smolensky,et al.  Schema Selection and Stochastic Inference in Modular Environments , 1983, AAAI.

[42]  T. Sejnowski.  Statistical constraints on synaptic plasticity. , 1977, Journal of theoretical biology.

[43]  G. Lynch,et al.  Synapses, circuits, and the beginnings of memory , 1986 .

[44]  T. Sejnowski,et al.  Storing covariance with nonlinearly interacting neurons , 1977, Journal of mathematical biology.

[45]  J.A. Anderson.  Two models for memory organization using interacting traces , 1970 .

[46]  J J Hopfield,et al.  Neurons with graded response have collective computational properties like those of two-state neurons. , 1984, Proceedings of the National Academy of Sciences of the United States of America.

[47]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[48]  W. Levy,et al.  Partial quantification of the associative synaptic learning rule of the dentate gyrus , 1983, Neuroscience.

[49]  A G Barto,et al.  Toward a modern theory of adaptive networks: expectation and prediction. , 1981, Psychological review.

[50]  Geoffrey E. Hinton,et al.  Separating Figure from Ground with a Parallel Network , 1986, Perception.

[51]  T. Sejnowski,et al.  Predicting the secondary structure of globular proteins using neural network models. , 1988, Journal of molecular biology.

[52]  S Dehaene,et al.  Spin glass model of learning by selection. , 1986, Proceedings of the National Academy of Sciences of the United States of America.

[53]  A. Gamba,et al.  Further experiments with PAPA , 1961 .

[54]  A G Barto,et al.  Learning by statistical cooperation of self-interested neuron-like computing elements. , 1985, Human neurobiology.

[55]  Geoffrey E. Hinton,et al.  Learning distributed representations of concepts. , 1989 .

[56]  S. Kelso,et al.  Hebbian synapses in hippocampus. , 1986, Proceedings of the National Academy of Sciences of the United States of America.

[57]  Terrence J. Sejnowski,et al.  Open questions about computation in cerebral cortex , 1986 .

[58]  R. Rescorla,et al.  A theory of Pavlovian conditioning : Variations in the effectiveness of reinforcement and nonreinforcement , 1972 .

[59]  S. Kaplan.  The Physiology of Thought , 1950 .

[60]  R Lewin,et al.  RNA plasmid discovered in maize mitochondria. , 1986, Science.