Learning in Neural Networks with Material Synapses

We discuss the long term maintenance of acquired memory in synaptic connections of a perpetually learning electronic device. This is affected by ascribing each synapse a finite number of stable states in which it can maintain for indefinitely long periods. Learning uncorrelated stimuli is expressed as a stochastic process produced by the neural activities on the synapses. In several interesting cases the stochastic process can be analyzed in detail, leading to a clarification of the performance of the network, as an associative memory, during the process of uninterrupted learning. The stochastic nature of the process and the existence of an asymptotic distribution for the synaptic values in the network imply generically that the memory is a palimpsest but capacity is as low as log N for a network of N neurons. The only way we find for avoiding this tight constraint is to allow the parameters governing the learning process (the coding level of the stimuli; the transition probabilities for potentiation and depression and the number of stable synaptic levels) to depend on the number of neurons. It is shown that a network with synapses that have two stable states can dynamically learn with optimal storage efficiency, be a palimpsest, and maintain its (associative) memory for an indefinitely long time provided the coding level is low and depression is equilibrated against potentiation. We suggest that an option so easily implementable in material devices would not have been overlooked by biology. Finally we discuss the stochastic learning on synapses with variable number of stable synaptic states.

[1]  Walter J. Freeman,et al.  The Hebbian paradigm reintegrated: Local reverberations as internal representations , 1995, Behavioral and Brain Sciences.

[2]  W. Krauth,et al.  Storage capacity of memory networks with binary couplings , 1989 .

[3]  David R. Cox,et al.  The Theory of Stochastic Processes , 1967, The Mathematical Gazette.

[4]  D. Amit,et al.  Statistical mechanics of neural networks near saturation , 1987 .

[5]  Frank der van Velde,et al.  Association and computation with cell assemblies , 1995, Behavioral and Brain Sciences.

[6]  David C. Krakauer,et al.  An evolutionary perspective on Hebb's reverberatory representations , 1995, Behavioral and Brain Sciences.

[7]  Gérard Weisbuch,et al.  Scaling laws for the attractors of Hopfield networks , 1985 .

[8]  James J. Wright,et al.  How do local reverberations achieve global integration? , 1995, Behavioral and Brain Sciences.

[9]  Gj Dalenoort,et al.  What's in a cell assembly? , 1995, Behavioral and Brain Sciences.

[10]  Y. Miyashita Neuronal correlate of visual associative long-term memory in the primate temporal cortex , 1988, Nature.

[11]  Josef P. Rauschecker,et al.  Reverberations of Hebbian thinking , 1995, Behavioral and Brain Sciences.

[12]  Davide Badoni,et al.  Learning Attractor Neural Network: The Electronic Implementation , 1992, Int. J. Neural Syst..

[13]  Peter C. M. Molenaar,et al.  How to decide whether a neural representation is a cognitive concept? , 1995, Behavioral and Brain Sciences.

[14]  H. C. LONGUET-HIGGINS,et al.  Non-Holographic Associative Memory , 1969, Nature.

[15]  Anders Lansner,et al.  Distributed cell assemblies and detailed cell models , 1995, Behavioral and Brain Sciences.

[16]  D. Amit The Hebbian paradigm reintegrated: Local reverberations as internal representations , 1995, Behavioral and Brain Sciences.

[17]  Daniel J. Amit,et al.  Empirical and theoretical active memory: The proper context , 1995, Behavioral and Brain Sciences.

[18]  Jacek Iwanski Replica symmetry breaking calculation of the storage capacity of neural networks with discrete couplings , 1994 .

[19]  G. Parisi A memory which forgets , 1986 .

[20]  Nicolas Brunel,et al.  Adequate input for learning in attractor neural networks , 1993 .

[21]  E. M.,et al.  Statistical Mechanics , 2021, On Complementarity.

[22]  Masahiko Morita,et al.  Another ANN model for the Miyashita experiments , 1995, Behavioral and Brain Sciences.

[23]  Edoardo Amaldi,et al.  Stability-Capacity Diagram of a Neural Network with Ising Bonds , 1989 .

[24]  Michael Hucka,et al.  Hebb's accomplishments misunderstood , 1995, Behavioral and Brain Sciences.

[25]  Joaquin M. Fuster,et al.  Not the module does memory make – but the network , 1995, Behavioral and Brain Sciences.

[26]  Hubert Preissl,et al.  Local or transcortical assemblies? Some evidence from cognitive neuroscience , 1995, Behavioral and Brain Sciences.

[27]  Daniel J. Amit,et al.  Constraints on learning in dynamic synapses , 1992, Network: Computation in Neural Systems.

[28]  Daniel J. Amit,et al.  Conversion of Temporal Correlations Between Stimuli to Spatial Correlations Between Attractors , 1999, Neural Computation.

[29]  H. Gutfreund,et al.  Capacity of neural networks with discrete synaptic couplings , 1990 .

[30]  Shimon Edelman,et al.  What's in a cell assembly? , 1995, Behavioral and Brain Sciences.

[31]  D. Amit,et al.  Constraints on learning in dynamic synapses , 1992 .

[32]  Eric Chown,et al.  Reverberation reconsidered: On the path to cognitive theory , 1995, Behavioral and Brain Sciences.

[33]  Morris W. Hirsch,et al.  Mathematics of Hebbian attractors , 1995, Behavioral and Brain Sciences.

[34]  Wolfgang Klimesch,et al.  The functional meaning of reverberations for sensoric and contextual encoding , 1995, Behavioral and Brain Sciences.

[35]  Stanislas Dehaene,et al.  Networks of Formal Neurons and Memory Palimpsests , 1986 .

[36]  P. Milner,et al.  Attractors – don't get sucked in , 1995, Behavioral and Brain Sciences.

[37]  Ralph E. Hoffman,et al.  Additional tests of Amit's attractor neural networks , 1995, Behavioral and Brain Sciences.

[38]  I. I. Gikhman Theory of stochastic processes , 1974 .

[39]  Jean Petitot,et al.  The problems of cognitive dynamical models , 1995, Behavioral and Brain Sciences.

[40]  M. Tsodyks ASSOCIATIVE MEMORY IN NEURAL NETWORKS WITH BINARY SYNAPSES , 1990 .

[41]  Elie Bienenstock,et al.  Where the adventure is , 1995, Behavioral and Brain Sciences.

[42]  N. S. Barnett,et al.  Private communication , 1969 .

[43]  Ehud Ahissar,et al.  Are single-cell data sufficient for testing neural network models? , 1995, Behavioral and Brain Sciences.

[44]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[45]  K. Stratford,et al.  Synaptic transmission between individual pyramidal neurons of the rat visual cortex in vitro , 1991, The Journal of neuroscience : the official journal of the Society for Neuroscience.