Willshaw model: Associative memory with sparse coding and low firing rates.

The Willshaw model of associative memory, implemented in a fully connected network with stochastic asynchronous dynamics, is studied. In addition to Willshaw's learning rule, the network contains uniform synaptic inhibition, of relative strength K, and negative neural threshold -\ensuremath{\theta},\ensuremath{\theta}g0. The P stored memories are sparsely coded. The total number of on bits in each memory is Nf, where f is much smaller than 1 but much larger than lnN/N. Mean-field theory of the system is solved in the limit where C==exp(-${f}^{2}$P) is finite. Memory states are stable (at zero temperature), as long as Cg${h}_{0}$==K-1+\ensuremath{\theta} and ${h}_{0}$g0. When C${h}_{0}$ or ${h}_{0}$0, P retrieval phases, highly correlated with the memory states, exist. These phases are only partially frozen at low temperature, so that the full memories can be retrieved from them by averaging over the dynamic fluctuations of the neural activity. In particular, when ${h}_{0}$0 the retrieval phases at low temperatures correspond to freezing of most of the population in a quiescent state while the rest are active with a time average that can be significantly smaller than the saturation level. These features resemble, to some extent, the observed patterns of neural activity in the cortex, in experiments of short-term memory tasks. The maximal value of P for which stable retrieval phases exist, scales as ${f}^{\mathrm{\ensuremath{-}}3}$/\ensuremath{\Vert}lnf\ensuremath{\Vert} for f\ensuremath{\gg}1/lnN, and as ${f}^{\mathrm{\ensuremath{-}}2}$ln(Nf/\ensuremath{\Vert}lnf\ensuremath{\Vert}) for f\ensuremath{\ll}1/lnN. Numerical simulations of the model with N=1000 and f=0.04 are presented. We also discuss the possible realization of the model in a biologically plausible architecture, where the inhibition is provided by special inhibitory neurons.

[1]  J. Griffith.  On the stability of brain-like structures. , 1963, Biophysical journal.

[2]  R. Glauber.  Time‐Dependent Statistics of the Ising Model , 1963 .

[3]  H. C. LONGUET-HIGGINS,et al.  Non-Holographic Associative Memory , 1969, Nature.

[4]  Professor Dr. Valentino Braitenberg,et al.  On the Texture of Brains , 1977, Heidelberg Science Library.

[5]  J. M. Oshorn.  Proc. Nat. Acad. Sei , 1978 .

[6]  Professor Moshe Abeles,et al.  Local Cortical Circuits , 1982, Studies of Brain Function.

[7]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[8]  J M Fuster,et al.  Neuronal firing in the inferotemporal cortex of the monkey in a visual memory task , 1982, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[9]  Gonzalo Viana Di Prisco,et al.  Hebb synaptic plasticity , 1984, Progress in Neurobiology.

[10]  William B. Levy,et al.  Synaptic modification, neuron selectivity, and nervous system organization , 1985 .

[11]  Sompolinsky,et al.  Spin-glass models of neural networks. , 1985, Physical review. A, General physics.

[12]  Kanter,et al.  Temporal association in asymmetric neural networks. , 1986, Physical review letters.

[13]  J. Hopfield,et al.  Computing with neural circuits: a model. , 1986, Science.

[14]  K. Binder,et al.  Spin glasses: Experimental facts, theoretical concepts, and open questions , 1986 .

[15]  S. Kelso,et al.  Hebbian synapses in hippocampus. , 1986, Proceedings of the National Academy of Sciences of the United States of America.

[16]  D. Amit,et al.  Statistical mechanics of neural networks near saturation , 1987 .

[17]  E. G. Jones.  Cerebral Cortex , 1987, Cerebral Cortex.

[18]  S K Khanna,et al.  Electronic hardware implementations of neural networks. , 1987, Applied optics.

[19]  Sompolinsky,et al.  Information storage in neural networks with low levels of activity. , 1987, Physical review. A, General physics.

[20]  M. Tsodyks.  Associative Memory in Asymmetric Diluted Network with Low Level of Activity , 1988 .

[21]  H Sompolinsky,et al.  Associative neural network model for the generation of temporal patterns. Theory and application to central pattern generators. , 1988, Biophysical journal.

[22]  M. Tsodyks,et al.  The Enhanced Storage Capacity in Neural Networks with Low Activity Level , 1988 .

[23]  Y. Miyashita,et al.  Neuronal correlate of pictorial short-term memory in the primate temporal cortexYasushi Miyashita , 1988, Nature.

[24]  E. W. Kairiss,et al.  Long-term synaptic potentiation. , 1988, Science.

[25]  E. Gardner.  The space of interactions in neural network models , 1988 .

[26]  Nava Rubin,et al.  Neural networks with low local firing rates , 1989 .

[27]  Martin R. Evans,et al.  Random dilution in a neural network for biased patterns , 1989 .

[28]  A Treves,et al.  Associative memory neural network with low temporal spiking rates. , 1989, Proceedings of the National Academy of Sciences of the United States of America.

[29]  J. Buhmann,et al.  Associative memory with high information content. , 1989, Physical review. A, General physics.

[30]  Alessandro Treves,et al.  Low firing rates: an effective Hamiltonian for excitatory neurons , 1989 .

[31]  Moshe Abeles,et al.  Corticonics: Neural Circuits of Cerebral Cortex , 1991 .