Sparsely coded associative memories: capacity and dynamical properties

We consider very sparsely coded associative memories of binary neurons, for both Hebbian and covariant learning rules. We calculate explicitly their maximal capacity both in terms of patterns, and in terms of information content, taking into account the correlation of local fields, and we investigate its dependence on the degree of sparsity. The sparseness of the coding enhances both the memory capacity and the information capacity, whatever the chosen scheme. The study of the retrieval dynamics shows that, as soon as the number of patterns stored exceeds some critical value, retrieval becomes limited to the states with the same activity as the prototype patterns.

[1]  H. C. LONGUET-HIGGINS,et al.  Non-Holographic Associative Memory , 1969, Nature.

[2]  L. Nadel,et al.  The Hippocampus as a Cognitive Map , 1978 .

[3]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[4]  R. Traub,et al.  Models of synchronized hippocampal bursts in the presence of inhibition. I. Single population events. , 1987, Journal of neurophysiology.

[5]  Dennis K. Branstad.  Considerations for security in the OSI architecture , 1987, IEEE Network.

[6]  Santosh S. Venkatesh,et al.  The capacity of the Hopfield associative memory , 1987, IEEE Trans. Inf. Theory.

[7]  M. Tsodyks,et al.  The Enhanced Storage Capacity in Neural Networks with Low Activity Level , 1988 .

[8]  E. Gardner.  The space of interactions in neural network models , 1988 .

[9]  Günther Palm,et al.  On the asymptotic information storage capacity of neural networks , 1988 .

[10]  E. Rolls.  The representation and storage of information in neural networks in the primate cerebral cortex and hippocampus , 1989 .

[11]  Shun-ichi Amari,et al.  Characteristics of sparsely encoded associative memory , 1989, Neural Networks.

[12]  H. Horner,et al.  Transients and basins of attraction in neutral network models , 1989 .

[13]  D. Amit,et al.  Optimised network for sparsely coded patterns , 1989 .

[14]  J. Buhmann,et al.  Associative memory with high information content. , 1989, Physical review. A, General physics.

[15]  H. Horner.  Neural networks with low levels of activity: Ising vs. McCulloch-Pitts neurons , 1989 .

[16]  Sompolinsky,et al.  Willshaw model: Associative memory with sparse coding and low firing rates. , 1990, Physical review. A, Atomic, molecular, and optical physics.

[17]  Jean-Pierre Nadal,et al.  Information storage in sparsely coded memory nets , 1990 .

[18]  Shun-ichi Amari,et al.  Mathematical foundations of neurocomputing , 1990, Proc. IEEE.