Information storage in sparsely coded memory nets

We study simple, feedforward, neural networks for pattern storage and retrieval, with information theory criteria. Two Hebbian learning rules are considered, with emphasis on sparsely coded patterns. We address the question: under which conditions is the optimal information storage reached in the error-full regime?For the model introduced some time ago by Willshaw, Buneman and Longuet-Higgins, the information stored goes through a maximum, which may be found within the error-less or the error-full regimes according to the value of the coding rate. However, it eventually vanishes as learning goes on and more patterns are stored.For the original Hebb learning rule, where reinforcement occurs whenever both input and output neurons are active, the information stored reaches a stationary value, 1/(π ln 2), when the net is overloaded beyond its threshold for errors. If the coding rate f′ of the output pattern is small enough, the information storage goes through a maximum, which saturates the Gardner bound, 1/(...

[1]  G Palm.  Computing with neural networks. , 1987, Science.

[2]  Schulten,et al.  Associative memory with high information content. , 1989, Physical review. A, General physics.

[3]  Haim Sompolinsky,et al.  STATISTICAL MECHANICS OF NEURAL NETWORKS , 1988 .

[4]  D. Amit,et al.  Optimised network for sparsely coded patterns , 1989 .

[5]  H. C. LONGUET-HIGGINS,et al.  Non-Holographic Associative Memory , 1969, Nature.

[6]  M. Tsodyks.  Associative Memory in Asymmetric Diluted Network with Low Level of Activity , 1988 .

[7]  Günther Palm,et al.  Brain Theory - Reprint Volume , 1988 .

[8]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[9]  Sompolinsky,et al.  Information storage in neural networks with low levels of activity. , 1987, Physical review. A, General physics.

[10]  David L. Goodstein.  Richard P. Feynman, Teacher , 1989 .

[11]  Werner Krauth,et al.  Critical storage capacity of the J = ± 1 neural network , 1989 .

[12]  M. Tsodyks,et al.  The Enhanced Storage Capacity in Neural Networks with Low Activity Level , 1988 .

[13]  E. Gardner.  The space of interactions in neural network models , 1988 .

[14]  Günther Palm.  On the asymptotic information storage capacity of neural networks , 1988 .

[15]  E. Gardner,et al.  Optimal storage properties of neural network models , 1988 .

[16]  D. O. Hebb,et al.  The organization of behavior , 1988 .

[17]  E. Gardner.  Maximum Storage Capacity in Neural Networks , 1987 .

[18]  E. Gardner,et al.  Three unfinished works on the optimal storage capacity of networks , 1989 .