The space of interactions in neural network models

The typical fraction of the space of interactions between each pair of N Ising spins which solve the problem of storing a given set of p random patterns as N-bit spin configurations is considered. The volume is calculated explicitly as a function of the storage ratio, alpha =p/N, of the value kappa (>0) of the product of the spin and the magnetic field at each site and of the magnetisation, m. Here m may vary between 0 (no correlation) and 1 (completely correlated). The capacity increases with the correlation between patterns from alpha =2 for correlated patterns with kappa =0 and tends to infinity as m tends to 1. The calculations use a saddle-point method and the order parameters at the saddle point are assumed to be replica symmetric. This solution is shown to be locally stable. A local iterative learning algorithm for updating the interactions is given which will converge to a solution of given kappa provided such solutions exist.

[1]  Thomas M. Cover,et al.  Geometrical and Statistical Properties of Systems of Linear Inequalities with Applications in Pattern Recognition , 1965, IEEE Trans. Electron. Comput..

[2]  H. C. LONGUET-HIGGINS,et al.  Non-Holographic Associative Memory , 1969, Nature.

[3]  S. Kirkpatrick,et al.  Solvable Model of a Spin-Glass , 1975 .

[4]  S. Edwards,et al.  Theory of spin glasses , 1975 .

[5]  D. Thouless,et al.  Stability of the Sherrington-Kirkpatrick solution of a spin glass model , 1978 .

[6]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[7]  I. Guyon,et al.  Information storage and retrieval in spin-glass like neural networks , 1985 .

[8]  Sompolinsky,et al.  Storing infinite numbers of patterns in a spin-glass model of neural networks. , 1985, Physical review letters.

[9]  Sompolinsky,et al.  Spin-glass models of neural networks. , 1985, Physical review. A, General physics.

[10]  Marc Mézard,et al.  Solvable models of working memories , 1986 .

[11]  Kanter,et al.  Temporal association in asymmetric neural networks. , 1986, Physical review letters.

[12]  K. Schilling,et al.  Lattice Gauge Theory: A Challenge in Large-Scale Computing , 1986 .

[13]  E. Gardner.  Structure of metastable states in the Hopfield model , 1986 .

[14]  Baldi,et al.  Number of stable points for spin-glasses and neural networks of higher orders. , 1987, Physical review letters.

[15]  D. Amit,et al.  Statistical mechanics of neural networks near saturation , 1987 .

[16]  W. Krauth,et al.  Learning algorithms with optimal stability in neural networks , 1987 .

[17]  E. Gardner.  Maximum Storage Capacity in Neural Networks , 1987 .

[18]  Opper,et al.  Learning of correlated patterns in spin-glass networks by local learning rules. , 1987, Physical review letters.

[19]  D. J. Wallace,et al.  Learning and memory properties in fully connected networks , 1987 .

[20]  S. Venkatesh.  Epsilon capacity of neural networks , 1987 .

[21]  D. J. Wallace,et al.  Dynamics and statistical mechanics of the Hopfield model , 1987 .

[22]  P. Leath,et al.  The failure distribution in percolation models of breakdown , 1987 .

[23]  Kanter,et al.  Associative recall of memory without errors. , 1987, Physical review. A, General physics.

[24]  E. Gardner,et al.  Optimal storage properties of neural network models , 1988 .

[25]  B. Forrest.  Content-addressability and learning in neural networks , 1988 .

[26]  Teuvo Kohonen,et al.  Self-Organization and Associative Memory, Third Edition , 1989, Springer Series in Information Sciences.