Mutual Information of Three-State Low Activity Diluted Neural Networks with Self-Control

The influence of a macroscopic time-dependent threshold on the retrieval process of three-state extremely diluted neural networks is examined. If the threshold is chosen appropriately in function of the noise and the pattern activity of the network, adapting itself in the course of the time evolution, it guarantees an autonomous functioning of the network. It is found that this self-control mechanism considerably improves the retrieval quality, especially in the limit of low activity, including the storage capacity, the basins of attraction and the information content. The mutual information is shown to be the relevant parameter to study the retrieval quality of such low activity models. Numerical results confirm these observations.

[1]  M. Tsodyks.  Associative Memory in Asymmetric Diluted Network with Low Level of Activity , 1988 .

[2]  Horn,et al.  Neural networks with dynamical thresholds. , 1989, Physical review. A, General physics.

[3]  Shun-ichi Amari,et al.  Characteristics of sparsely encoded associative memory , 1989, Neural Networks.

[4]  D. O. Hebb,et al.  The organization of behavior , 1988 .

[5]  Simon Schultz,et al.  Stability of the replica symmetric solution for the information conveyed by a neural network , 1998 .

[6]  Masato Okada,et al.  Notions of Associative Memory and Sparse Coding , 1996, Neural Networks.

[7]  Désiré Bollé,et al.  On the parallel dynamics of theQ-state Potts andQ-Ising neural networks , 1993 .

[8]  S. Kaplan.  The Physiology of Thought , 1950 .

[9]  Günther Palm,et al.  Iterative retrieval of sparsely coded associative memory patterns , 1996, Neural Networks.

[10]  Engel,et al.  Basin of attraction in networks of multistate neurons. , 1993, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[11]  Anders Krogh,et al.  Introduction to the theory of neural computation , 1994, The advanced book program.

[12]  Jeferson Jacob Arenzon,et al.  Simulating highly diluted neural networks , 1994 .

[13]  Shun-ichi Amari,et al.  Statistical neurodynamics of associative memory , 1988, Neural Networks.

[14]  C. J. Perez-Vicente.  Sparse Coding and Information in Hebbian Neural Networks , 1989 .

[15]  J. Nadal,et al.  Nonlinear feedforward networks with stochastic outputs: infomax implies redundancy reduction. , 1998, Network.

[16]  J. Buhmann,et al.  Associative memory with high information content. , 1989, Physical review. A, General physics.

[17]  Désiré Bollé,et al.  Retrieval and chaos in extremely dilutedQ-Ising neural networks , 1994 .

[18]  H. Horner.  Neural networks with low levels of activity: Ising vs. McCulloch-Pitts neurons , 1989 .

[19]  Désiré Bollé,et al.  SELF-CONTROL IN SPARSELY CODED NETWORKS , 1998 .

[20]  Jonathan S. Yedidia.  Neural networks that use three-state neurons , 1989 .

[21]  Richard E. Blahut,et al.  Principles and practice of information theory , 1987 .

[22]  Masato Okada,et al.  A hierarchy of macrodynamical equations for associative memory , 1995, Neural Networks.

[23]  Katsunori Kitano,et al.  LETTER TO THE EDITOR: Retrieval dynamics of neural networks for sparsely coded sequential patterns , 1998, cond-mat/9805135.

[24]  E. Gardner,et al.  An Exactly Solvable Asymmetric Neural Network Model , 1987 .

[25]  Sompolinsky,et al.  Information storage in neural networks with low levels of activity. , 1987, Physical review. A, General physics.

[26]  H. C. LONGUET-HIGGINS,et al.  Non-Holographic Associative Memory , 1969, Nature.