Auto-associative memory with two-stage dynamics of nonmonotonic neurons

Dynamical properties of a neural auto-associative memory with two-stage neurons are investigated theoretically. The two-stage neuron is a model whose output is determined by a two-stage nonlinear function of the internal field of the neuron (internal field is a weighted sum of outputs of the other neurons). The model is general, including nonmonotonic neurons as well as monotonic ones. Recent studies on associative memory revealed superiority of nonmonotonic neurons to monotonic ones. The present paper supplies theoretical verification on the high performance of nonmonotonic neurons and proves that the capacity of the auto-associative memory with two-stage neurons is O(n/ radicallog n), in contrast to O(n/log n) of simple threshold neurons. There is also a discussion of recall processes, where the radius of basin of attraction of memorized patterns is clarified. An intuitive explanation on why the performance is improved by nonmonotonic neurons is also provided by showing the correspondence of the recall processes of the two-stage-neuron net and orthogonal learning.

[1]  Teuvo Kohonen,et al.  Self-Organization and Associative Memory , 1988 .

[2]  Masato Okada,et al.  Statistical Neurodynamics of the Sparsely Encoded Associative Memory , 1996 .

[3]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[4]  Masato Okada,et al.  A hierarchy of macrodynamical equations for associative memory , 1995, Neural Networks.

[5]  Yasuji Sawada,et al.  Associative memory network composed of neurons with hysteretic property , 1990, Neural Networks.

[6]  H. Yanai,et al.  Integrator neurons for analog neural networks , 1990 .

[7]  H.-F. Yanai,et al.  A theory on a neural net with nonmonotone neurons , 1993, IEEE International Conference on Neural Networks.

[8]  Hidetoshi Nishimori,et al.  Retrieval process of an associative memory with a general input-output function , 1993, Neural Networks.

[9]  Shiino,et al.  Self-consistent signal-to-noise analysis of the statistical behavior of analog neural networks and enhancement of the storage capacity. , 1993, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[10]  Masahiko Morita,et al.  Associative memory with nonmonotone dynamics , 1993, Neural Networks.

[11]  S. Yoshizawa,et al.  Dynamics of an auto-associative neural network model with arbitrary connectivity and noise in the threshold , 1991 .

[12]  Kei Kobayashi.  On the capacity of a neuron with a non-monotone output function , 1991 .

[13]  Sompolinsky,et al.  Storing infinite numbers of patterns in a spin-glass model of neural networks. , 1985, Physical review letters.

[14]  E. Gardner.  The space of interactions in neural network models , 1988 .

[15]  Shun-ichi Amari,et al.  Learning Patterns and Pattern Sequences by Self-Organizing Nets of Threshold Elements , 1972, IEEE Transactions on Computers.

[16]  Santosh S. Venkatesh,et al.  The capacity of the Hopfield associative memory , 1987, IEEE Trans. Inf. Theory.

[17]  J. Austin.  Associative memory , 1987 .

[18]  M. Shiino,et al.  Self-consistent signal-to-noise analysis and its application to analogue neural networks with asymmetric connections , 1992 .

[19]  E. Gardner,et al.  Zero temperature parallel dynamics for infinite range spin glasses and neural networks , 1987 .

[20]  S. Amari,et al.  Sparsely coded associative memories: capacity and dynamical properties , 1991 .

[21]  Shun-ichi Amari,et al.  Characteristics of sparsely encoded associative memory , 1989, Neural Networks.

[22]  Masahiko Morita,et al.  Capacity of associative memory using a nonmonotonic neuron model , 1993, Neural Networks.