Bayesian retrieval in associative memories with storage errors

It is well known that for finite-sized networks, onestep retrieval in the autoassociative Willshaw net is a suboptimal way to extract the information stored in the synapses. Iterative retrieval strategies are much better, but have hitherto only had heuristic justification. We show how they emerge naturally from considerations of probabilistic inference under conditions of noisy and partial input and a corrupted weight matrix. We start from the conditional probability distribution over possible patterns for retrieval. This contains all possible information that is available to an observer of the network and the initial input. Since this distribution is over exponentially many patterns, we use it to develop two approximate, but tractable, iterative retrieval methods. One performs maximum likelihood inference to find the single most likely pattern, using the (negative log of the) conditional probability as a Lyapunov function for retrieval. In physics terms, if storage errors are present, then the modified iterative update equations contain an additional antiferromagnetic interaction term and site dependent threshold values. The second method makes a mean field assumption to optimize a tractable estimate of the full conditional probability distribution. This leads to iterative mean field equations which can be interpreted in terms of a network of neurons with sigmoidal responses but with the same interactions and thresholds as in the maximum likelihood update equations. In the absence of storage errors, both models become very similiar to the Willshaw model, where standard retrieval is iterated using a particular form of linear threshold strategy.

[1]  Günther Palm,et al.  Iterative retrieval of sparsely coded associative memory patterns , 1996, Neural Networks.

[2]  Radford M. Neal.  A new view of the EM algorithm that justifies incremental and other variants , 1993 .

[3]  Sompolinsky,et al.  Information storage in neural networks with low levels of activity. , 1987, Physical review. A, General physics.

[4]  Wayne A. Wickelgran.  Context-sensitive coding, associative memory, and serial order in (speech) behavior. , 1969 .

[5]  D. O. Hebb,et al.  The organization of behavior , 1988 .

[6]  Geoffrey E. Hinton,et al.  A View of the Em Algorithm that Justifies Incremental, Sparse, and other Variants , 1998, Learning in Graphical Models.

[7]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[8]  Masahiko Morita,et al.  Associative memory with nonmonotone dynamics , 1993, Neural Networks.

[9]  E. Gardner.  Structure of metastable states in the Hopfield model , 1986 .

[10]  D. J. Wallace,et al.  Enlarging the attractor basins of neural networks with noisy external fields , 1991 .

[11]  Kanter,et al.  Associative recall of memory without errors. , 1987, Physical review. A, General physics.

[12]  M. Tsodyks,et al.  The Enhanced Storage Capacity in Neural Networks with Low Activity Level , 1988 .

[13]  Santosh S. Venkatesh,et al.  The capacity of the Hopfield associative memory , 1987, IEEE Trans. Inf. Theory.

[14]  D. Amit.  The Hebbian paradigm reintegrated: Local reverberations as internal representations , 1995, Behavioral and Brain Sciences.

[15]  A F Gmitro,et al.  Hopfield model associative memory with nonzero-diagonal terms in memory matrix. , 1988, Applied optics.

[16]  Daniel J. Amit,et al.  Neural potentials as stimuli for attractor neural networks , 1990 .

[17]  David J. Willshaw,et al.  Improving recall from an associative memory , 1995, Biological Cybernetics.

[18]  Philippe De Wilde.  The Magnitude of the Diagonal Elements in Neural Networks , 1997, Neural Networks.

[19]  J J Hopfield,et al.  Neurons with graded response have collective computational properties like those of two-state neurons. , 1984, Proceedings of the National Academy of Sciences of the United States of America.

[20]  Peter Whittle,et al.  Artificial memories: Capacity, basis rate and inference , 1997, Neural Networks.

[21]  Teuvo Kohonen,et al.  Self-Organization and Associative Memory , 1988 .

[22]  Jay Buckinghamts.  On setting unit thresholds in an incompletely connected associative net , 1993 .

[23]  U. Rueckert,et al.  TOLERANCE OF A BINARY ASSOCIATIVE MEMORY TOWARDS STUCK-AT-FAULTS , 1991 .

[24]  James A. Anderson,et al.  Neurocomputing: Foundations of Research , 1988 .

[25]  D. J. C. MacKay.  Maximum Entropy Connections: Neural Networks , 1991 .

[26]  Kaoru Nakano,et al.  Associatron-A Model of Associative Memory , 1972, IEEE Trans. Syst. Man Cybern..

[27]  Günther Palm,et al.  Information storage and effective data retrieval in sparse matrices , 1989, Neural Networks.

[28]  John Robinson,et al.  Statistical analysis of the dynamics of a sparse associative memory , 1992, Neural Networks.

[29]  D. Amit,et al.  Statistical mechanics of neural networks near saturation , 1987 .

[30]  Günther Palm,et al.  Information capacity in recurrent McCulloch-Pitts networks with sparsely coded memory states , 1992 .

[31]  Shun-ichi Amari,et al.  Auto-associative memory with two-stage dynamics of nonmonotonic neurons , 1996, IEEE Trans. Neural Networks.

[32]  Toshimichi Saito,et al.  An associative memory including time-variant self-feedback , 1994, Neural Networks.

[33]  Christian Mazza,et al.  On the Storage Capacity of Nonlinear Neural Networks , 1997, Neural Networks.

[34]  Michael Recce,et al.  A search for the optimal thresholding sequence in an associative memory , 1996 .

[35]  E. Gardner.  The space of interactions in neural network models , 1988 .

[36]  Shun-ichi Amari,et al.  Learning Patterns and Pattern Sequences by Self-Organizing Nets of Threshold Elements , 1972, IEEE Transactions on Computers.

[37]  Andreas Engel,et al.  Improved retrieval in neural networks with external fields , 1989 .

[38]  A. R. Gardner-Medwin.  The recall of events through the learning of associations between their parts , 1976, Proceedings of the Royal Society of London. Series B. Biological Sciences.

[39]  H. C. LONGUET-HIGGINS,et al.  Non-Holographic Associative Memory , 1969, Nature.

[40]  Sompolinsky,et al.  Willshaw model: Associative memory with sparse coding and low firing rates. , 1990, Physical review. A, Atomic, molecular, and optical physics.

[41]  Teuvo Kohonen,et al.  Correlation Matrix Memories , 1972, IEEE Transactions on Computers.