Improved bidirectional retrieval of sparse patterns stored by Hebbian learning

The Willshaw model is asymptotically the most efficient neural associative memory (NAM), but its finite version is hampered by high retrieval errors. Iterative retrieval has been proposed in a large number of different models to improve performance in auto-association tasks. In this paper, bidirectional retrieval for the hetero-associative memory task is considered: we define information efficiency as a general performance measure for bidirectional associative memory (BAM) and determine its asymptotic bound for the bidirectional Willshaw model. For the finite Willshaw model, an efficient new bidirectional retrieval strategy is proposed, the appropriate combinatorial model analysis is derived, and implications of the proposed sparse BAM for applications and brain theory are discussed. The distribution of the dendritic sum in the finite Willshaw model given by Buckingham and Willshaw [Buckingham, J., & Willshaw, D. (1992). Performance characteristics of associative nets. Network, 3, 407-414] allows no fast numerical evaluation. We derive a combinatorial formula with a highly reduced evaluation time that is used in the improved error analysis of the basic model and for estimation of the retrieval error in the naive model extension, where bidirectional retrieval is employed in the hetero-associative Willshaw model. The analysis rules out the naive BAM extension as a promising improvement. A new bidirectional retrieval algorithm - called crosswise bidirectional (CB) retrieval - is presented. The cross talk error is significantly reduced without employing more complex learning procedures or dummy augmentation in the pattern coding, as proposed in other refined BAM models [Wang, Y. F., Cruz, J. B., & Mulligan, J. H. (1990). Two coding strategies for bidirectional associative memory. IEEE Trans. Neural Networks, 1(1), 81-92; Leung, C.-S., Chan, L.-W., & Lai, E. (1995). Stability, capacity and statistical dynamics of second-order bidirectional associative memory. IEEE Trans. Syst. Man Cybern., 25(10), 1414-1424]. The improved performance of CB retrieval is shown by a combinatorial analysis of the first step and by simulation experiments: it allows very efficient hetero-associative mapping, as well as auto-associative completion for sparse patterns - the experimentally achieved information efficiency is close to the asymptotic bound. The different retrieval methods in the hetero-associative Willshaw matrix are discussed as Boolean linear optimization problems. The improved BAM model opens interesting new perspectives, for instance, in information retrieval it allows efficient data access providing segmentation of ambiguous user input, relevance feedback and relevance ranking. Finally, we discuss BAM models as functional models for reciprocal cortico-cortical pathways, and the implication of this for a more flexible version of Hebbian cell-assemblies.

[1]  David J. Willshaw,et al.  Improving recall from an associative memory , 1995, Biological Cybernetics.

[2]  Michael Recce,et al.  A search for the optimal thresholding sequence in an associative memory , 1996 .

[3]  Robert Hecht-Nielsen,et al.  A BAM with increased information storage capacity , 1988, IEEE 1988 International Conference on Neural Networks.

[4]  Günther Palm,et al.  Bidirectional Retrieval from Associative Memory , 1997, NIPS.

[5]  Andrew Chi-Sing Leung,et al.  Stability, capacity, and statistical dynamics of second-order bidirectional associative memory , 1995, IEEE Trans. Syst. Man Cybern..

[6]  Mohamad H. Hassoun Dynamic heteroassociative neural memories , 1989, Neural Networks.

[7]  Jerry L. Potter Associative Computing , 1992, Frontiers of Computer Science.

[8]  Günther Palm,et al.  On the asymptotic information storage capacity of neural networks , 1988 .

[9]  D. Amit The Hebbian paradigm reintegrated: Local reverberations as internal representations , 1995, Behavioral and Brain Sciences.

[10]  T. M. Mayhew,et al.  Anatomy of the Cortex: Statistics and Geometry. , 1991 .

[11]  Günther Palm,et al.  Information storage and effective data retrieval in sparse matrices , 1989, Neural Networks.

[12]  Mnh,et al.  Histologie du Système Nerveux de Lʼhomme et des Vertébrés , 1998 .

[13]  R. Eckmiller,et al.  Neural Computers , 1989, Springer Study Edition.

[14]  P. Goldman-Rakic,et al.  Preface: Cerebral Cortex Has Come of Age , 1991 .

[15]  Friedrich Gebhardt Text signatures by superimposed coding of letter triplets and quadruplets , 1987, Inf. Syst..

[16]  Peter Dayan,et al.  Bayesian retrieval in associative memories with storage errors , 1998, IEEE Trans. Neural Networks.

[17]  Stephen K. Reed,et al.  Book reviewIntelligence, information processing, and analogical reasoning: by Robert J. Sternberg. Hillsdale, New Jersey: Lawrence Erlbaum Associates, 1977. xi + 348 pp. $19.95 , 1977 .

[18]  J. Austin Associative memory , 1987 .

[19]  K. Vairavan,et al.  Performance Characteristics of a , 1995 .

[20]  Jay Buckinghamts On setting unit thresholds in an incompletely connected associative net , 1993 .

[21]  Günther Palm,et al.  Information capacity in recurrent McCulloch-Pitts networks with sparsely coded memory states , 1992 .

[22]  J. L. van Hemmen,et al.  Increasing the efficiency of a neural network through unlearning , 1990 .

[23]  D. J. Felleman,et al.  Distributed hierarchical processing in the primate cerebral cortex. , 1991, Cerebral cortex.

[24]  Prof. Dr. Valentino Braitenberg,et al.  Anatomy of the Cortex , 1991, Studies of Brain Function.

[25]  A. R. Gardner-Medwin The recall of events through the learning of associations between their parts , 1976, Proceedings of the Royal Society of London. Series B. Biological Sciences.

[26]  G. Hartmann,et al.  Parallel Processing in Neural Systems and Computers , 1990 .

[27]  Jose B. Cruz,et al.  Two coding strategies for bidirectional associative memory , 1990, IEEE Trans. Neural Networks.

[28]  Günther Palm,et al.  Bidirectional completion of cell assemblies in the cortex , 1998 .

[29]  G. A. Kohring Performance enhancement of Willshaw type networks through the use of limit cycles , 1990 .

[30]  B Kosko,et al.  Adaptive bidirectional associative memories. , 1987, Applied optics.

[31]  H. C. LONGUET-HIGGINS,et al.  Non-Holographic Associative Memory , 1969, Nature.

[32]  John Robinson,et al.  Statistical analysis of the dynamics of a sparse associative memory , 1992, Neural Networks.

[33]  Friedrich T. Sommer,et al.  Associative Data Storage and Retrieval in Neural Networks , 1996 .

[34]  Thomas Wennekers,et al.  Gamma-oscillations support optimal retrieval in associative memories of two-compartment neurons , 1999, Neurocomputing.

[35]  Günther Palm,et al.  Iterative retrieval of sparsely coded associative memory patterns , 1996, Neural Networks.

[36]  James M. Bower,et al.  Computational Neuroscience: Trends in Research , 1996 .

[37]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[38]  D. Signorini,et al.  Neural networks , 1995, The Lancet.

[39]  A. Krikelis,et al.  Associative processing and processors , 1994, Computer.

[40]  Francis Crick,et al.  REM sleep and neural nets , 1995, Behavioural Brain Research.