On optimal nonlinear associative recall

The problem of determining the nonlinear function (“blackbox”) which optimally associates (on given criteria) two sets of data is considered. The data are given as discrete, finite column vectors, forming two matricesX (“input”) andY (“output”) with the same numbers of columns and an arbitrary numbers of rows. An iteration method based on the concept of the generalized inverse of a matrix provides the polynomial mapping of degreek onX by whichY is retrieved in an optimal way in the least squares sense. The results can be applied to a wide class of problems since such polynomial mappings may approximate any continuous real function from the “input” space to the “output” space to any required degree of accuracy. Conditions under which the optimal estimate is linear are given. Linear transformations on the input key-vectors and analogies with the “whitening” approach are also discussed. Conditions of “stationarity” on the processes of whichX andY are assumed to represent a set of sample sequences can be easily introduced. The optimal linear estimate is given by a discrete counterpart of the Wiener-Hopf equation and, if the key-signals are noise-like, the holographic-like scheme of associative memory is obtained, as the optimal nonlinear estimator. The theory can be applied to the system identification problem. It is finally suggested that the results outlined here may be relevant to the construction of models of associative, distributed memory.

[1]  J. Doob.  Stochastic processes , 1953 .

[2]  R. Penrose.  A Generalized inverse for matrices , 1955 .

[3]  R. Penrose.  On best approximate solutions of linear matrix equations , 1956, Mathematical Proceedings of the Cambridge Philosophical Society.

[4]  Jacob Katzenelson,et al.  The Design of Nonlinear Filters and Control Systems. Part I , 1962, Inf. Control..

[5]  P. J. V. Heerden.  A New Optical Method of Storing and Retrieving Information , 1963 .

[6]  J. Barrett.  The Use of Functionals in the Analysis of Non-linear Physical Systems† , 1963 .

[7]  P. J. van Heerden,et al.  Theory of Optical Information Storage in Solids , 1963 .

[8]  Y. W. Lee,et al.  Measurement of the Wiener Kernels of a Non-linear System by Cross-correlation† , 1965 .

[9]  H. C. Klonguet-Higgins,et al.  Holographic model of temporal recall. , 1968, Nature.

[10]  D. Gabor.  Associative holographic memories , 1969 .

[11]  D. Marr.  A theory of cerebellar cortex , 1969, The Journal of physiology.

[12]  J. Dieudonne.  Foundations of Modern Analysis , 1969 .

[13]  D. Willshaw,et al.  Theories of associative recall , 1970, Quarterly Reviews of Biophysics.

[14]  D. Marr.  A theory for cerebral neocortex , 1970, Proceedings of the Royal Society of London. Series B. Biological Sciences.

[15]  David Willshaw,et al.  Models of distributed associative memory , 1971 .

[16]  D. Willshaw.  A simple network capable of inductive generalization , 1972, Proceedings of the Royal Society of London. Series B. Biological Sciences.

[17]  James A. Anderson,et al.  A simple neural network generating an interactive memory , 1972 .

[18]  Teuvo Kohonen,et al.  Correlation Matrix Memories , 1972, IEEE Transactions on Computers.

[19]  Teuvo Kohonen,et al.  Representation of Associated Data by Matrix Operators , 1973, IEEE Transactions on Computers.

[20]  Leon N. Cooper,et al.  A possible organization of animal memory and learning , 1973 .

[21]  T. Poggio.  Processing of visual information in flies: from a phenomenological problem towards the nervous mechanisms , 1974 .

[22]  T. Poggio,et al.  Convolution and correlation algebras , 1973, Kybernetik.

[23]  E. Pfaffelhuber.  Correlation memory models — a first approximation in a general learning scheme , 1975, Biological Cybernetics.

[24]  T. Poggio,et al.  Holographic aspects of temporal memory and optomotor responses , 2004, Kybernetik.

[25]  Tomaso Poggio,et al.  The orientation of flies towards visual patterns: On the search for the underlying functional interactions , 1975, Biological Cybernetics.

[26]  J. A. Anderson,et al.  A memory storage model utilizing spatial correlation functions , 1968, Kybernetik.

[27]  T. Poggio,et al.  On holographic models of memory , 1973, Kybernetik.