Tensor Product Variable Binding and the Representation of Symbolic Structures in Connectionist Systems

Abstract A general method, the tensor product representation, is defined for the connectionist representation of value/variable bindings. The technique is a formalization of the idea that a set of value/variable pairs can be represented by accumulating activity in a collection of units each of which computes the product of a feature of a variable and a feature of its value. The method allows the fully distributed representation of bindings and symbolic structures. Fully and partially localized special cases of the tensor product representation reduce to existing cases of connectionist representations of structured data. The representation rests on a principled analysis of structure; it saturates gracefully as larger structures are represented; it permits recursive construction of complex representations from simpler ones; it respects the independence of the capacities to generate and maintain multiple bindings in parallel; it extends naturally to continuous structures and continuous representational patterns; it permits values to also serve as variables; and it enables analysis of the interference of symbolic structures stored in associative memories. It has also served as the basis for working connectionist models of high-level cognitive tasks.

[1]  James L. McClelland,et al.  On learning the past-tenses of English verbs: implicit rules or parallel distributed processing , 1986 .

[2]  Donald A. Norman,et al.  The perception of multiple objects: a parallel, distributed processing approach , 1987 .

[3]  Geoffrey E. Hinton,et al.  Distributed Representations , 1986, The Philosophy of Artificial Intelligence.

[4]  Paul Smolensky.  Neural and Conceptual Interpretations of Parallel Distributed Processing Models , 1986 .

[5]  Tunc Geveci,et al.  Advanced Calculus , 2014, Nature.

[6]  Jerome A. Feldman,et al.  Neural Representation of Conceptual Knowledge. , 1986 .

[7]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[8]  Geoffrey E. Hinton.  A Parallel Computation that Assigns Canonical Object-Based Frames of Reference , 1981, IJCAI.

[9]  James L. McClelland,et al.  An interactive activation model of context effects in letter perception: Part 2. The contextual enhancement effect and some tests and extensions of the model. , 1982, Psychological review.

[10]  David S. Touretzky,et al.  A distributed connectionist representation for concept structures , 1987 .

[11]  F. W. Warner.  Foundations of Differentiable Manifolds and Lie Groups , 1971 .

[12]  James L. McClelland.  The programmable blackboard model of reading , 1986 .

[13]  Geoffrey E. Hinton,et al.  Learning Representations by Recirculation , 1987, NIPS.

[14]  J. Fodor,et al.  Connectionism and cognitive architecture: A critical analysis , 1988, Cognition.

[15]  C. P. Dolan.  Tensor manipulation networks: connectionist and symbolic approaches to comprehension, learning, and planning , 1989 .

[16]  Geoffrey E. Hinton,et al.  GEMINI: Gradient Estimation Through Matrix Inversion After Noise Injection , 1988, NIPS.

[17]  James L. McClelland,et al.  Mechanisms of Sentence Processing: Assigning Roles to Constituents of Sentences , 1986 .

[18]  P. Smolensky.  THE CONSTITUENT STRUCTURE OF CONNECTIONIST MENTAL STATES: A REPLY TO FODOR AND PYLYSHYN , 2010 .

[19]  S. Pinker,et al.  On language and connectionism: Analysis of a parallel distributed processing model of language acquisition , 1988, Cognition.

[20]  A. Treisman,et al.  Illusory conjunctions in the perception of objects , 1982, Cognitive Psychology.

[21]  Mark A. Fanty,et al.  Context-free parsing with connectionist networks , 1987 .

[22]  James L. McClelland,et al.  Parallel Distributed Processing: Explorations in the Microstructure of Cognition : Psychological and Biological Models , 1986 .

[23]  P. Smolensky.  Connectionism, Constituency, and the Language of Thought ; CU-CS-416-88 , 1988 .

[24]  Paul Smolensky,et al.  Information processing in dynamical systems: foundations of harmony theory , 1986 .

[25]  James L. McClelland,et al.  An interactive activation model of context effects in letter perception: I. An account of basic findings. , 1981 .

[26]  T. Bever,et al.  The relation between linguistic structure and associative theories of language learning—A constructive critique of some connectionist learning models , 1988, Cognition.

[27]  Jeffrey L. Elman,et al.  Interactive processes in speech perception: the TRACE model , 1986 .

[28]  W. Freeman.  Second Commentary: On the proper treatment of connectionism by Paul Smolensky (1988) - Neuromachismo Rekindled , 1989 .

[29]  Geoffrey E. Hinton,et al.  Symbols Among the Neurons: Details of a Connectionist Inference Architecture , 1985, IJCAI.

[30]  Geoffrey E. Hinton,et al.  A general framework for parallel distributed processing , 1986 .

[31]  J. Feldman.  Four frames suffice: A provisional model of vision and space , 1985, Behavioral and Brain Sciences.

[32]  Clarence E. Rose,et al.  What is tensor analysis? , 1938, Electrical Engineering.

[33]  Terrence J. Sejnowski,et al.  Parallel Networks that Learn to Pronounce English Text , 1987, Complex Syst..

[34]  Charles P. Dolan,et al.  Tensor Product Production System: a Modular Architecture and Representation , 1989 .

[35]  P. Smolensky.  On variable binding and the representation of symbolic structures in connectionist systems , 1987 .

[36]  Charles P. Dolan,et al.  Implementing a Connectionist Production System Using Tensor Products ; CU-CS-411-88 , 1988 .

[37]  Bernard Widrow,et al.  Adaptive switching circuits , 1988 .

[38]  James L. McClelland,et al.  An interactive activation model of context effects in letter perception: part 1.: an account of basic findings , 1988 .

[39]  Michael I. Jordan.  An introduction to linear algebra in parallel distributed processing , 1986 .