Natural Language Processing With Modular PDP Networks and Distributed Lexicon

An approach to connectionist natural language processing is proposed, which is based on hierarchically organized modular parallel distributed processing (PDP) networks and a central lexicon of distributed input/output representations. The modules communicate using these representations, which are global and publicly available in the system. The representations are developed automatically by all networks while they are learning their processing tasks. The resulting representations reflect the regularities in the subtasks, which facilitates robust processing in the face of noise and damage, supports improved generalization, and provides expectations about possible contexts. The lexicon can be extended by cloning new instances of the items, that is, by generating a number of items with known processing properties and distinct identities. This technique combinatorially increases the processing power of the system. The recurrent FGREP module, together with a central lexicon, is used as a basic building block in modeling higher level natural language tasks. A single module is used to form case-role representations of sentences from word-by-word sequential natural language input. A hierarchical organization of four recurrent FGREP modules (the DISPAR system) is trained to produce fully expanded paraphrases of script-based stories, where unmentioned events and role fillers are inferred.

[1]  H. Ritter.  Combining self-organizing maps , 1989, International 1989 Joint Conference on Neural Networks.

[2]  Michael I. Jordan,et al.  Task Decomposition through Competition in A , 1990 .

[3]  Richard Edward Cullingford,et al.  Script application: computer understanding of newspaper stories. , 1977 .

[4]  Teuvo Kohonen,et al.  Self-Organization and Associative Memory , 1988 .

[5]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[6]  Michael G. Dyer,et al.  Argument comprehension and retrieval for editorial text , 1990, Knowl. Based Syst..

[7]  Bruce G. Buchanan,et al.  The MYCIN Experiments of the Stanford Heuristic Programming Project , 1985 .

[8]  Charles P. Dolan,et al.  Implementing a Connectionist Production System Using Tensor Products ; CU-CS-411-88 , 1988 .

[9]  Geoffrey E. Hinton,et al.  The appeal of parallel distributed processing , 1986 .

[10]  C. P. Dolan.  Tensor manipulation networks: connectionist and symbolic approaches to comprehension, learning, and planning , 1989 .

[11]  James L. McClelland,et al.  Learning and Applying Contextual Constraints in Sentence Comprehension , 1990, Artif. Intell..

[12]  Logical Abilities in Children , 1974 .

[13]  R. Schank,et al.  Inside Computer Understanding , 1981 .

[14]  David S. Touretzky.  Connectionism and Compositional Semantics , 1989 .

[15]  Ii Gerald Francis Dejong.  Skimming stories in real time: an experiment in integrated understanding. , 1979 .

[16]  Ajay N. Jain.  A connectionist architecture for sequential symbolic domains , 1989 .

[17]  Richard E. Korf,et al.  Planning as Search: A Quantitative Approach , 1987, Artif. Intell..

[18]  Geoffrey E. Hinton.  Learning distributed representations of concepts. , 1989 .

[19]  James L. McClelland,et al.  On learning the past-tenses of English verbs: implicit rules or parallel distributed processing , 1986 .

[20]  Paul Smolensky,et al.  Virtual Memories and Massive Generalization in Connectionist Combinatorial Learning ; CU-CS-431-89 , 1989 .

[21]  M R Quillian,et al.  Word concepts: a theory and simulation of some basic semantic capabilities. , 1967, Behavioral science.

[22]  R. Miikkulainen,et al.  A modular neural network architecture for sequential paraphrasing of script-based stories , 1989, International 1989 Joint Conference on Neural Networks.

[23]  G. Reeke.  The society of mind , 1991 .

[24]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[25]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[26]  Herbert A. Simon,et al.  The Sciences of the Artificial , 1970 .

[27]  Jerome A. Feldman,et al.  Neural Representation of Conceptual Knowledge. , 1986 .

[28]  M. Dyer.  In-Depth Understanding: A Computer Model of Integrated Processing for Narrative Comprehension , 1983 .

[29]  James L. McClelland,et al.  Mechanisms of Sentence Processing: Assigning Roles to Constituents of Sentences , 1986 .

[30]  Risto Miikkulainen,et al.  Discern: a distributed artificial neural network model of script processing and memory , 1990 .

[31]  Dana H. Ballard,et al.  Modular Learning in Neural Networks , 1987, AAAI.

[32]  Vincent Kanade,et al.  Clustering Algorithms , 2021, Wireless RF Energy Transfer in the Massive IoT Era.

[33]  James L. McClelland,et al.  Parallel Distributed Processing: Explorations in the Microstructure of Cognition : Psychological and Biological Models , 1986 .

[34]  Noel E. Sharkey,et al.  Mixing Binary and Continuous Connection Schemes for Knowledge Access , 1986, AAAI.

[35]  R. Miikkulainen,et al.  Forming global representations with extended backpropagation , 1988, IEEE 1988 International Conference on Neural Networks.

[36]  Michael I. Jordan,et al.  Task Decomposition Through Competition in a Modular Connectionist Architecture: The What and Where Vision Tasks , 1990, Cogn. Sci..

[37]  Risto Miikkulainen,et al.  A Pdp Architecture for Processing Sentences With Relative Clauses , 1990, COLING.

[38]  Risto Miikkulainen,et al.  Script Recognition with Hierarchical Feature Maps , 1992 .

[39]  James L. McClelland,et al.  Learning Subsequential Structure in Simple Recurrent Networks , 1988, NIPS.

[40]  Michael I. Jordan.  Attractor dynamics and parallelism in a connectionist sequential machine , 1990 .

[41]  Terrence J. Sejnowski,et al.  Parallel Networks that Learn to Pronounce English Text , 1987, Complex Syst..

[42]  Geoffrey E. Hinton,et al.  A general framework for parallel distributed processing , 1986 .

[43]  John B. Black,et al.  Scripts in memory for text , 1979, Cognitive Psychology.

[44]  Roger C. Schank,et al.  Scripts, plans, goals and understanding: an inquiry into human knowledge structures , 1978 .

[45]  Michael G. Dyer,et al.  Storing and Generalizing Multiple Instances While Maintaining Knowledge-Level Parallelism , 1989, IJCAI.