Toward a connectionist model of recursion in human linguistic performance

Naturally occurring speech contains only a limited amount of complex recursive structure, and this is reflected in the empirically documented difficulties that people experience when processing such structures. We present a connectionist model of human performance in processing recursive language structures. The model is trained on simple artificial languages. We find that the qualitative performance profile of the model matches human behavior, both on the relative difficulty of center-embedding and cross-dependency, and between the processing of these complex recursive structures and right-branching recursive constructions; We analyze how these differences in performance are reflected in the internal representations of the model by performing discriminant analyses on these representations both before and after training. Furthermore, we show how a network trained to process recursive structures can also generate such structures in a probabilistic fashion. This work suggests a novel explanation of people's limited recursive performance, without assuming the existence of a mentally represented competence grammar allowing unbounded recursion.

[1]  Jordan B. Pollack,et al.  Recursive Distributed Representations , 1990, Artif. Intell..

[2]  Mitchell P. Marcus,et al.  A theory of syntactic recognition for natural language , 1979 .

[3]  Walter S. Stolz,et al.  A study of the ability to decode grammatically novel sentences , 1967 .

[4]  Morten H. Christiansen,et al.  Learning to Segment Speech Using Multiple Cues: A Connectionist Model , 1998 .

[5]  P. A. Reich The Finiteness of Natural Language , 1969 .

[6]  P. Niyogi,et al.  A language learning model for finite parameter spaces , 1996, Cognition.

[7]  Nick Chater,et al.  Distributional Information: A Powerful Cue for Acquiring Syntactic Categories , 1998, Cogn. Sci..

[8]  J. Kimball Seven principles of surface structure parsing in natural language , 1973 .

[9]  Garrison W. Cottrell,et al.  Toward Connectionist Parsing , 1982, AAAI.

[10]  Paul Smolensky Grammar-based connectionist approaches to language , 1999 .

[11]  C. Lee Giles,et al.  Learning and Extracting Finite State Automata with Second-Order Recurrent Neural Networks , 1992, Neural Computation.

[12]  Janet Wiles,et al.  Operators and Curried Functions: Training and Analysis of Simple Recurrent Networks , 1991, NIPS.

[13]  G. Miller Some psychological studies of grammar. , 1962 .

[14]  Dennis Norris,et al.  A dynamic-net model of human speech recognition , 1991 .

[15]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[16]  N. Chater,et al.  AUTONOMY, IMPLEMENTATION AND COGNITIVE ARCHITECTURE - A REPLY , 1990 .

[17]  J. Elman Learning and development in neural networks: the importance of starting small , 1993, Cognition.

[18]  Noam Chomsky,et al.  वाक्यविन्यास का सैद्धान्तिक पक्ष = Aspects of the theory of syntax , 1965 .

[19]  Frederick Jelinek,et al.  Basic Methods of Probabilistic Context Free Grammars , 1992 .

[20]  James L. McClelland,et al.  Finite State Automata and Simple Recurrent Networks , 1989, Neural Computation.

[21]  Janet Wiles,et al.  Intersecting Regions: The Key to Combinatorial Structure in Hidden Unit Space , 1992, NIPS.

[22]  Jeffrey L. Elman,et al.  Distributed Representations, Simple Recurrent Networks, and Grammatical Structure , 1991, Mach. Learn..

[23]  Gary S. Dell,et al.  Connectionist models of language production: lexical access and grammatical encoding , 1999, Cogn. Sci..

[24]  Robert F. Hadley Systematicity in Connectionist Language Learning , 1994 .

[25]  Robert C. Berwick,et al.  The Grammatical Basis of Linguistic Performance: Language Use and Acquisition , 1986 .

[26]  E. Gibson,et al.  Memory Limitations and Structural Forgetting: The Perception of Complex Ungrammatical Sentences as Grammatical , 1999 .

[27]  Aravind K. Joshi,et al.  Processing crossed and nested dependencies: An automation perspective on the psycholinguistic results , 1990 .

[28]  Donald J. Foss,et al.  Some effects of memory limitation upon sentence comprehension and recall , 1970 .

[29]  Allen Newell,et al.  Computer science as empirical inquiry: symbols and search , 1976, CACM.

[30]  Eric Brill,et al.  Deducing linguistic structure from the statistics of large corpora , 1990 .

[31]  D. M. Green,et al.  Signal detection theory and psychophysics , 1966 .

[32]  Stuart M. Shieber,et al.  Evidence against the context-freeness of natural language , 1985 .

[33]  Noam Chomsky Review of B.F. Skinner, Verbal Behavior , 1959 .

[34]  Michael C. Mozer,et al.  Connectionist Music Composition Based on Melodic and Stylistic Constraints , 1990, NIPS.

[35]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[36]  Mark A. Fanty,et al.  Context-free parsing with connectionist networks , 1987 .

[37]  M. Just,et al.  From the SelectedWorks of Marcel Adam Just 1992 A capacity theory of comprehension : Individual differences in working memory , 2017 .

[38]  T. Gelder,et al.  On Being Systematically Connectionist , 1994 .

[39]  N. Chater,et al.  Autonomy, implementation and cognitive architecture: A reply to Fodor and Pylyshyn , 1990, Cognition.

[40]  M. Just,et al.  Individual differences in syntactic processing: The role of working memory , 1991 .

[41]  Kenneth Ward Church,et al.  Poor Estimates of Context are Worse than None , 1990, HLT.

[42]  Norman Cliff,et al.  Analyzing Multivariate Data , 1987 .

[43]  Janet D. Fodor,et al.  The sausage machine: A new two-stage parsing model , 1978, Cognition.

[44]  Carl Vogel,et al.  Cross-Serial Dependencies Are Not Hard to Process , 1996, COLING.

[45]  Nick Chater,et al.  FINDING LINGUISTIC STRUCTURE WITH RECURRENT NEURAL NETWORKS , 1992 .

[46]  Eugene Charniak,et al.  Statistical language learning , 1997 .

[47]  William D. Marslen-Wilson,et al.  Crossed and nested dependencies in German and Dutch , 1986 .

[48]  George A. Miller,et al.  Free Recall of Self-Embedded English Sentences , 1964, Inf. Control..

[49]  K. W. Church On memory limitations in natural language processing , 1982 .

[50]  Stephen Pulman,et al.  Grammars, parsers, and memory limitations , 1986 .

[51]  Lawrence E. Marks,et al.  Scaling of grammaticalness of self-embedded English sentences , 1968 .

[52]  John R. Anderson The Architecture of Cognition , 1983 .

[53]  David J. Chalmers,et al.  Syntactic Transformations on Distributed Representations , 1990 .

[54]  E. Wanner The ATN and the sausage machine: Which one is baloney? , 1980, Cognition.

[55]  D Burns,et al.  Sentence comprehension and memory for embedded structure , 1977, Memory & cognition.

[56]  C. Lee Giles,et al.  Extraction, Insertion and Refinement of Symbolic Rules in Dynamically Driven Recurrent Neural Networks , 1993 .

[57]  Aravind K. Joshi,et al.  A Processing Model for Free Word Order Languages , 1995, ArXiv.

[58]  Geoffrey K. Pullum,et al.  Natural languages and context-free languages , 1982 .

[59]  M. Braine,et al.  Short-term memory limitations on decoding self-embedded sentences , 1974 .

[60]  R. Duncan Luce,et al.  Individual Choice Behavior , 1959 .

[61]  Carson T. Schütze The empirical base of linguistics: Grammaticality judgments and linguistic methodology , 1998 .

[62]  Morten H. Christiansen,et al.  Generalization and connectionist language learning , 1994 .

[63]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[64]  E. Gibson Linguistic complexity: locality of syntactic dependencies , 1998, Cognition.

[65]  Geoffrey K. Pullum,et al.  Computationally Relevant Properties of Natural Languages and Their Grammars , 1985 .