Broad-Coverage Parsing with Neural Networks

Subsymbolic systems have been successfully used to model several aspects of human language processing. Such parsers are appealing because they allow revising the interpretation as words are incrementally processed. Yet, it has been very hard to scale them up to realistic language due to training time, limited memory, and the difficulty of representing linguistic structure. In this study, we show that it is possible to keep track of long-distance dependencies and to parse into deeper structures than before based on two techniques: a localist encoding of the input sequence and a dynamic unrolling of the network according to the parse tree. With these techniques, the system can nonmonotonically parse a corpus of realistic sentences into parse trees labelled with grammatical tags from a broad-coverage Head-driven Phrase Structure Grammar of English.

[1]  Christoph Goller,et al.  Learning task-dependent distributed representations by backpropagation through structure , 1996, Proceedings of International Conference on Neural Networks (ICNN'96).

[2]  R. Miikkulainen.  Dyslexic and Category-Specific Aphasic Impairments in a Self-Organizing Feature Map Model of the Lexicon , 1997, Brain and Language.

[3]  David J. Chalmers,et al.  Syntactic Transformations on Distributed Representations , 1990 .

[4]  T. Shallice,et al.  Perseverative and Semantic Influences on Visual Object Naming Errors in Optic Aphasia: A Connectionist Account , 1993, Journal of Cognitive Neuroscience.

[5]  Geoffrey E. Hinton.  Learning and Applying Contextual Constraints in Sentence Comprehension , 1991 .

[6]  James Hammerton.  Clause identification with long short-term memory , 2001, CoNLL.

[7]  Stephan Oepen,et al.  The (new) LKB system , 1999 .

[8]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[9]  Risto Miikkulainen,et al.  Combining Maps and Distributed Representations for Shift-Reduce Parsing , 1998, Hybrid Neural Systems.

[10]  Ann Q. Gates,et al.  TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING , 2005 .

[11]  Lai-Wan Chan,et al.  Analyzing Holistic Parsers: Implications for Robust Parsing and Systematicity , 2001, Neural Computation.

[12]  David S. Touretzky.  Connectionism and Compositional Semantics , 1989 .

[13]  James Henderson,et al.  Incremental Syntactic Parsing of Natural Language Corpora with Simple Synchrony Networks , 2001, IEEE Trans. Knowl. Data Eng..

[14]  Paul W. Munro,et al.  A Network for Encoding, Decoding and Translating Locative Prepositions , 1991 .

[15]  Ronald J. Williams,et al.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks , 1989, Neural Computation.

[16]  Risto Miikkulainen,et al.  Subsymbolic natural language processing - an integrated model of scripts, lexicon, and memory , 1993, Neural network modeling and connectionism.

[17]  Ivan A. Sag,et al.  Book Reviews: Head-driven Phrase Structure Grammar and German in Head-driven Phrase-structure Grammar , 1996, CL.

[18]  Nick Chater,et al.  Toward a connectionist model of recursion in human linguistic performance , 1999, Cogn. Sci..

[19]  Mark S. Seidenberg,et al.  The emergence of grammaticality in connectionist networks. , 1999 .

[20]  Sandiway Fong,et al.  Natural Language Grammatical Inference with Recurrent Neural Networks , 2000, IEEE Trans. Knowl. Data Eng..

[21]  George Berg,et al.  A Connectionist Parser with Recursive Sentence Structure and Lexical Disambiguation , 1992, AAAI.

[22]  Jeffrey L. Elman,et al.  A Connectionist Simulation of the Empirical Acquisition of Grammatical Relations , 1998, Hybrid Neural Systems.

[23]  Yoshua Bengio,et al.  Learning long-term dependencies with gradient descent is difficult , 1994, IEEE Trans. Neural Networks.

[24]  Jordan B. Pollack,et al.  Recursive Distributed Representations , 1990, Artif. Intell..