Dynamic encoding of structural uncertainty in gradient symbols

An important achievement in modeling online language comprehension is the discovery of the relationship between processing difficulty and surprisal (Hale, 2001; Levy, 2008). However, it is not clear how structural uncertainty can be represented and updated in a continuoustime continuous-state dynamical system model, a reasonable abstraction of neural computation. In this study, we investigate the Gradient Symbolic Computation (GSC) model (Smolensky et al., 2014) and show how it can dynamically encode and update structural uncertainty via the gradient activation of symbolic constituents. We claim that surprisal is closely related to the amount of change in the optimal activation state driven by a new word input. In a simulation study, we demonstrate that the GSC model implementing a simple probabilistic symbolic grammar can simulate the effect of surprisal on processing time. Our model provides a mechanistic account of the effect of surprisal, bridging between probabilistic symbolic models and subsymbolic connectionist models.

[1]  William Ramsey.  THE HARMONIC MIND: FROM NEURAL COMPUTATION TO OPTIMALITY‐THEORETIC GRAMMAR—VOLUME 1: COGNITIVE ARCHITECTURE AND VOLUME 2: LINGUISTIC AND PHILOSOPHICAL IMPLICATIONS , 2009 .

[2]  M. Tanenhaus.  Afterword The impact of “The cognitive basis for linguistic structures” , 2013 .

[3]  Gerard Kempen,et al.  The Unification Space implemented as a localist neural net: predictions and error-tolerance in a constraint-based parser , 2009, Cognitive Neurodynamics.

[4]  Geoffrey E. Hinton.  Tensor Product Variable Binding and the Representation of Symbolic Structures in Connectionist Systems , 1991 .

[5]  Reinhold Kliegl,et al.  Parsing costs as predictors of reading difficulty: An evaluation using the Potsdam Sentence Corpus , 2008, Journal of Eye Movement Research.

[6]  Daniel Jurafsky,et al.  A Probabilistic Model of Lexical and Syntactic Access and Disambiguation , 1996, Cogn. Sci..

[7]  P. Smolensky,et al.  Neural and conceptual interpretation of PDP models , 1986 .

[8]  Daniel C. Richardson,et al.  Effects of merely local syntactic coherence on sentence processing , 2004 .

[9]  R. Levy.  Expectation-based syntactic comprehension , 2008, Cognition.

[10]  John Hale,et al.  A Probabilistic Earley Parser as a Psycholinguistic Model , 2001, NAACL.

[11]  Matthew Goldrick,et al.  Optimization and Quantization in Gradient Symbol Systems: A Framework for Integrating the Continuous and the Discrete in Cognition , 2014, Cogn. Sci..

[12]  Pyeong Whan Cho,et al.  Bifurcation analysis of a Gradient Symbolic Computation model of incremental processing , 2016, CogSci.

[13]  Timothy E. J. Behrens,et al.  Dissociable effects of surprise and model update in parietal and anterior cingulate cortex , 2013, Proceedings of the National Academy of Sciences.

[14]  Nathaniel J. Smith,et al.  The effect of word predictability on reading time is logarithmic , 2013, Cognition.

[15]  Pyeong Whan Cho,et al.  Incremental parsing in a continuous dynamical system: sentence processing in Gradient Symbolic Computation , 2017 .

[16]  W. Tabor,et al.  Evidence for self-organized sentence processing: digging-in effects. , 2004, Journal of experimental psychology. Learning, memory, and cognition.

[17]  William Schuler,et al.  Left-Corner Parsing With Distributed Associative Memory Produces Surprisal and Locality Effects. , 2018, Cognitive science.

[18]  Lyn Frazier,et al.  Sentence processing: A tutorial review. , 1987 .

[19]  John Hale,et al.  Uncertainty About the Rest of the Sentence , 2006, Cogn. Sci..

[20]  Frank Keller,et al.  Data from eye-tracking corpora as evidence for theories of syntactic processing complexity , 2008, Cognition.

[21]  Christina M. Krause.  The Psychological Reality of Local Coherences in Sentence Processing , 2005 .