I like coffee with cream and dog? Change in an implicit probabilistic representation captures meaning processing in the brain

The N400 component of the event-related brain potential has aroused much interest because it is thought to provide an online measure of meaning processing in the brain. Yet, the underlying process remains incompletely understood and actively debated. Here, we present a computationally explicit account of this process and the emerging representation of sentence meaning. We simulate N400 amplitudes as the change induced by an incoming stimulus in an implicit and probabilistic representation of meaning captured by the hidden unit activation pattern in a neural network model of sentence comprehension, and we propose that the process underlying the N400 also drives implicit learning in the network. The model provides a unified account of 16 distinct findings from the N400 literature and connects human language processing with successful deep learning approaches to language processing.

[1]  J. Haldane The interaction of nature and nurture. , 1946, Annals of eugenics.

[2]  Marcia K. Johnson,et al.  Contextual prerequisites for understanding: Some investigations of comprehension and recall , 1972 .

[3]  D. Swinney Lexical access during sentence comprehension: (Re)consideration of context effects , 1979 .

[4]  D. G. MacKay,et al.  Metaphor and Thought , 1980 .

[5]  M. Kutas,et al.  Reading senseless sentences: brain potentials reflect semantic incongruity. , 1980, Science.

[6]  Salim Roukos,et al.  Brain potentials related to stages of sentence verification. , 1983, Psychophysiology.

[7]  J. Fodor Modularity of mind , 1983 .

[8]  M. Kutas,et al.  Brain potentials during reading reflect word expectancy and semantic association , 1984, Nature.

[9]  M. Rugg The effects of semantic priming and work repetition on event-related potentials. , 1985, Psychophysiology.

[10]  Janet L. McDonald,et al.  The Development of Sentence Comprehension Strategies in English and Dutch. , 1986 .

[11]  J. Fodor,et al.  Connectionism and cognitive architecture: A critical analysis , 1988, Cognition.

[12]  James L. McClelland,et al.  A distributed, developmental model of word recognition and naming. , 1989, Psychological review.

[13]  Geoffrey E. Hinton Connectionist Learning Procedures , 1989, Artif. Intell..

[14]  James L. McClelland,et al.  Sentence comprehension: A parallel distributed processing approach , 1989, Language and Cognitive Processes.

[15]  James L. McClelland,et al.  Learning and Applying Contextual Constraints in Sentence Comprehension , 1990, Artif. Intell..

[16]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[17]  M. Kutas,et al.  Influences of semantic and syntactic context on open- and closed-class words , 1991, Memory & cognition.

[18]  Geoffrey E. Hinton Learning and Applying Contextual Constraints in Sentence Comprehension , 1991 .

[19]  M. Just,et al.  From the SelectedWorks of Marcel Adam Just 1992 A capacity theory of comprehension : Individual differences in working memory , 2017 .

[20]  Mark F. St. John,et al.  The Story Gestalt: A Model of Knowledge-Intensive Processes in Text Comprehension , 1992, Cogn. Sci..

[21]  M. Kutas,et al.  An Event-Related Potential (ERP) Analysis of Semantic Congruity and Repetition Effects in Sentences , 1992, Journal of Cognitive Neuroscience.

[22]  P. Holcomb,et al.  Event-related brain potentials elicited by syntactic anomaly , 1992 .

[23]  Peter M. Todd,et al.  Learning and connectionist representations , 1993 .

[24]  Peter Hagoort,et al.  The Processing Nature of the N400: Evidence from Masked Priming , 1993, Journal of Cognitive Neuroscience.

[25]  G. McCarthy,et al.  Language-related field potentials in the anterior-medial temporal lobe: I. Intracranial distribution and neural generators , 1995, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[26]  G. McCarthy,et al.  Language-related field potentials in the anterior-medial temporal lobe: II. Effects of word type and semantic priming , 1995, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[27]  E. Bizzi,et al.  The Cognitive Neurosciences , 1996 .

[28]  E. Vogel,et al.  Word meanings can be accessed but not reported during the attentional blink , 1996, Nature.

[29]  Peter Dayan,et al.  A Neural Substrate of Prediction and Reward , 1997, Science.

[30]  M. Farah,et al.  Role of left inferior prefrontal cortex in retrieval of semantic knowledge: a reevaluation. , 1997, Proceedings of the National Academy of Sciences of the United States of America.

[31]  M. Kutas,et al.  The N400 in a semantic categorization task across 6 decades. , 1998, Electroencephalography and clinical neurophysiology.

[32]  M. Kutas,et al.  Expect the Unexpected: Event-related Brain Response to Morphosyntactic Violations , 1998 .

[33]  F. Pulvermüller Words in the brain's language , 1999, Behavioral and Brain Sciences.

[34]  G. Altmann,et al.  Incremental interpretation at verbs: restricting the domain of subsequent reference , 1999, Cognition.

[35]  Colin M. Brown,et al.  Semantic Integration in Sentences and Discourse: Evidence from the N400 , 1999, Journal of Cognitive Neuroscience.

[36]  C Van Petten,et al.  Time course of word identification and semantic integration in spoken language. , 1999, Journal of experimental psychology. Learning, memory, and cognition.

[37]  Kara D. Federmeier,et al.  A Rose by Any Other Name: Long-Term Memory Structure and Sentence Processing , 1999 .

[38]  A. Borst Seeing smells: imaging olfactory learning in bees , 1999, Nature Neuroscience.

[39]  Colin M. Brown,et al.  ERP effects of listening to speech compared to reading: the P600/SPS to syntactic violations in spoken sentences and rapid serial visual presentation , 2000, Neuropsychologia.

[40]  Bobby D. Bryant and Risto Miikkulainen From Word Stream To Gestalt: A Direct Semantic Parse For Complex Sentences , 2001 .

[41]  M. Koivisto,et al.  Cognitive representations underlying the N400 priming effect. , 2001, Brain research. Cognitive brain research.

[42]  David C. Plaut,et al.  A connectionist model of sentence comprehension and production , 2002 .

[43]  Karl G. D. Bailey,et al.  Good-Enough Representations in Language Comprehension , 2002 .

[44]  A. Sanford,et al.  Depth of processing in language comprehension: not noticing the evidence , 2002, Trends in Cognitive Sciences.

[45]  A. Friederici Towards a neural basis of auditory sentence processing , 2002, Trends in Cognitive Sciences.

[46]  S. Bookheimer Functional MRI of language: new approaches to understanding the cortical organization of semantic processing. , 2002, Annual review of neuroscience.

[47]  H. Heinze,et al.  Perceptual Priming Versus Explicit Memory: Dissociable Neural Correlates at Encoding , 2002, Journal of Cognitive Neuroscience.

[48]  James L. McClelland,et al.  The parallel distributed processing approach to semantic cognition , 2003, Nature Reviews Neuroscience.

[49]  G. Altmann,et al.  The time-course of prediction in incremental sentence processing: Evidence from anticipatory eye-movements , 2003 .

[50]  D. Caplan,et al.  Electrophysiological distinctions in processing conceptual relationships within simple sentences. , 2003, Brain research. Cognitive brain research.

[51]  Bruce D. McCandliss,et al.  The visual word form area: expertise for reading in the fusiform gyrus , 2003, Trends in Cognitive Sciences.

[52]  L. Osterhout,et al.  Neural correlates of second-language word learning: minimal instruction produces rapid change , 2004, Nature Neuroscience.

[53]  H. Barber,et al.  Syllable-frequency effects in visual word recognition: evidence from ERPs , 2004, Neuroreport.

[54]  John C J Hoeks,et al.  Seeing words in context: the interaction of lexical and sentence level information during reading. , 2004, Brain research. Cognitive brain research.

[55]  A. Friederici,et al.  N400-like Semantic Incongruity Effect in 19-Month-Olds: Processing Known Words in Picture Contexts , 2004, Journal of Cognitive Neuroscience.

[56]  N. Dronkers,et al.  Lesion analysis of the brain areas involved in language comprehension , 2004, Cognition.

[57]  H. Kolk,et al.  An ERP study of P600 effects elicited by semantic anomalies. , 2005, Brain research. Cognitive brain research.

[58]  Karl J. Friston,et al.  A theory of cortical responses , 2005, Philosophical Transactions of the Royal Society B: Biological Sciences.

[59]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[60]  L. Osterhout,et al.  The independence of combinatory semantic processing: Evidence from event-related potentials , 2005 .

[61]  Colin M. Brown,et al.  The cascaded nature of lexical selection and integration in auditory sentence processing. , 2006, Journal of experimental psychology. Learning, memory, and cognition.

[62]  Mante S. Nieuwland,et al.  When Peanuts Fall in Love: N400 Evidence for the Power of Discourse , 2005, Journal of Cognitive Neuroscience.

[63]  A. Jongman,et al.  A comparison of semantic and syntactic event related potentials generated by children and adults , 2006, Brain and Language.

[64]  Mark Steyvers,et al.  Topics in semantic representation. , 2007, Psychological review.

[65]  J. Debruille,et al.  The N400 potential could index a semantic inhibition , 2007, Brain Research Reviews.

[66]  Kara D. Federmeier,et al.  Multiple effects of sentential constraint on word processing , 2007, Brain Research.

[67]  L. Gleitman,et al.  On the give and take between event apprehension and utterance formulation. , 2007, Journal of memory and language.

[68]  J. Polich Updating P300: An integrative theory of P3a and P3b , 2007, Clinical Neurophysiology.

[69]  R. Levy Expectation-based syntactic comprehension , 2008, Cognition.

[70]  Ellen F. Lau,et al.  A cortical network for semantics: (de)constructing the N400 , 2008, Nature Reviews Neuroscience.

[71]  Mante S. Nieuwland,et al.  When the Truth Is Not Too Hard to Handle , 2008, Psychological science.

[72]  R. Jacobs,et al.  Perception of speech reflects optimal use of probabilistic speech cues , 2008, Cognition.

[73]  Kara D. Federmeier,et al.  Chapter 1 Time for Meaning: Electrophysiology Provides Insights into the Dynamics of Representation and Processing in Semantic Memory , 2009 .

[74]  Gabriella Vigliocco,et al.  Integrating experiential and distributional data to learn semantic representations. , 2009, Psychological review.

[75]  K. Rayner,et al.  Eye movement evidence that readers maintain and act on uncertainty about past linguistic input , 2009, Proceedings of the National Academy of Sciences.

[76]  Pierre Baldi,et al.  Bayesian surprise attracts human attention , 2005, Vision Research.

[77]  A. Turken,et al.  The Neural Architecture of the Language Comprehension Network: Converging Evidence from Lesion and Connectivity Analyses , 2011, Front. Syst. Neurosci..

[78]  Kara D. Federmeier,et al.  Thirty years and counting: finding meaning in the N400 component of the event-related brain potential (ERP). , 2011, Annual review of psychology.

[79]  Lukás Burget,et al.  Strategies for training large scale neural network language models , 2011, 2011 IEEE Workshop on Automatic Speech Recognition & Understanding.

[80]  Giosuè Baggio,et al.  The balance between memory and unification in semantics: A dynamic account of the N400 , 2011 .

[81]  C. Van Petten,et al.  Prediction during language comprehension: benefits, costs, and ERP components. , 2012, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[82]  C. Van Petten,et al.  Lexical versus conceptual anticipation during sentence processing: frontal positivity and N400 ERP components. , 2012, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[83]  A. Wagner,et al.  Distributed representations in memory: insights from functional brain imaging. , 2012, Annual review of psychology.

[84]  D. Plaut,et al.  A neurally plausible Parallel Distributed Processing model of Event-Related Potential word reading data , 2012, Brain and Language.

[85]  John C. J. Hoeks,et al.  A time and place for language comprehension: mapping the N400 and the P600 to a minimal cortical network , 2013, Front. Hum. Neurosci..

[86]  W. Sommer,et al.  Differential Task Effects on N400 and P600 Elicited by Semantic and Syntactic Violations , 2014, PloS one.

[87]  Milena Rabovsky,et al.  Simulating the N400 ERP component as semantic network error: Insights from a feature-based connectionist attractor model of word meaning , 2014, Cognition.

[88]  F. De Filippis,et al.  A Selected Core Microbiome Drives the Early Stages of Three Popular Italian Cheese Manufactures , 2014, PloS one.

[89]  Katherine A. DeLong,et al.  Predictability, plausibility, and two late ERP positivities during written sentence comprehension , 2014, Neuropsychologia.

[90]  M. Schlesewsky,et al.  The P600-as-P3 hypothesis revisited: Single-trial analyses reveal that the late EEG positivity following linguistically deviant material is reaction time aligned , 2014, Brain and Language.

[91]  Blair C. Armstrong,et al.  PSPs and ERPs: Applying the dynamics of post-synaptic potentials to individual units in simulation of temporally extended Event-Related Potential reading data , 2014, Brain and Language.

[92]  Jeffrey Pennington,et al.  GloVe: Global Vectors for Word Representation , 2014, EMNLP.

[93]  S. Frank,et al.  The ERP response to the amount of information conveyed by words in sentences , 2015, Brain and Language.

[94]  Phil Blunsom,et al.  Teaching Machines to Read and Comprehend , 2015, NIPS.

[95]  Brian MacWhinney,et al.  The handbook of language emergence , 2015 .

[96]  Gina R Kuperberg,et al.  Separate streams or probabilistic inference? What the N400 can tell us about the comprehension of events , 2016, Language, cognition and neuroscience.

[97]  Yvonne Feierabend Foundations Of Language Brain Meaning Grammar Evolution , 2016 .

[98]  S. Gotts Incremental learning of perceptual and conceptual representations and the puzzle of neural repetition suppression , 2016, Psychonomic bulletin & review.

[99]  George Kurian,et al.  Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation , 2016, ArXiv.

[100]  Samuel J Cheyette,et al.  Modeling the N400 ERP component as transient semantic over-activation within a neural network model of word comprehension , 2017, Cognition.