Forget-me-some: General versus special purpose models in a hierarchical probabilistic task

Humans build models of their environments and act according to what they have learnt. In simple experimental environments, such model-based behaviour is often well accounted for as if subjects are ideal Bayesian observers. However, more complex probabilistic tasks require more sophisticated forms of inference that are sufficiently computationally and statistically taxing as to demand approximation. Here, we study properties of two approximation schemes in the context of a serial reaction time task in which stimuli were generated from a hierarchical Markov chain. One, pre-existing, scheme was a generically powerful variational method for hierarchical inference which has recently become popular as an account of psychological and neural data across a wide swathe of probabilistic tasks. A second, novel, scheme was more specifically tailored to the task at hand. We show that the latter model fit significantly better than the former. This suggests that our subjects were sensitive to many of the particular constraints of a complex behavioural task. Further, the tailored model provided a different perspective on the effects of cholinergic manipulations in the task. Neither model fit the behaviour on more complex contingencies that competently. These results illustrate the benefits and challenges that come with the general and special purpose modelling approaches and raise important questions of how they can advance our current understanding of learning mechanisms in the brain.

[1]  Michael I. Jordan,et al.  Advances in Neural Information Processing Systems 30 , 1995 .

[2]  C. Gallistel,et al.  The rat approximates an ideal detector of changes in rates of reward: implications for the law of effect. , 2001, Journal of experimental psychology. Animal behavior processes.

[3]  W. Kunde,et al.  Response-effect compatibility in manual choice reaction tasks. , 2001, Journal of experimental psychology. Human perception and performance.

[4]  Peter Dayan,et al.  Acetylcholine in cortical inference , 2002, Neural Networks.

[5]  David S. Touretzky,et al.  Long-Term Reward Prediction in TD Models of the Dopamine System , 2002, Neural Computation.

[6]  O. Hikosaka,et al.  Chunking during human visuomotor sequence learning , 2003, Experimental Brain Research.

[7]  Kae Nakamura,et al.  Emergence of rhythm during motor learning , 2004, Trends in Cognitive Sciences.

[8]  S. Ghirlanda Retrospective revaluation as simple associative learning. , 2005, Journal of experimental psychology. Animal behavior processes.

[9]  Angela J. Yu,et al.  Uncertainty, Neuromodulation, and Attention , 2005, Neuron.

[10]  Karl J. Friston,et al.  Encoding uncertainty in the hippocampus , 2006, Neural Networks.

[11]  R. Dolan,et al.  Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans , 2006, Nature.

[12]  Angela J. Yu,et al.  Phasic norepinephrine: A neural interrupt signal for unexpected events , 2006, Network.

[13]  P. Dayan,et al.  Tonic dopamine: opportunity costs and the control of response vigor , 2007, Psychopharmacology.

[14]  R. Shadmehr,et al.  Interacting Adaptive Processes with Different Timescales Underlie Short-Term Motor Learning , 2006, PLoS biology.

[15]  Timothy E. J. Behrens,et al.  Learning the value of information in an uncertain world , 2007, Nature Neuroscience.

[16]  E. Miller,et al.  A Neural Circuit Model of Flexible Sensorimotor Mapping: Learning and Forgetting on Multiple Timescales , 2007, Neuron.

[17]  Paul R. Schrater,et al.  Structure Learning in Human Sequential Decision-Making , 2008, NIPS.

[18]  Jonathan D. Cohen,et al.  Sequential effects: Superstition or rational behavior? , 2008, NIPS.

[19]  Karl J. Friston,et al.  Predictive coding under the free-energy principle , 2009, Philosophical Transactions of the Royal Society B: Biological Sciences.

[20]  J. Tenenbaum,et al.  Structured statistical models of inductive reasoning. , 2009, Psychological review.

[21]  Melody Dye,et al.  The Effects of Feature-Label-Order and Their Implications for Symbolic Learning , 2010, Cogn. Sci..

[22]  Y. Niv,et al.  Learning latent structure: carving nature at its joints , 2010, Current Opinion in Neurobiology.

[23]  Karl J. Friston,et al.  Observing the Observer (I): Meta-Bayesian Models of Learning and Decision-Making , 2010, PloS one.

[24]  Robert C. Wilson,et al.  An Approximately Bayesian Delta-Rule Model Explains the Dynamics of Belief Updating in a Changing Environment , 2010, The Journal of Neuroscience.

[25]  Maxime J Parent,et al.  Movement chunking during sequence learning is a dopamine-dependant process: a study conducted in Parkinson’s disease , 2010, Experimental Brain Research.

[26]  W. Penny,et al.  Time Scales of Representation in the Human Brain: Weighing Past Information to Predict Future Events , 2011, Front. Hum. Neurosci..

[27]  Peter Dayan,et al.  Vigor in the Face of Fluctuating Rates of Reward: An Experimental Examination , 2011, Journal of Cognitive Neuroscience.

[28]  H. Seo,et al.  A reservoir of time constants for memory traces in cortical neurons , 2011, Nature Neuroscience.

[29]  Timothy E. J. Behrens,et al.  Perceptual Classification in a Rapidly Changing Environment , 2011, Neuron.

[30]  Pradeep Shenoy,et al.  Rational Decision-Making in Inhibitory Control , 2011, Front. Hum. Neurosci..

[31]  N. Daw,et al.  Dissociating hippocampal and striatal contributions to sequential prediction learning , 2012, The European journal of neuroscience.

[32]  Scott T. Grafton,et al.  Differential Recruitment of the Sensorimotor Putamen and Frontoparietal Cortex during Motor Chunking in Humans , 2012, Neuron.

[33]  J. Rothwell,et al.  Action Reprogramming in Parkinson's Disease: Response to Prediction Error Is Modulated by Levels of Dopamine , 2012, The Journal of Neuroscience.

[34]  Stefan J. Kiebel,et al.  Evidence for neural encoding of Bayesian surprise in human somatosensation , 2012, NeuroImage.

[35]  C. Mathys,et al.  Hierarchical Prediction Errors in Midbrain and Basal Forebrain during Sensory Learning , 2013, Neuron.

[36]  Carolyn E. Jones,et al.  Gradual extinction prevents the return of fear: implications for the discovery of state , 2013, Front. Behav. Neurosci..

[37]  Joshua I. Gold,et al.  A Mixture of Delta-Rules Approximation to Bayesian Inference in Change-Point Problems , 2013, PLoS Comput. Biol..

[38]  Ulrik R Beierholm,et al.  Dopamine Modulates Reward-Related Vigor , 2013, Neuropsychopharmacology.

[39]  Matthew H. Wilder,et al.  Sequential effects in response time reveal learning mechanisms and event representations. , 2013, Psychological review.

[40]  David J. Freedman,et al.  A hierarchy of intrinsic timescales across primate cortex , 2014, Nature Neuroscience.

[41]  T. Robbins,et al.  Serotonin Depletion Induces ‘Waiting Impulsivity’ on the Human Four-Choice Serial Reaction Time Task: Cross-Species Translational Significance , 2014, Neuropsychopharmacology.

[42]  Karl J. Friston,et al.  Cholinergic Stimulation Enhances Bayesian Belief Updating in the Deployment of Spatial Attention , 2014, The Journal of Neuroscience.

[43]  He Huang,et al.  Sequential effects: A Bayesian analysis of prior bias on reaction time and behavioral choice , 2014, CogSci.

[44]  Karl J. Friston,et al.  Uncertainty in perception and the Hierarchical Gaussian Filter , 2014, Front. Hum. Neurosci..

[45]  Klaas E. Stephan,et al.  Inferring on the Intentions of Others by Hierarchical Bayesian Learning , 2014, PLoS Comput. Biol..

[46]  Sven Bestmann,et al.  The Role of Dopamine in Motor Flexibility , 2014, Journal of Cognitive Neuroscience.

[47]  Robert C. Wilson,et al.  Reinforcement Learning in Multidimensional Environments Relies on Attention Mechanisms , 2015, The Journal of Neuroscience.

[48]  P. Dayan,et al.  Pharmacological Fingerprints of Contextual Uncertainty , 2016, PLoS biology.

[49]  Florent Meyniel,et al.  Human Inferences about Sequences: A Minimal Transition Probability Model , 2016, bioRxiv.

[50]  S. Thompson-Schill,et al.  Varying Timescales of Stimulus Integration Unite Neural Adaptation and Prototype Formation , 2016, Current Biology.

[51]  S. Cavanagh,et al.  Autocorrelation structure at rest predicts value correlates of single neurons during reward-guided choice , 2016, eLife.

[52]  Joshua W. Brown,et al.  Neural Mechanisms of Credit Assignment in a Multicue Environment , 2016, The Journal of Neuroscience.

[53]  William T. Newsome,et al.  Learning fast and slow: deviations from the matching law can reflect an optimal strategy under uncertainty , 2017, bioRxiv.

[54]  C. Mathys,et al.  Representational Uncertainty in the Brain During Threat Conditioning and the Link With Psychopathic Traits. , 2017, Biological psychiatry. Cognitive neuroscience and neuroimaging.

[55]  G. Rees,et al.  Adults with autism over-estimate the volatility of the sensory environment , 2017, Nature Neuroscience.

[56]  Gergő Orbán,et al.  Measuring and Filtering Reactive Inhibition is Essential for Assessing Serial Decision Making and Learning , 2017, Journal of experimental psychology. General.

[57]  Albert R. Powers,et al.  Pavlovian conditioning–induced hallucinations result from overweighting of perceptual priors , 2017, Science.

[58]  Lilian A. E. Weber,et al.  Hierarchical prediction errors in midbrain and septum during social learning , 2017, Social cognitive and affective neuroscience.

[59]  Elyse H. Norton,et al.  Suboptimal Criterion Learning in Static and Dynamic Environments , 2017, PLoS Comput. Biol..

[60]  P. Sterzer,et al.  The Neural Correlates of Hierarchical Predictions for Perceptual Decisions , 2018, The Journal of Neuroscience.