Cerebellar learning using perturbations

The cerebellum aids the learning and execution of fast co-ordinated movements, with acquired information being stored by plasticity of parallel fibre–Purkinje cell synapses. According to the current consensus, erroneously active parallel fibre synapses are depressed by complex spikes arising as climbing fibres signal movement errors. However, this theory cannot solve the credit assignment problem of using the limited information from a global movement evaluation to optimise behaviour by guiding the plasticity in numerous neurones. We identify the possible implementation of an algorithm solving this problem, whereby spontaneous complex spikes perturb ongoing movements, create an eligibility trace for plasticity and signal resulting error changes to guide plasticity. These error changes are extracted by adaptively cancelling the average error. This framework, stochastic gradient descent with estimated global errors, generates specific predictions for synaptic plasticity rules that contradict the current consensus. However, in vitro plasticity experiments under physiological conditions verified our predictions, highlighting the sensitivity of plasticity studies to unphysiological conditions. Using numerical and analytical approaches we demonstrate the convergence and estimate the capacity of learning in our implementation. Finally, a similar mechanism may operate during optimisation of action sequences by the basal ganglia, where dopamine could both initiate movements and signal rewards, analogously to the dual perturbation and correction role of the climbing fibre outlined here.

[1]  C. Harris On the optimal control of behaviour: a stochastic perspective , 1998, Journal of Neuroscience Methods.

[2]  Germund Hesslow,et al.  Cerebellum and conditioned reflexes , 1998, Trends in Cognitive Sciences.

[3]  I A Silver,et al.  Intracellular and extracellular changes of [Ca2+] in hypoxia and ischemia in rat brain in vivo , 1990, The Journal of general physiology.

[4]  T. Sejnowski,et al.  Storing covariance with nonlinearly interacting neurons , 1977, Journal of mathematical biology.

[5]  H. Wigström,et al.  Large long-lasting potentiation in the dentate gyrus in vitro during blockade of inhibition , 1983, Brain Research.

[6]  W. Regehr,et al.  Inhibitory Regulation of Electrically Coupled Neurons in the Inferior Olive Is Mediated by Asynchronous Release of GABA , 2009, Neuron.

[7]  Tahl Holtzman,et al.  Electrophysiological Localization of Eyeblink-Related Microzones in Rabbit Cerebellar Cortex , 2010, The Journal of Neuroscience.

[8]  H. Wigström,et al.  Facilitated induction of hippocampal long-lasting potentiation during blockade of inhibition , 1983, Nature.

[9]  Henrik Jörntell,et al.  Receptive Field Remodeling Induced by Skin Stimulation in Cerebellar Neurons in vivo , 2011, Front. Neural Circuits.

[10]  Eduardo F. Morales,et al.  An Introduction to Reinforcement Learning , 2011 .

[11]  C. Hansel,et al.  Bidirectional Parallel Fiber Plasticity in the Cerebellum under Climbing Fiber Control , 2004, Neuron.

[12]  Peter Dayan,et al.  A Neural Substrate of Prediction and Reward , 1997, Science.

[13]  D. Kleinfeld,et al.  Reversing cerebellar long-term depression , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[14]  Germund Hesslow,et al.  Changes in complex spike activity during classical conditioning , 2014, Front. Neural Circuits.

[15]  I. Murakami Correlations between fixation stability and visual motion sensitivity , 2004, Vision Research.

[16]  M. Mauk,et al.  Learning-Induced Plasticity in Deep Cerebellar Nucleus , 2006, The Journal of Neuroscience.

[17]  Masao Ito,et al.  Long-lasting depression of parallel fiber-Purkinje cell transmission induced by conjunctive stimulation of parallel fibers and climbing fibers in the cerebellar cortex , 1982, Neuroscience Letters.

[18]  Richard Apps,et al.  Cerebellar cortical organization: a one-map hypothesis , 2009, Nature Reviews Neuroscience.

[19]  Chris I. De Zeeuw,et al.  Motor Learning Requires Purkinje Cell Synaptic Potentiation through Activation of AMPA-Receptor Subunit GluA3 , 2017, Neuron.

[20]  J. Simpson,et al.  Spatial organization of visual messages of the rabbit's cerebellar flocculus. II. Complex and simple spike responses of Purkinje cells. , 1988, Journal of neurophysiology.

[21]  S. G. Lisberger,et al.  Detection of tracking errors by visual climbing fiber inputs to monkey cerebellar flocculus during pursuit eye movements , 1986, Neuroscience Letters.

[22]  Richard S. Sutton,et al.  Neuronlike adaptive elements that can solve difficult learning control problems , 1983, IEEE Transactions on Systems, Man, and Cybernetics.

[23]  Riccardo Zucca,et al.  Number of Spikes in Climbing Fibers Determines the Direction of Cerebellar Learning , 2013, The Journal of Neuroscience.

[24]  R. Keep,et al.  Brain fluid calcium concentration and response to acute hypercalcaemia during development in the rat. , 1988, The Journal of physiology.

[25]  Ronald F Rogers,et al.  The cerebellum is necessary for rabbit classical eyeblink conditioning with a non-somatosensory (photic) unconditioned stimulus , 1999, Behavioural Brain Research.

[26]  M. Yartsev,et al.  Pausing Purkinje Cells in the Cerebellum of the Awake Cat , 2008, Front. Syst. Neurosci..

[27]  P. Thier,et al.  Encoding of movement time by populations of cerebellar Purkinje cells , 2000, Nature.

[28]  I. Raman,et al.  Integration of Purkinje Cell Inhibition by Cerebellar Nucleo-Olivary Neurons , 2015, The Journal of Neuroscience.

[29]  John H Freeman,et al.  Synapse formation is associated with memory storage in the cerebellum , 2002, Proceedings of the National Academy of Sciences of the United States of America.

[30]  J. Simpson,et al.  Microcircuitry and function of the inferior olive , 1998, Trends in Neurosciences.

[31]  Germund Hesslow,et al.  Bidirectional plasticity of Purkinje cells matches temporal features of learning. , 2014, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[32]  M. Häusser,et al.  Initiation and spread of sodium action potentials in cerebellar purkinje cells , 1994, Neuron.

[33]  Michael Häusser,et al.  Linking Synaptic Plasticity and Spike Output at Excitatory and Inhibitory Synapses onto Cerebellar Purkinje Cells , 2007, The Journal of Neuroscience.

[34]  Masao Ito,et al.  Climbing fibre induced depression of both mossy fibre responsiveness and glutamate sensitivity of cerebellar Purkinje cells , 1982, The Journal of physiology.

[35]  Xiaohui Xie,et al.  Learning in neural networks by reinforcement of irregular spiking. , 2004, Physical review. E, Statistical, nonlinear, and soft matter physics.

[36]  Yutaka Hirata,et al.  The vestibulo-ocular reflex as a model system for motor learning: what is the role of the cerebellum? , 2008, The Cerebellum.

[37]  F. Crépel,et al.  Pairing of pre‐ and postsynaptic activities in cerebellar Purkinje cells induces long‐term changes in synaptic efficacy in vitro. , 1991, The Journal of physiology.

[38]  N. Barmack,et al.  Cerebellar Climbing Fibers Modulate Simple Spikes in Purkinje Cells , 2003, The Journal of Neuroscience.

[39]  J. Raymond,et al.  Elimination of climbing fiber instructive signals during motor learning , 2009, Nature Neuroscience.

[40]  P. Ascher,et al.  High-Affinity Zinc Inhibition of NMDA NR1–NR2A Receptors , 1997, The Journal of Neuroscience.

[41]  Richard Mooney,et al.  Neurobiology of song learning , 2009, Current Opinion in Neurobiology.

[42]  J. Simpson,et al.  Discharges in Purkinje cell axons during climbing fiber activation. , 1971, Brain research.

[43]  Martin Garwicz,et al.  Common principles of sensory encoding in spinal reflex modules and cerebellar climbing fibres , 2002, The Journal of physiology.

[44]  Henrik Jörntell,et al.  Properties of Somatosensory Synaptic Integration in Cerebellar Granule Cells In Vivo , 2006, The Journal of Neuroscience.

[45]  Martijn Schonewille,et al.  Zonal organization of the mouse flocculus: Physiology, input, and output , 2006, The Journal of comparative neurology.

[46]  Paolo Bazzigaluppi,et al.  Properties of the Nucleo-Olivary Pathway: An In Vivo Whole-Cell Patch Clamp Study , 2012, PloS one.

[47]  K. Caddy,et al.  The number of Purkinje cells and olive neurones in the normal and Lurcher mutant mouse , 1976, Brain Research.

[48]  W. T. Thach Discharge of Purkinje and cerebellar nuclear neurons during rapidly alternating arm movements in the monkey. , 1968, Journal of neurophysiology.

[49]  T. Sejnowski,et al.  A Computational Model of Birdsong Learning by Auditory Experience and Auditory Feedback , 1998 .

[50]  Wei Zhang,et al.  Long-Term Depression at the Mossy Fiber–Deep Cerebellar Nucleus Synapse , 2006, The Journal of Neuroscience.

[51]  M. Ito,et al.  Neural design of the cerebellar motor control system. , 1972, Brain research.

[52]  Nicolas Brunel,et al.  Optimal Properties of Analog Perceptrons with Excitatory Weights , 2013, PLoS Comput. Biol..

[53]  H. Jörntell,et al.  In Vivo Analysis of Inhibitory Synaptic Inputs and Rebounds in Deep Cerebellar Nuclear Neurons , 2011, PloS one.

[54]  Nicolas Brunel,et al.  STDP in a Bistable Synapse Model Based on CaMKII and Associated Signaling Pathways , 2007, PLoS Comput. Biol..

[55]  James A. Mortimer,et al.  Cerebellar responses to teleceptive stimuli in alert monkeys , 1975, Brain Research.

[56]  J. Nadal,et al.  Optimal Information Storage and the Distribution of Synaptic Weights Perceptron versus Purkinje Cell , 2004, Neuron.

[57]  M. Glickstein,et al.  Discrete lesions of the cerebellar cortex abolish the classically conditioned nictitating membrane response of the rabbit , 1984, Behavioural Brain Research.

[58]  D. Robinson Adaptive gain control of vestibuloocular reflex by the cerebellum. , 1976, Journal of neurophysiology.

[59]  M. Konishi The role of auditory feedback in the control of vocalization in the white-crowned sparrow. , 1965, Zeitschrift fur Tierpsychologie.

[60]  Tatsuya Kimura,et al.  Cerebellar complex spikes encode both destinations and errors in arm movements , 1998, Nature.

[61]  Henrik Jörntell,et al.  Cutaneous receptive fields and topography of mossy fibres and climbing fibres projecting to cat cerebellar C3 zone , 1998, The Journal of physiology.

[62]  R F Schild,et al.  On the inferior olive of the albino rat , 1970, The Journal of comparative neurology.

[63]  R. Llinás,et al.  Electrophysiology of guinea‐pig cerebellar nuclear cells in the in vitro brain stem‐cerebellar preparation. , 1988, The Journal of physiology.

[64]  H. Sompolinsky,et al.  Bistability of cerebellar Purkinje cells modulated by sensory stimulation , 2005, Nature Neuroscience.

[65]  Y. Yarom,et al.  Electrotonic coupling in the inferior olivary nucleus revealed by simultaneous double patch recordings. , 2002, Journal of neurophysiology.

[66]  R. Llinás,et al.  Oscillatory properties of guinea‐pig inferior olivary neurones and their pharmacological modulation: an in vitro study. , 1986, The Journal of physiology.

[67]  S. Wang,et al.  Coincidence detection in single dendritic spines mediated by calcium release , 2000, Nature Neuroscience.

[68]  Wade G. Regehr,et al.  Timing dependence of the induction of cerebellar LTD , 2008, Neuropharmacology.

[69]  Yoshiko Kojima,et al.  Encoding of action by the Purkinje cells of the cerebellum , 2015, Nature.

[70]  伊藤 正男 The cerebellum and neural control , 1984 .

[71]  I. Raman,et al.  Mechanisms of Potentiation of Mossy Fiber EPSCs in the Cerebellar Nuclei by Coincident Synaptic Excitation and Inhibition , 2008, The Journal of Neuroscience.

[72]  Roger Y Tsien,et al.  A new form of cerebellar long-term potentiation is postsynaptic and depends on nitric oxide but not cAMP , 2002, Proceedings of the National Academy of Sciences of the United States of America.

[73]  Ronald J. Williams,et al.  Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning , 2004, Machine Learning.

[74]  Henrik Jörntell,et al.  Reciprocal Bidirectional Plasticity of Parallel Fiber Receptive Fields in Cerebellar Purkinje Cells and Their Afferent Interneurons , 2002, Neuron.

[75]  Peter Thier,et al.  Cerebellum-dependent motor learning: lessons from adaptation of eye movements in primates. , 2014, Progress in brain research.

[76]  J. Albus A Theory of Cerebellar Function , 1971 .

[77]  M. Häusser,et al.  Encoding of Oscillations by Axonal Bursts in Inferior Olive Neurons , 2009, Neuron.

[78]  D. Tank,et al.  Widespread State-Dependent Shifts in Cerebellar Activity in Locomoting Mice , 2012, PloS one.

[79]  H. Seung,et al.  Model of birdsong learning based on gradient estimation by dynamic perturbation of neural conductances. , 2007, Journal of neurophysiology.

[80]  Boris Barbour,et al.  Presynaptic NR2A-containing NMDA receptors implement a high-pass filter synaptic plasticity rule , 2009, Proceedings of the National Academy of Sciences.

[81]  S. Wang,et al.  Order-Dependent Coincidence Detection in Cerebellar Purkinje Neurons at the Inositol Trisphosphate Receptor , 2008, The Journal of Neuroscience.

[82]  Aaron S. Andalman,et al.  Vocal Experimentation in the Juvenile Songbird Requires a Basal Ganglia Circuit , 2005, PLoS biology.

[83]  Laure Rondi-Reig,et al.  T-type channel blockade impairs long-term potentiation at the parallel fiber–Purkinje cell synapse and cerebellar learning , 2013, Proceedings of the National Academy of Sciences.

[84]  I. Raman,et al.  Potentiation of Mossy Fiber EPSCs in the Cerebellar Nuclei by NMDA Receptor Activation followed by Postinhibitory Rebound Current , 2006, Neuron.

[85]  L. Optican,et al.  Cerebellar-dependent adaptive control of primate saccadic system. , 1980, Journal of neurophysiology.

[86]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[87]  Chris I. De Zeeuw,et al.  Axonal Sprouting and Formation of Terminals in the Adult Cerebellum during Associative Motor Learning , 2013, The Journal of Neuroscience.

[88]  Marvin Minsky,et al.  Steps toward Artificial Intelligence , 1995, Proceedings of the IRE.

[89]  Jennifer L. Raymond,et al.  Timing Rules for Synaptic Plasticity Matched to Behavioral Function , 2018, Neuron.

[90]  Professor Dr. John C. Eccles,et al.  The Cerebellum as a Neuronal Machine , 1967, Springer Berlin Heidelberg.

[91]  A. Konnerth,et al.  Synaptic‐ and agonist‐induced excitatory currents of Purkinje cells in rat cerebellar slices. , 1991, The Journal of physiology.

[92]  Laurentiu S. Popa,et al.  What Features of Limb Movements are Encoded in the Discharge of Cerebellar Neurons? , 2011, The Cerebellum.

[93]  H. Sompolinsky,et al.  Purkinje cells in awake behaving animals operate at the upstate membrane potential , 2006, Nature Neuroscience.

[94]  Katrina Y. Choe,et al.  Circuit Mechanisms Underlying Motor Memory Formation in the Cerebellum , 2015, Neuron.

[95]  S. Khosrovani,et al.  Olivary subthreshold oscillations and burst activity revisited , 2012, Front. Neural Circuits.

[96]  Riccardo Zucca,et al.  Climbing Fiber Regulation of Spontaneous Purkinje Cell Activity and Cerebellum-Dependent Blink Responses1,2,3 , 2016, eNeuro.

[97]  M. Barrot,et al.  Clusters of cerebellar Purkinje cells control their afferent climbing fiber discharge , 2013, Proceedings of the National Academy of Sciences of the United States of America.

[98]  Henrik Jörntell,et al.  Receptive Field Plasticity Profoundly Alters the Cutaneous Parallel Fiber Synaptic Input to Cerebellar Interneurons In Vivo , 2003, The Journal of Neuroscience.

[99]  Nicolas Brunel,et al.  Burst-Dependent Bidirectional Plasticity in the Cerebellum Is Driven by Presynaptic NMDA Receptors. , 2016, Cell reports.

[100]  M. Häusser,et al.  Determinants of Action Potential Propagation in Cerebellar Purkinje Cell Axons , 2005, The Journal of Neuroscience.

[101]  John Porrill,et al.  Provided for Non-commercial Research and Educational Use Only. Not for Reproduction, Distribution or Commercial Use. Decorrelation Learning in the Cerebellum: Computational Analysis and Experimental Questions 7 Author's Personal Copy , 2022 .

[102]  B. Sakmann,et al.  Spine Ca2+ Signaling in Spike-Timing-Dependent Plasticity , 2006, The Journal of Neuroscience.

[103]  Naoshige Uchida,et al.  Erratum: Arithmetic and local circuitry underlying dopamine prediction errors , 2015, Nature.

[104]  Ayako M Watabe,et al.  The mechanisms of the strong inhibitory modulation of long‐term potentiation in the rat dentate gyrus , 2011, The European journal of neuroscience.

[105]  M. Häusser,et al.  Integration of quanta in cerebellar granule cells during sensory processing , 2004, Nature.

[106]  H. Seung,et al.  Learning in Spiking Neural Networks by Reinforcement of Stochastic Synaptic Transmission , 2003, Neuron.

[107]  S. Grillner,et al.  Mechanisms for selection of basic motor programs – roles for the striatum and pallidum , 2005, Trends in Neurosciences.

[108]  D. Marr A theory of cerebellar cortex , 1969, The Journal of physiology.

[109]  Yan Yang,et al.  Duration of complex-spikes grades Purkinje cell plasticity and cerebellar motor learning , 2014, Nature.

[110]  G. Hesslow,et al.  The secondary spikes of climbing fibre responses recorded from Purkinje cell axons in cat cerebellum. , 1986, The Journal of physiology.

[111]  C I De Zeeuw,et al.  Climbing fibre collaterals contact neurons in the cerebellar nuclei that provide a GABAergic feedback to the inferior olive. , 1997, Neuroscience.

[112]  T. Shiida,et al.  Visual influence on rabbit horizontal vestibulo-ocular reflex presumably effected via the cerebellar flocculus. , 1974, Brain research.

[113]  Zayd M. Khaliq,et al.  Axonal Propagation of Simple and Complex Spikes in Cerebellar Purkinje Neurons , 2005, The Journal of Neuroscience.

[114]  E. Bauswein,et al.  Simple and complex spike activity of cerebellar Purkinje cells during active and passive movements in the awake monkey. , 1983, The Journal of physiology.

[115]  R. Tsien,et al.  Long-term depression in cerebellar Purkinje neurons results from coincidence of nitric oxide and depolarization-induced Ca2+ transients , 1995, Neuron.

[116]  M. Howe,et al.  Rapid signaling in distinct dopaminergic axons during locomotion and reward , 2016, Nature.

[117]  J. Eccles,et al.  The excitatory synaptic action of climbing fibres on the Purkinje cells of the cerebellum , 1966, The Journal of physiology.

[118]  W. Schultz Responses of midbrain dopamine neurons to behavioral trigger stimuli in the monkey. , 1986, Journal of neurophysiology.

[119]  Reza Shadmehr,et al.  A memory of errors in sensorimotor learning , 2014, Science.

[120]  M. Sakurai Synaptic modification of parallel fibre‐Purkinje cell transmission in in vitro guinea‐pig cerebellar slices. , 1987, The Journal of physiology.

[121]  A. Fuchs,et al.  Complex spike activity signals the direction and size of dysmetric saccade errors. , 2008, Progress in brain research.

[122]  N H Barmack,et al.  Eye movements evoked by microstimulation of dorsal cap of inferior olive in the rabbit. , 1980, Journal of neurophysiology.

[123]  H. Jörntell,et al.  Relation Between Cutaneous Receptive Fields and Muscle Afferent Input to Climbing Fibres Projecting to the Cerebellar C3 Zone in the Cat , 1996, The European journal of neuroscience.

[124]  G. A. Clark,et al.  Initial localization of the memory trace for a basic form of learning. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[125]  David Attwell,et al.  Short‐ and long‐term depression of rat cerebellar parallel fibre synaptic transmission mediated by synaptic crosstalk , 2007, The Journal of physiology.

[126]  James V. Stone,et al.  Decorrelation control by the cerebellum achieves oculomotor plant compensation in simulated vestibulo-ocular reflex , 2002, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[127]  Rhea R. Kimpo,et al.  Gating of neural error signals during motor learning , 2014, eLife.

[128]  M. Mlonyeni,et al.  The number of Purkinje cells and inferior olivary neurones in the cat , 1973, The Journal of comparative neurology.

[129]  T. Ruigrok,et al.  Multiple cerebellar zones are involved in the control of individual muscles: a retrograde transneuronal tracing study with rabies virus in the rat , 2008, The European journal of neuroscience.

[130]  R. A. Hensbroek,et al.  Intraburst and Interburst Signaling by Climbing Fibers , 2007, The Journal of Neuroscience.

[131]  R. F. Thompson,et al.  Inhibitory cerebello-olivary projections and blocking effect in classical conditioning. , 1998, Science.

[132]  R. Lathe Phd by thesis , 1988, Nature.

[133]  D. Linden,et al.  Polarity of Long-Term Synaptic Gain Change Is Related to Postsynaptic Spike Firing at a Cerebellar Inhibitory Synapse , 1998, Neuron.

[134]  Rhea R. Kimpo,et al.  Cerebellar Purkinje cell activity drives motor learning , 2013, Nature Neuroscience.

[135]  P. Dean,et al.  Synaptic Plasticity in Medial Vestibular Nucleus Neurons: Comparison with Computational Requirements of VOR Adaptation , 2010, PloS one.

[136]  M. Garwicz,et al.  Functional relation between corticonuclear input and movements evoked on microstimulation in cerebellar nucleus interpositus anterior in the cat , 2004, Experimental Brain Research.

[137]  Shogo Ohmae,et al.  Climbing fibers encode a temporal-difference prediction error during cerebellar learning in mice , 2015, Nature Neuroscience.