Predictive Hebbian learning

established from the perspective of psychological experiments, the neural mechanisms that underlie this preA creature presented with an uncertain and diction are less well understood. Prediction, and its apvariable environment needs to anticipate inpropriate use for action, is essentially a computational portant future events or risk diminished chanconcept, but this still leads a wide range of possible theces for survival. These events can include ories to explain existing data. the presence of food, destructive stimuli, and One way for an animal to learn to make predictions is for potential mates. In short, a nervous system it to have a system that reports on its current best guess, must have means to generate guesses about its and to have learning be contingent on err-or-s in this premost likely next state and the most likely next diction: learning only happens if the animal becomes state of the world. Psychologists have studied surprised based on its prediction. This is the underconditions under which animals can learn to lying mechanism behind essentially all adaptation rules predict future reward and punishment. In this in engineering (Kalman, 1960; Widrow & Stearns, 1985) paper, we review the computational theory that may be relevant for understanding this and particular learning rules in psychology (Rescorla & form of learning. Some of the central nlechWagner, 1972; Mackintosh, 1983; Pearce & Hall, 1980). JVe consider below the general requirements for such a anisms required for predictive learning have been discovered in both vertebrate Ljungberg signal in the brain. et al’s (1992) and invertebrate brains (HanlThe construction, delivery and use of an error signal remer, 1994). lated to predictions about future stimuli would require the following: Z) access to a representation of the phenomenon to be Prediction predicted such as the amount of reward or food. t~) accem to the current predictions so that they can be Animals are capable of predicting events aud the cousecolnparecl to the phenonlenon to be predicted. quences of those events on the basis of the sensory inforLIL) capacity to influence plasticity (directly or indirectly) ]U structures responsible for constructing the predication they receive and directing their actions accorcltious. ing to those predictions (see Dickenson, 1980; Mackintosh, 1983; Gallistel, 1990 and Gluck and Thompson, ZU) sufficiently wide broadcast of the error signal so that 1987 for reviews). Although these conclusions are well stimuli in different modalities can be used to make and respond to the predictions. *Howard Hughes Medical Institute, The Salk Institute for Biological Studies, 10010 North Torrey Pines Road, La These general requirements are met by a number of difJolla, CA 92037, and Department of Biology, University of fusely projecting system of axons that are thought to California, San Diego, La Jolla, CA 92093. report, in part on salient events in the world and within t Department of Brain and Cognitive Science, MIT, Canlthe organism. bridge MA 02139. These axons release neuromodulators t Division of Neuroscience, Baylor College of Medicine, that can influence the effectiveness of synapses in these Houston TX 77030. areas. This anatomical motif is a common feature of many nervous systems and is not unique to vertebrate Permission to make digital/hard copies of all or part of this material withbrains. Invertebrates have analogous sets of neurons out fee is granted provided that the copies are not made or distributed for profit or commercial advantage, the ACM copyright/server with extensive axonal arborizations that deliver neuronotice, the title of the pub Iication and its date appear, and notice is given modulators to widespread target regions (Hawkins and that copyright is by permission of the Association for Computing Machinery, Kandel, 1984; Greenough and Bailey, 1988; Hammer, Inc. (ACM). To copy otherwise, to republish,to post on servers or to 1994). Experiulent al evidence from both behavioral and redistribute to lists, requires specific permission and/or fee. COLT’ ’95 Santa Cruz, CA USA@ 1995 ACM 0-89723-5/95/0007. .$3.50 physiological work suggests that these systems influence

[1]  P. Dayan,et al.  A framework for mesencephalic dopamine systems based on predictive Hebbian learning , 1996, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[2]  M. Hammer An identified neuron mediates the unconditioned stimulus in associative olfactory learning in honeybees , 1993, Nature.

[3]  W. Schultz,et al.  Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response task , 1993, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[4]  J. Pearce,et al.  A model for Pavlovian learning: Variations in the effectiveness of conditioned but not of unconditioned stimuli. , 1980 .

[5]  E. Kandel,et al.  Is there a cell-biological alphabet for simple forms of learning? , 1984 .

[6]  W. Schultz,et al.  Responses of monkey dopamine neurons during learning of behavioral reactions. , 1992, Journal of neurophysiology.

[7]  J. Pearce,et al.  A model for Pavlovian learning: variations in the effectiveness of conditioned but not of unconditioned stimuli. , 1980, Psychological review.

[8]  R. Rescorla A theory of pavlovian conditioning: The effectiveness of reinforcement and non-reinforcement , 1972 .

[9]  C. H. Bailey,et al.  The anatomy of a memory: convergence of results across a diversity of tests , 1988, Trends in Neurosciences.

[10]  Bernard Widrow,et al.  Adaptive Signal Processing , 1985 .

[11]  J. Rauschecker,et al.  Mechanisms of visual plasticity: Hebb synapses, NMDA receptors, and beyond. , 1991, Physiological reviews.

[12]  P. Dayan Computational modelling , 1994, Current Opinion in Neurobiology.

[13]  Richard S. Sutton,et al.  Time-Derivative Models of Pavlovian Reinforcement , 1990 .

[14]  R. F. Thompson,et al.  Modeling the neural substrates of associative learning and memory: a computational approach. , 1987, Psychological review.

[15]  A G Barto,et al.  Toward a modern theory of adaptive networks: expectation and prediction. , 1981, Psychological review.