A neuronal model of classical conditioning

A neuronal model of classical conditioning is proposed. The model is most easily described by contrasting it with a still influential neuronal model first analyzed by Hebb (1949). It is proposed that the Hebbian model be modified in three ways to yield a model more in accordance with animal learning phenomena. First, instead of correlating pre- and postsynaptic levels of activity, changes in pre- and postsynaptic levels of activity should be correlated to determine the changes in synaptic efficacy that represent learning. Second, instead of correlating approximately simultaneous pre- and postsynaptic signals, earlier changes in presynaptic signals should be correlated with later changes in postsynaptic signals. Third, a change in the efficacy of a synapse should be proportional to the current efficacy of the synapse, accounting for the initial positive acceleration in the S-shaped acquisition curves observed in animal learning. The resulting model, termed a drive-reinforcement model of single neuron function, suggests that nervous system activity can be understood in terms of two classes of neuronal signals: drives that are defined to be signal levels and reinforcers that are defined to be changes in signal levels. Defining drives and reinforcers in this way, in conjunction with the neuronal model, suggests a basis for a neurobiological theory of learning. The proposed neuronal model is an extension of the Sutton-Barto (1981) model, which in turn can be seen as a temporally refined extension of the Rescorla-Wagner (1972) model. It is shown that the proposed neuronal model predicts a wide range of classical conditioning phenomena, including delay and trace conditioning, conditioned and unconditioned stimulus duration and amplitude effects, partial reinforcement effects, interstimulus interval effects, second-order conditioning, conditioned inhibition, extinction, reacquisition effects, backward conditioning, blocking, overshadowing, compound conditioning, and discriminative stimulus effects. The neuronal model also eliminates some inconsistencies with the experimental evidence that occur with the Rescorla-Wagner and Sutton-Barto models. Implications of the neuronal model for animal learning theory, connectionist and neural network modeling, artificial intelligence, adaptive control theory, and adaptive signal processing are discussed. It is concluded that real-time learning mechanisms that do not require evaluative feedback from the environment are fundamental to natural intelligence and may have implications for artificial intelligence. Experimental tests of the model are suggested.

[1]  E. Custers,et al.  Psychology of Learning. , 1921 .

[2]  I. Pavlov,et al.  Lectures on conditioned reflexes , 1928 .

[3]  B. Skinner,et al.  Principles of Behavior , 1944 .

[4]  E. Hilgard,et al.  Theories of Learning , 1981 .

[5]  S. S. Stevens,et al.  Handbook of experimental psychology , 1951 .

[6]  D. Mcclelland,et al.  The Achievement Motive , 1954 .

[7]  S. Freud The Standard Edition of the Complete Psychological Works of Sigmund Freud , 1953 .

[8]  W. R. Mcallister Eyelid conditioning as a function of the CS-US interval. , 1953, Journal of Experimental Psychology.

[9]  N. Miller,et al.  Learning motivated by electrical stimulation of the brain. , 1954, The American journal of physiology.

[10]  James L Olds,et al.  Positive reinforcement produced by electrical stimulation of septal area and other regions of rat brain. , 1954, Journal of comparative and physiological psychology.

[11]  W. A. Clark,et al.  Simulation of self-organizing systems by digital computer , 1954, Trans. IRE Prof. Group Inf. Theory.

[12]  D. Premack Toward empirical behavior laws. I. positive reinforcement. , 1959, Psychological review.

[13]  J. H. Curtis,et al.  Learning Theory and Behavior , 1960 .

[14]  A. A. Mullin,et al.  Principles of neurodynamics , 1962 .

[15]  E. Feigenbaum,et al.  Computers and Thought , 1963 .

[16]  H. Helson,et al.  Adaptation-level theory , 1964 .

[17]  B. L. Ginsborg THE PHYSIOLOGY OF SYNAPSES , 1964 .

[18]  Marshall C. Yovits,et al.  Self-organizing systems 1962 , 1964 .

[19]  W. McCulloch,et al.  Embodiments of Mind , 1966 .

[20]  W. F. Prokasy,et al.  Classical conditioning : a symposium , 1965 .

[21]  Aspects of learning and memory , 1966 .

[22]  I. Russell Animal Learning and Memory , 1966 .

[23]  L. Kamin Predictability, surprise, attention, and conditioning , 1967 .

[24]  R. Bolles Theory of Motivation , 1967 .

[25]  J. Konorski Integrative activity of the brain : an interdisciplinary approach , 1967 .

[26]  L. Kamin Attention-like processes in classical conditioning , 1967 .

[27]  K. Haberlandt,et al.  Stimulus selection in animal discrimination learning. , 1968, Journal of experimental psychology.

[28]  P. W. Frey,et al.  Classical conditioning of the rabbit eyelid response as a function of interstimulus interval. , 1968, Journal of comparative and physiological psychology.

[29]  P. L. Brown,et al.  Auto-shaping of the pigeon's key-peck. , 1968, Journal of the experimental analysis of behavior.

[30]  R. L. Barron Self-organizing and learning control systems. , 1968 .

[31]  D. Bindra Neuropsychological interpretation of the effects of drive and incentive-motivation on general activity and instrumental behavior. , 1968 .

[32]  Hans L. And Darrell R. Moore Oestreicher Cybernetic Problems in Bionics , 1968 .

[33]  E. Kandel,et al.  Cellular neurophysiological approaches in the study of learning. , 1968, Physiological reviews.

[34]  R. Rescorla Pavlovian conditioned inhibition , 1969 .

[35]  JOHN W. Moore,et al.  Auditory differential conditioning of the rabbit nictitating membrane response: III. Effects of US shock intensity and duration , 1969 .

[36]  B. Campbell,et al.  Punishment and aversive behavior , 1969 .

[37]  A. Klopf,et al.  An Evolutionary Pattern Recognition Network , 1969 .

[38]  Component strength in a compound CS as a function of number of acquisition trials. , 1969 .

[39]  L. Squire Mechanisms of memory. , 1986, Lancet.

[40]  S. R. Coleman,et al.  Classical conditioning of the rabbit's nictitating membrane response at backward, simultaneous, and forward CS-US intervals. , 1969, Journal of comparative and physiological psychology.

[41]  N. Mackintosh,et al.  Mechanisms of animal discrimination learning , 1971 .

[42]  R. Rescorla Variation in the effectiveness of reinforcement and nonreinforcement following prior inhibitory conditioning , 1971 .

[43]  R. Rescorla Summation and retardation tests of latent inhibition. , 1971, Journal of comparative and physiological psychology.

[44]  K. Spence,et al.  Essays in neobehaviorism : a memorial volume to Kenneth W. Spence , 1971 .

[45]  A. R. Wagner,et al.  CS habituation produces a “latent inhibition effect” but no active “conditioned inhibition” , 1972 .

[46]  R. Bolles Reinforcement, expectancy, and learning. , 1972 .

[47]  R. Rescorla,et al.  A theory of Pavlovian conditioning : Variations in the effectiveness of reinforcement and nonreinforcement , 1972 .

[48]  W. F. Prokasy,et al.  Classical conditioning II: Current research and theory. , 1972 .

[49]  K. B. Madsen,et al.  Pleasure, reward, preference : their nature, determinants, and role in behavior , 1973 .

[50]  Bernard Widrow,et al.  Punish/Reward: Learning with a Critic in Adaptive Threshold Systems , 1973, IEEE Trans. Syst. Man Cybern..

[51]  JOHN W. Moore,et al.  Effect of preacquisition UCS exposure on classical conditioning of the rabbit's nictitating membrane response☆ , 1973 .

[52]  D. Berlyne CHAPTER 1 – The Vicissitudes of Aplopathematic and Thelematoscopic Pneumatology (or The Hydrography of Hedonism)1 , 1973 .

[53]  T. Bliss,et al.  Long‐lasting potentiation of synaptic transmission in the dentate area of the anaesthetized rabbit following stimulation of the perforant path , 1973, The Journal of physiology.

[54]  H. M. Jenkins,et al.  The form of the auto-shaped response with food or water reinforcers. , 1973, Journal of the experimental analysis of behavior.

[55]  N. Mackintosh The psychology of animal learning , 1974 .

[56]  P. Werbos,et al.  Beyond Regression : "New Tools for Prediction and Analysis in the Behavioral Sciences , 1974 .

[57]  JOHN W. Moore,et al.  Latent inhibition of the rabbit’s nictitating membrane response as a function of CS intensity , 1974 .

[58]  C. L. Hull A behavior system: An introduction to behavior theory concerning the individual organism , 1974 .

[59]  A. R. Wagner,et al.  Backward conditioning to a CS following an expected vs. a surprising UCS , 1975 .

[60]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[61]  W. Calvin Generation of spike trains in CNS neurons , 1975, Brain Research.

[62]  J. Gray,et al.  Elements of a two-process theory of learning , 1975 .

[63]  N. Mackintosh A Theory of Attention: Variations in the Associability of Stimuli with Reinforcement , 1975 .

[64]  A. Harry Klopf,et al.  A comparison of natural and artificial intelligence , 1975, SGAR.

[65]  J. J. Keene Reward-associated excitation and pain-associated inhibition lasting seconds in rat medial pallidal units , 1975, Experimental Neurology.

[66]  D. Bindra A theory of intelligent behavior , 1976 .

[67]  J. Ayres,et al.  One-trial simultaneous and backward fear conditioning as reflected in conditioned suppression of licking in rats , 1976 .

[68]  Raymond J. Bandlow Theories of Learning, 4th Edition. By Ernest R. Hilgard and Gordon H. Bower. Englewood Cliffs, N.J.: Prentice-Hall, Inc., 1975 , 1976 .

[69]  C. Heth Simultaneous and backward fear conditioning as a function of number of CS-UCS pairings. , 1976, Journal of experimental psychology. Animal behavior processes.

[70]  R. F. Thompson,et al.  The search for the engram. , 1976, The American psychologist.

[71]  Stephen A. Ritz,et al.  Distinctive features, categorical perception, and probability learning: some applications of a neural model , 1977 .

[72]  Ian H. Witten,et al.  An Adaptive Optimal Controller for Discrete-Time Markov Environments , 1977, Inf. Control..

[73]  L. Stein,et al.  Neuropharmacology of Reward and Punishment , 1977 .

[74]  James L Olds Drives and reinforcements : behavioral studies of hypothalamic functions / by James Olds , 1977 .

[75]  Alan S. Brown,et al.  Fundamentals and applications of learning , 1977 .

[76]  G. D. Steinhauer,et al.  A procedure for autoshaping the pigeon's key peck to an auditory stimulus. , 1977, Journal of the experimental analysis of behavior.

[77]  C. Flaherty Learning and memory , 1977 .

[78]  V. Mountcastle,et al.  Parietal lobe mechanisms for directed visual attention. , 1977, Journal of neurophysiology.

[79]  A. D. McClellan,et al.  Learning: a model system for physiological studies. , 1978, Science.

[80]  J. Ayres,et al.  CS and US duration effects in one-trial simultaneous fear conditioning as assessed by conditioned suppression of licking in rats , 1978 .

[81]  K. Spence Behavior Theory and Conditioning , 1978 .

[82]  D. Bindra How adaptive behavior is produced: a perceptual-motivational alternative to response reinforcements , 1978, Behavioral and Brain Sciences.

[83]  A. Dickinson,et al.  Classical conditioning in animals. , 1978, Annual review of psychology.

[84]  E. Kremer The Rescorla-Wagner model: losses in associative strength in compound conditioned stimuli. , 1978, Journal of experimental psychology. Animal behavior processes.

[85]  B. Schwartz Psychology of Learning and Behavior , 1978 .

[86]  G. Glaser,et al.  Antiepileptic drugs : mechanisms of action , 1980 .

[87]  E. Roberts Epilepsy and antiepileptic drugs: a speculative synthesis. , 1980, Advances in neurology.

[88]  John W. Moore A Memorial to A. H. Black , 1980 .

[89]  JOHN W. Moore,et al.  Erratum to: Formation of attentional-associative networks in real time: Role of the hippocampus and implications for conditioning , 1980 .

[90]  W. F. Hill,et al.  Mechanisms of Learning and Motivation: A Memorial Volume to Jerzy Konorski , 1980 .

[91]  E. Kandel,et al.  Classical conditioning in a simple withdrawal reflex in Aplysia californica , 1981, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[92]  A. Baranyi,et al.  Long-term facilitation of excitatory synaptic transmission in single motor cortical neurones of the cat produced by repetitive pairing of synaptic potentials and action potentials following intracellular stimulation , 1981, Neuroscience Letters.

[93]  A G Barto,et al.  Toward a modern theory of adaptive networks: expectation and prediction. , 1981, Psychological review.

[94]  A. Baranyi,et al.  Synaptic facilitation requires paired activation of convergent pathways in the neocortex , 1981, Nature.

[95]  P. Balsam,et al.  Rapid contextual conditioning in autoshaping. , 1981, Journal of experimental psychology. Animal behavior processes.

[96]  Richard S. Sutton,et al.  Goal Seeking Components for Adaptive Intelligence: An Initial Assessment. , 1981 .

[97]  M. Bitterman,et al.  Compound conditioning in honeybees. , 1982 .

[98]  Masao Ito,et al.  Climbing fibre induced depression of both mossy fibre responsiveness and glutamate sensitivity of cerebellar Purkinje cells , 1982, The Journal of physiology.

[99]  Stephen Grossberg,et al.  Studies of mind and brain , 1982 .

[100]  Charles D. Woody,et al.  Memory, learning, and higher function , 1982 .

[101]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[102]  Kunihiko Fukushima,et al.  Neocognitron: A Self-Organizing Neural Network Model for a Mechanism of Visual Pattern Recognition , 1982 .

[103]  Lashon B. Booker,et al.  Intelligent Behavior as an Adaptation to the Task Environment , 1982 .

[104]  R. Sutton,et al.  Simulation of anticipatory responses in classical conditioning by a neuron-like adaptive element , 1982, Behavioural Brain Research.

[105]  S. Amari,et al.  Competition and Cooperation in Neural Nets , 1982 .

[106]  Stephen Grossberg,et al.  Absolute stability of global pattern formation and parallel memory storage by competitive neural networks , 1983, IEEE Transactions on Systems, Man, and Cybernetics.

[107]  E. Kandel,et al.  A cellular mechanism of classical conditioning in Aplysia: activity-dependent amplification of presynaptic facilitation. , 1983, Science.

[108]  Daniel Levine Neural Population Modeling and Psychology: A Review , 1983 .

[109]  John S. Edwards,et al.  The Hedonistic Neuron: A Theory of Memory, Learning and Intelligence , 1983 .

[110]  J. Byrne,et al.  Associative conditioning of single sensory neurons suggests a cellular mechanism for learning. , 1983, Science.

[111]  Richard S. Sutton,et al.  Neuronlike adaptive elements that can solve difficult learning control problems , 1983, IEEE Transactions on Systems, Man, and Cybernetics.

[112]  Steven Edward Hampson,et al.  A neural model of adaptive behavior , 1983 .

[113]  R. F. Thompson,et al.  Cellular processes of learning and memory in the mammalian CNS. , 1983, Annual review of neuroscience.

[114]  Ralph R. Miller,et al.  Information processing in animals : memory mechanisms , 1983 .

[115]  Richard S. Sutton,et al.  Temporal credit assignment in reinforcement learning , 1984 .

[116]  J J Hopfield,et al.  Neurons with graded response have collective computational properties like those of two-state neurons. , 1984, Proceedings of the National Academy of Sciences of the United States of America.

[117]  F. Crick Function of the thalamic reticular complex: the searchlight hypothesis. , 1984, Proceedings of the National Academy of Sciences of the United States of America.

[118]  E. Kandel,et al.  A test of Hebb's postulate at identified synapses which mediate classical conditioning in Aplysia , 1984, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[119]  M. Ito,et al.  The modifiable neuronal network of the cerebellum. , 1984, The Japanese journal of physiology.

[120]  P. Anandan,et al.  Pattern-recognizing stochastic learning automata , 1985, IEEE Transactions on Systems, Man, and Cybernetics.

[121]  JOHN W. Moore,et al.  Real-time attentional models for classical conditioning and the hippocampus , 1985 .

[122]  Ralph R. Miller,et al.  Information processing in animals : conditioned inhibition , 1985 .

[123]  Allen I. Selverston,et al.  Model Neural Networks and Behavior , 1985, Springer US.

[124]  William B. Levy,et al.  Synaptic modification, neuron selectivity, and nervous system organization , 1985 .

[125]  J. Hopfield,et al.  The Logic of Limax Learning , 1985 .

[126]  Psychobiology: the neurobiology of motivation and reward. , 1985, Science.

[127]  Jerome A. Feldman,et al.  Connectionist Models and Their Applications: Introduction , 1985 .

[128]  Bernard Widrow,et al.  Adaptive Signal Processing , 1985 .

[129]  C. Flaherty Animal Learning and Cognition , 1985 .

[130]  A G Barto,et al.  Learning by statistical cooperation of self-interested neuron-like computing elements. , 1985, Human neurobiology.

[131]  Geoffrey E. Hinton,et al.  A Learning Algorithm for Boltzmann Machines , 1985, Cogn. Sci..

[132]  D. Alkon,et al.  Cellular mechanisms of learning, memory, and information storage. , 1985, Annual review of psychology.

[133]  A. R. Wagner,et al.  The effects of signaling the US in backward conditioning: A shift from excitatory to inhibitory learning , 1985 .

[134]  D. Zipser A model of hippocampal learning during classical conditioning. , 1986, Behavioral neuroscience.

[135]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[136]  J. Gould The biology of learning. , 1986, Annual review of psychology.

[137]  S. Kelso,et al.  Differential conditioning of associative synaptic enhancement in hippocampal brain slices. , 1986, Science.

[138]  T. Carew,et al.  Invertebrate learning and memory: from behavior to molecules. , 1986, Annual review of neuroscience.

[139]  A. Harry Klopf,et al.  A drive-reinforcement model of single neuron function , 1987 .

[140]  J. Hopfield,et al.  Computing with neural circuits: a model. , 1986, Science.

[141]  Geoffrey E. Hinton,et al.  Experiments on Learning by Back Propagation. , 1986 .

[142]  John H. Holland,et al.  Escaping brittleness: the possibilities of general-purpose learning algorithms applied to parallel rule-based systems , 1995 .

[143]  James L. McClelland,et al.  Psychological and biological models , 1986 .

[144]  S. Kelso,et al.  Hebbian synapses in hippocampus. , 1986, Proceedings of the National Academy of Sciences of the United States of America.

[145]  Geoffrey E. Hinton,et al.  Learning and relearning in Boltzmann machines , 1986 .

[146]  E. Kandel,et al.  Effects of interstimulus interval and contingency on classical conditioning of the Aplysia siphon withdrawal reflex , 1986, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[147]  G. Lynch,et al.  Induction of synaptic potentiation in hippocampus by patterned stimulation involves two events. , 1986, Science.

[148]  James L. McClelland,et al.  Parallel Distributed Processing: Explorations in the Microstructure of Cognition : Psychological and Biological Models , 1986 .

[149]  R. Sutton,et al.  Simulation of the classically conditioned nictitating membrane response by a neuron-like adaptive element: Response topography, neuronal firing, and interstimulus intervals , 1986, Behavioural Brain Research.

[150]  Masao Ito Long-term depression as a memory process in the cerebellum , 1986, Neuroscience Research.

[151]  C. Woody Understanding the cellular basis of memory and learning. , 1986, Annual review of psychology.

[152]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[153]  B. Kosco Differential Hebbian learning , 1987 .

[154]  Richard F. Thompson,et al.  Modeling the Neural Substrates of Associative Learning and Memory: A Computational Approach , 1987 .

[155]  J. Byrne Cellular analysis of associative learning. , 1987, Physiological reviews.

[156]  J. Ayres,et al.  Extending conditioned stimuli before versus after unconditioned stimuli: Implications for real-time models of conditioning. , 1987 .

[157]  N. Donegan,et al.  Conditioned diminution and facilitation of the UR: A sometimes opponent-process interpretation. , 1987 .

[158]  JOHN W. Moore,et al.  Simulation of a Classically Conditioned Response: Components of the Input Trace and a Cerebellar Neural Network Implementation of the Sutton-Barto-Desmond Model. , 1987 .

[159]  R. Rescorla Behavioral studies of Pavlovian conditioning. , 1988, Annual review of neuroscience.

[160]  D. O. Hebb,et al.  The organization of behavior , 1988 .

[161]  S. Iversen,et al.  Handbook of Psychopharmacology , 1988, Springer US.

[162]  Y. Frégnac The Neural and Molecular Bases of Learning (Life Sciences Report 38 — Dahlem Konferenzen) edited by J. P. Changeux and M. Konishi, John Wiley & Sons, 1987. £55.00 (xiii + 559 pages) ISBN 0 471 91569 6 , 1988, Trends in Neurosciences.

[163]  E. Kehoe A layered network model of associative learning: learning to learn and configuration. , 1988, Psychological review.

[164]  Geoffrey E. Hinton,et al.  Parallel Models of Associative Memory , 1989 .

[165]  Teuvo Kohonen,et al.  Self-Organization and Associative Memory, Third Edition , 1989, Springer Series in Information Sciences.

[166]  W. Pitts,et al.  A Logical Calculus of the Ideas Immanent in Nervous Activity (1943) , 2021, Ideas That Created the Future.

[167]  L. Squire,et al.  The Neuropsychology of Memory , 1990 .