Brain-Inspired Motion Learning in Recurrent Neural Network With Emotion Modulation

Based on basic emotion modulation theory and the neural mechanisms of generating complex motor patterns, we introduce a novel emotion-modulated learning rule to train a recurrent neural network, which enables a complex musculoskeletal arm and a robotic arm to perform goal-directed tasks with high accuracy and learning efficiency. Specifically, inspired by the fact that emotions can modulate the process of learning and decision making through neuromodulatory system, we present a model of emotion generation and modulation to adjust the parameters of learning adaptively, including the reward prediction error, the speed of learning, and the randomness in action selection. Additionally, we use Oja learning rule to adjust the recurrent weights in delayed-reinforcement tasks, which outperforms the Hebbian update rule in terms of stability and accuracy. In the experimental section, we use a musculoskeletal model of the human upper limb and a robotic arm to perform goal-directed tasks through trial-and-reward learning, respectively. The results show that emotion-based methods are able to control the arm with higher accuracy and a faster learning rate. Meanwhile, emotional Oja agent is superior to emotional Hebbian one in term of performance.

[1]  H. Seung,et al.  Model of birdsong learning based on gradient estimation by dynamic perturbation of neural conductances. , 2007, Journal of neurophysiology.

[2]  Peijie Yin,et al.  Human-inspired motion model of upper-limb with fast response and learning ability – a promising direction for robot system and control , 2016 .

[3]  Joseph E LeDoux,et al.  A call to action: overcoming anxiety through active coping. , 2001, The American journal of psychiatry.

[4]  E. Rolls Limbic systems for emotion and for memory, but no single limbic system , 2015, Cortex.

[5]  B. Richmond,et al.  Neuronal Signals in the Monkey Ventral Striatum Related to Progress through a Predictable Series of Trials , 1998, The Journal of Neuroscience.

[6]  T. Klingberg Training and plasticity of working memory , 2010, Trends in Cognitive Sciences.

[7]  P. Goldman-Rakic,et al.  Dopamine synaptic complex with pyramidal neurons in primate cerebral cortex. , 1989, Proceedings of the National Academy of Sciences of the United States of America.

[8]  Elizabeth Tricomi,et al.  Feedback signals in the caudate reflect goal achievement on a declarative memory task , 2008, NeuroImage.

[9]  E. Capaldi,et al.  The organization of behavior. , 1992, Journal of applied behavior analysis.

[10]  Amir F. Atiya,et al.  New results on recurrent network training: unifying the algorithms and accelerating convergence , 2000, IEEE Trans. Neural Networks Learn. Syst..

[11]  Christopher D. Harvey,et al.  Recurrent Network Models of Sequence Generation and Memory , 2016, Neuron.

[12]  Erkki Oja,et al.  Neural Networks, Principal Components, and Subspaces , 1989, Int. J. Neural Syst..

[13]  P. Petta,et al.  Computational models of emotion , 2010 .

[14]  Herbert Jaeger,et al.  Echo state network , 2007, Scholarpedia.

[15]  Karolina M. Lempert,et al.  Emotion and decision making: multiple modulatory neural circuits. , 2014, Annual review of neuroscience.

[16]  H. Lövheim A new three-dimensional model for emotions and monoamine neurotransmitters. , 2012, Medical hypotheses.

[17]  M. Hasselmo Neuromodulation: acetylcholine and memory consolidation , 1999, Trends in Cognitive Sciences.

[18]  John N. J. Reynolds,et al.  Dopamine-dependent plasticity of corticostriatal synapses , 2002, Neural Networks.

[19]  E. Miller,et al.  Effects of Visual Experience on the Representation of Objects in the Prefrontal Cortex , 2000, Neuron.

[20]  E. Izhikevich Solving the distal reward problem through linkage of STDP and dopamine signaling , 2007, BMC Neuroscience.

[21]  R. Lazarus Emotion and Adaptation , 1991 .

[22]  Wolfgang Maass,et al.  Emergence of complex computational structures from chaotic neural networks through reward-modulated Hebbian learning. , 2014, Cerebral cortex.

[23]  W. Schultz Reward functions of the basal ganglia , 2016, Journal of Neural Transmission.

[24]  Henning Sprekeler,et al.  Functional Requirements for Reward-Modulated Spike-Timing-Dependent Plasticity , 2010, The Journal of Neuroscience.

[25]  K. Doya Metalearning, neuromodulation, and emotion , 2000 .

[26]  Peijie Yin,et al.  Improving learning efficiency of recurrent neural network through adjusting weights of all layers in a biologically-inspired framework , 2017, 2017 International Joint Conference on Neural Networks (IJCNN).

[27]  L. F. Abbott,et al.  Generating Coherent Patterns of Activity from Chaotic Neural Networks , 2009, Neuron.

[28]  James N. Druckman,et al.  Emotion and the Framing of Risky Choice , 2008 .

[29]  J.J. Steil,et al.  Backpropagation-decorrelation: online recurrent learning with O(N) complexity , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).

[30]  Mathieu Cassotti,et al.  Fear and anger have opposite effects on risk seeking in the gain frame , 2015, Front. Psychol..

[31]  R. J. Williams,et al.  Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning , 2004, Machine Learning.

[32]  Robert E Kass,et al.  Functional network reorganization during learning in a brain-computer interface paradigm , 2008, Proceedings of the National Academy of Sciences.

[33]  John D E Gabrieli,et al.  Bottom-Up and Top-Down Processes in Emotion Generation , 2009, Psychological science.

[34]  Anca Velisar,et al.  Benchmarking of dynamic simulation predictions in two software platforms using an upper limb musculoskeletal model , 2015, Computer methods in biomechanics and biomedical engineering.

[35]  D. O. Hebb,et al.  The organization of behavior , 1988 .

[36]  Paolo Dario,et al.  Bio-inspired kinematical control of redundant robotic manipulators , 2016 .

[37]  Jonathan D. Cohen,et al.  Journal of Economic Perspectives—Volume 19, Number 4—Fall 2005—Pages 3–24 The Vulcanization of the Human Brain: A Neural Perspective on Interactions Between Cognition and Emotion , 2022 .

[38]  Heather L. Urry,et al.  Cognition and Emotion Lecture at the 2010 SPSP Emotion Preconference , 2011, Cognition & emotion.

[39]  Peter Brown,et al.  Midline Frontal Cortex Low-Frequency Activity Drives Subthalamic Nucleus Oscillations during Conflict , 2014, The Journal of Neuroscience.

[40]  C. Büchel,et al.  The neural bases of emotion regulation , 2015, Nature Reviews Neuroscience.

[41]  C. Carter,et al.  The Timing of Action-Monitoring Processes in the Anterior Cingulate Cortex , 2002, Journal of Cognitive Neuroscience.

[42]  Guangyu R. Yang,et al.  Training Excitatory-Inhibitory Recurrent Neural Networks for Cognitive Tasks: A Simple and Flexible Framework , 2016, PLoS Comput. Biol..

[43]  Joseph E LeDoux,et al.  Contributions of the Amygdala to Emotion Processing: From Animal Models to Human Behavior , 2005, Neuron.

[44]  Antonio R. Damasio,et al.  Emotion and the Human Brain , 2001 .

[45]  Kenji Doya,et al.  Metalearning and neuromodulation , 2002, Neural Networks.

[46]  Koh Hosoda,et al.  Development of a tendon-driven robotic finger for an anthropomorphic robotic hand , 2014, Int. J. Robotics Res..

[47]  Ila R Fiete,et al.  Gradient learning in spiking neural networks by dynamic perturbation of conductances. , 2006, Physical review letters.

[48]  E. Oja Simplified neuron model as a principal component analyzer , 1982, Journal of mathematical biology.

[49]  Catholijn M. Jonker,et al.  Emotion in reinforcement learning agents and robots: a survey , 2017, Machine Learning.

[50]  Joseph E LeDoux The emotional brain , 1996 .

[51]  Thomas Miconi Biologically plausible learning in recurrent neural networks reproduces neural dynamics observed during cognitive tasks , 2017 .

[52]  Joseph E LeDoux Emotion Circuits in the Brain , 2000 .

[53]  P. Glimcher,et al.  Midbrain Dopamine Neurons Encode a Quantitative Reward Prediction Error Signal , 2005, Neuron.

[54]  Wolfgang Maass,et al.  A Reward-Modulated Hebbian Learning Rule Can Explain Experimentally Observed Network Reorganization in a Brain Control Task , 2010, The Journal of Neuroscience.

[55]  M. Hasselmo The role of acetylcholine in learning and memory , 2006, Current Opinion in Neurobiology.

[56]  A G Barto,et al.  Toward a modern theory of adaptive networks: expectation and prediction. , 1981, Psychological review.

[57]  H. Barbas,et al.  Anatomic basis of cognitive-emotional interactions in the primate prefrontal cortex , 1995, Neuroscience & Biobehavioral Reviews.

[58]  Alexandru D. Iordan,et al.  Neural correlates of emotion–cognition interactions: A review of evidence from brain imaging investigations , 2011, Journal of cognitive psychology.