When Money Is Not Enough: Awareness, Success, and Variability in Motor Learning

When performing a skill such as throwing a dart, many different combinations of joint motions suffice to hit the target. The motor system adapts rapidly to reduce bias in the desired outcome (i.e., the first-order moment of the error); however, the essence of skill is to produce movements with less variability (i.e., to reduce the second-order moment). It is easy to see how feedback about success or failure could sculpt performance to achieve this aim. However, it is unclear whether the dimensions responsible for success or failure need to be known explicitly by the subjects, or whether learning can proceed without explicit awareness of the movement parameters that need to change. Here, we designed a redundant, two-dimensional reaching task in which we could selectively manipulate task success and the variability of action outcomes, whilst also manipulating awareness of the dimension along which performance could be improved. Variability was manipulated either by amplifying natural errors, leaving the correlation between the executed movement and the visual feedback intact, or by adding extrinsic noise, decorrelating movement and feedback. We found that explicit, binary, feedback about success or failure was only sufficient for learning when participants were aware of the dimension along which motor behavior had to change. Without such awareness, learning was only present when extrinsic noise was added to the feedback, but not when task success or variability was manipulated in isolation; learning was also much slower. Our results highlight the importance of conscious awareness of the relevant dimension during motor learning, and suggest that higher-order moments of outcome signals are likely to play a significant role in skill learning in complex tasks.

[1]  J. Gittins Bandit processes and dynamic allocation indices , 1979 .

[2]  J. Pearce,et al.  A model for Pavlovian learning: variations in the effectiveness of conditioned but not of unconditioned stimuli. , 1980, Psychological review.

[3]  F A Mussa-Ivaldi,et al.  Adaptive representation of dynamics during learning of a motor task , 1994, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[4]  Richard S. Sutton,et al.  Generalization in Reinforcement Learning: Successful Examples Using Sparse Coarse Coding , 1995, NIPS.

[5]  Andrew W. Moore,et al.  Reinforcement Learning: A Survey , 1996, J. Artif. Intell. Res..

[6]  Andrew G. Barto,et al.  Reinforcement learning , 1998 .

[7]  Daniel M. Wolpert,et al.  Signal-dependent noise determines motor planning , 1998, Nature.

[8]  Gregor Schöner,et al.  The uncontrolled manifold concept: identifying control variables for a functional task , 1999, Experimental Brain Research.

[9]  Reza Shadmehr,et al.  Learning of action through adaptive combination of motor primitives , 2000, Nature.

[10]  S. Kakade,et al.  Learning and selective attention , 2000, Nature Neuroscience.

[11]  J. Kruschke Toward a unified model of attention in associative learning , 2001 .

[12]  T. Flash,et al.  Planning Movements in a Simple Redundant Task , 2002, Current Biology.

[13]  M. Latash,et al.  Motor Control Strategies Revealed in the Structure of Motor Variability , 2002, Exercise and sport sciences reviews.

[14]  Michael I. Jordan,et al.  Optimal feedback control as a theory of motor coordination , 2002, Nature Neuroscience.

[15]  Daniel M Wolpert,et al.  Role of uncertainty in sensorimotor control. , 2002, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[16]  Emanuel Todorov,et al.  Cosine Tuning Minimizes Motor Errors , 2002, Neural Computation.

[17]  Reza Shadmehr,et al.  Quantifying Generalization from Trial-by-Trial Behavior of Adaptive Systems that Learn with Basis Functions: Theory and Experiments in Human Motor Control , 2003, The Journal of Neuroscience.

[18]  J. Hore,et al.  Kinematics of wrist joint flexion in overarm throws made by skilled subjects , 2004, Experimental Brain Research.

[19]  Michael S Landy,et al.  Statistical decision theory and the selection of rapid, goal-directed movements. , 2003, Journal of the Optical Society of America. A, Optics, image science, and vision.

[20]  J. Hore,et al.  Timing of finger opening and ball release in fast and accurate overarm throws , 2004, Experimental Brain Research.

[21]  M. Landy,et al.  Optimal Compensation for Changes in Task-Relevant Movement Variability , 2005, The Journal of Neuroscience.

[22]  J. Hore,et al.  Timing finger opening in overarm throwing based on a spatial representation of hand path. , 2005, Journal of neurophysiology.

[23]  Ferdinando A Mussa-Ivaldi,et al.  Remapping hand movements in a novel geometrical environment. , 2005, Journal of neurophysiology.

[24]  Pieter R. Roelfsema,et al.  Attention-Gated Reinforcement Learning of Internal Representations for Classification , 2005, Neural Computation.

[25]  P. Dayan,et al.  Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control , 2005, Nature Neuroscience.

[26]  Yasmin L. Hashambhoy,et al.  Neural Correlates of Reach Errors , 2005, The Journal of Neuroscience.

[27]  J. Krakauer,et al.  An Implicit Plan Overrides an Explicit Strategy during Visuomotor Adaptation , 2006, The Journal of Neuroscience.

[28]  K. Shenoy,et al.  A Central Source of Movement Variability , 2006, Neuron.

[29]  J. Peters,et al.  Using Reward-weighted Regression for Reinforcement Learning of Task Space Control , 2007, 2007 IEEE International Symposium on Approximate Dynamic Programming and Reinforcement Learning.

[30]  J. Krakauer,et al.  Sensory prediction errors drive cerebellum-dependent adaptation of reaching. , 2007, Journal of neurophysiology.

[31]  Peter W Battaglia,et al.  Humans Trade Off Viewing Time and Movement Duration to Improve Visuomotor Accuracy in a Fast Reaching Task , 2007, The Journal of Neuroscience.

[32]  S. Gepshtein,et al.  Optimality of human movement under natural variations of visual-motor uncertainty. , 2007, Journal of vision.

[33]  Michael S. Landy,et al.  Optimal Compensation for Temporal Uncertainty in Movement Planning , 2008, PLoS Comput. Biol..

[34]  M. Landy,et al.  Decision making, movement planning and statistical decision theory , 2008, Trends in Cognitive Sciences.

[35]  Stefan Schaal,et al.  2008 Special Issue: Reinforcement learning of motor skills with policy gradients , 2008 .

[36]  A. Luft,et al.  Dopaminergic signals in primary motor cortex , 2009, International Journal of Developmental Neuroscience.

[37]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[38]  Dagmar Sternad,et al.  Motor learning: changes in the structure of variability in a redundant task. , 2009, Advances in experimental medicine and biology.

[39]  Konrad P. Körding,et al.  Exploration and Exploitation During Sequential Search , 2009, Cogn. Sci..

[40]  K. Molina-Luna,et al.  Dopamine in Motor Cortex Is Necessary for Skill Learning and Synaptic Plasticity , 2009, PloS one.

[41]  Olivier White,et al.  Use-Dependent and Error-Based Learning of Motor Behaviors , 2010, The Journal of Neuroscience.

[42]  Takeo Watanabe,et al.  Perceptual learning rules based on reinforcers and attention , 2010, Trends in Cognitive Sciences.

[43]  R. Ivry,et al.  The coordination of movement: optimal feedback control and beyond , 2010, Trends in Cognitive Sciences.

[44]  Maura Casadio,et al.  Reorganization of finger coordination patterns during adaptation to rotation and scaling of a newly learned sensorimotor transformation. , 2011, Journal of neurophysiology.

[45]  Heidi M. Schambra,et al.  Reward Improves Long-Term Retention of a Motor Memory through Induction of Offline Memory Gains , 2011, Current Biology.

[46]  Vincent S. Huang,et al.  Rethinking Motor Learning and Savings in Adaptation Paradigms: Model-Free Memory for Successful Actions Combines with Internal Models , 2011, Neuron.

[47]  Dagmar Sternad,et al.  Neuromotor Noise, Error Tolerance and Velocity-Dependent Costs in Skilled Performance , 2011, PLoS Comput. Biol..

[48]  Ana Pekanovic,et al.  Dopaminergic Projections from Midbrain to Primary Motor Cortex Mediate Motor Skill Learning , 2011, The Journal of Neuroscience.

[49]  Maura Casadio,et al.  Sensory motor remapping of space in human-machine interfaces. , 2011, Progress in brain research.

[50]  Reza Shadmehr,et al.  Learning from Sensory and Reward Prediction Errors during Motor Adaptation , 2011, PLoS Comput. Biol..

[51]  D. Wolpert,et al.  Principles of sensorimotor learning , 2011, Nature Reviews Neuroscience.

[52]  M. Landy,et al.  Dynamic Estimation of Task-Relevant Variance in Movement under Risk , 2012, The Journal of Neuroscience.

[53]  Kurt A. Thoroughman,et al.  Beside the point: motor adaptation without feedback-based error correction in task-irrelevant conditions. , 2012, Journal of neurophysiology.

[54]  Eli Brenner,et al.  Random walk of motor planning in task-irrelevant dimensions. , 2013, Journal of neurophysiology.

[55]  J. Krakauer,et al.  Motor skill depends on knowledge of facts , 2013, Front. Hum. Neurosci..

[56]  Richard S. Sutton,et al.  Reinforcement Learning , 1992, Handbook of Machine Learning.