Dynamical Movement Primitives: Learning Attractor Models for Motor Behaviors

Nonlinear dynamical systems have been used in many disciplines to model complex behaviors, including biological motor control, robotics, perception, economics, traffic prediction, and neuroscience. While often the unexpected emergent behavior of nonlinear systems is the focus of investigations, it is of equal importance to create goal-directed behavior (e.g., stable locomotion from a system of coupled oscillators under perceptual guidance). Modeling goal-directed behavior with nonlinear systems is, however, rather difficult due to the parameter sensitivity of these systems, their complex phase transitions in response to subtle parameter changes, and the difficulty of analyzing and predicting their long-term behavior; intuition and time-consuming parameter tuning play a major role. This letter presents and reviews dynamical movement primitives, a line of research for modeling attractor behaviors of autonomous nonlinear dynamical systems with the help of statistical learning techniques. The essence of our approach is to start with a simple dynamical system, such as a set of linear differential equations, and transform those into a weakly nonlinear system with prescribed attractor dynamics by means of a learnable autonomous forcing term. Both point attractors and limit cycle attractors of almost arbitrary complexity can be generated. We explain the design principle of our approach and evaluate its properties in several example applications in motor control and robotics.

[1]  F. Huddle Coordination , 1966, Open Knowledge Institutions.

[2]  S. Chiba,et al.  Dynamic programming algorithm optimization for spoken word recognition , 1978 .

[3]  S. Grillner Control of Locomotion in Bipeds, Tetrapods, and Fish , 1981 .

[4]  P. J. Holmes,et al.  Nonlinear Oscillations, Dynamical Systems, and Bifurcations of Vector Fields , 1983, Applied Mathematical Sciences.

[5]  J. Hollerbach Dynamic Scaling of Manipulator Trajectories , 1983, 1983 American Control Conference.

[6]  T. Flash,et al.  The coordination of arm movements: an experimentally confirmed mathematical model , 1985, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[7]  Daniel E. Koditschek,et al.  Exact robot navigation by means of potential functions: Some topological considerations , 1987, Proceedings. 1987 IEEE International Conference on Robotics and Automation.

[8]  Bernard Friedland,et al.  Control System Design: An Introduction to State-Space Methods , 1987 .

[9]  S. Rossignol,et al.  Neural Control of Rhythmic Movements in Vertebrates , 1988 .

[10]  J. Kelso,et al.  Dynamics governs switching among patterns of coordination in biological movement , 1988 .

[11]  J A Kelso,et al.  Dynamic pattern generation in behavioral and neural systems. , 1988, Science.

[12]  Daniel Bullock,et al.  Chapter 11 Vite and Flete: Neural Modules for Trajectory Formation and Postural Control , 1989 .

[13]  E. A. Jackson,et al.  Perspectives of nonlinear dynamics , 1990 .

[14]  Abhijit S. Pandya,et al.  Dynamic pattern recognition of coordinated biological motion , 1990, Neural Networks.

[15]  Oussama Khatib,et al.  Real-Time Obstacle Avoidance for Manipulators and Mobile Robots , 1985, Autonomous Robot Vehicles.

[16]  Daniel E. Koditschek,et al.  From stable to chaotic juggling: theory, simulation, and experiments , 1990, Proceedings., IEEE International Conference on Robotics and Automation.

[17]  Weiping Li,et al.  Applied Nonlinear Control , 1991 .

[18]  S. Strogatz,et al.  Dynamics of a large system of coupled nonlinear oscillators , 1991 .

[19]  Daniel E. Koditschek,et al.  Exact robot navigation using artificial potential functions , 1992, IEEE Trans. Robotics Autom..

[20]  F. A. Mussa-lvaldi,et al.  Convergent force fields organized in the frog's spinal cord , 1993, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[21]  Stefan Schaal,et al.  Assessing the Quality of Learned Local Models , 1993, NIPS.

[22]  M. Latash Control of human movement , 1993 .

[23]  Steven H. Strogatz,et al.  Nonlinear Dynamics and Chaos , 2024 .

[24]  Daniel E. Koditschek,et al.  Further progress in robot juggling: solvable mirror laws , 1994, Proceedings of the 1994 IEEE International Conference on Robotics and Automation.

[25]  Linda B. Smith,et al.  A Dynamic Systems Approach to the Development of Cognition and Action , 2007, Journal of Cognitive Neuroscience.

[26]  A. Opstal Dynamic Patterns: The Self-Organization of Brain and Behavior , 1995 .

[27]  D. Lewkowicz,et al.  A dynamic systems approach to the development of cognition and action. , 2007, Journal of cognitive neuroscience.

[28]  S. Schaal,et al.  A Kendama Learning Robot Based on Bi-directional Theory , 1996, Neural Networks.

[29]  M. Turvey,et al.  Diffusive, Synaptic, and Synergetic Coupling: An Evaluation Through In-Phase and Antiphase Rhythmic Movements. , 1996, Journal of motor behavior.

[30]  Mitsuo Kawato,et al.  Equilibrium-Point Control Hypothesis Examined by Measured Arm Stiffness During Multijoint Movement , 1996, Science.

[31]  Mitsuo Kawato,et al.  TRAJECTORY FORMATION IN ARM MOVEMENTS: MINIMIZATION PRINCIPLES AND PROCEDURES , 1996 .

[32]  W. Warren,et al.  Resonance Tuning in Rhythmic Arm Movements. , 1996, Journal of motor behavior.

[33]  Bruno Siciliano,et al.  Modelling and Control of Robot Manipulators , 1997, Advanced Textbooks in Control and Signal Processing.

[34]  Mitsuo Kawato,et al.  Human arm stiffness and equilibrium-point trajectory during multi-joint movement , 1997, Biological Cybernetics.

[35]  Gavan Lintern,et al.  Dynamic patterns: The self-organization of brain and behavior , 1997, Complex.

[36]  Ferdinando A. Mussa-Ivaldi,et al.  Nonlinear force fields: a distributed system of control primitives for representing and learning movements , 1997, Proceedings 1997 IEEE International Symposium on Computational Intelligence in Robotics and Automation CIRA'97. 'Towards New Computational Principles for Robotics and Automation'.

[37]  Christopher G. Atkeson,et al.  Constructive Incremental Learning from Only Local Information , 1998, Neural Computation.

[38]  Jean-Jacques E. Slotine,et al.  On Contraction Analysis for Non-linear Systems , 1998, Autom..

[39]  S. Schaal,et al.  Programmable Pattern Generators , 1998 .

[40]  M. Arbib,et al.  Language within our grasp , 1998, Trends in Neurosciences.

[41]  Daniel E. Koditschek,et al.  Sequential Composition of Dynamically Dexterous Robot Behaviors , 1999, Int. J. Robotics Res..

[42]  Ferdinando A Mussa-Ivaldi,et al.  Modular features of motor control and learning , 1999, Current Opinion in Neurobiology.

[43]  Winfried Stefan Lohmiller,et al.  Contraction analysis of nonlinear systems , 1999 .

[44]  John Hallam,et al.  Evolving Swimming Controllers for a Simulated Lamprey with Inspiration from Neurobiology , 1999, Adapt. Behav..

[45]  Perry Y. Li,et al.  Passive velocity field control of mechanical manipulators , 1995, IEEE Trans. Robotics Autom..

[46]  Stefan Schaal,et al.  Is imitation learning the route to humanoid robots? , 1999, Trends in Cognitive Sciences.

[47]  Joshua G. Hale,et al.  Using Humanoid Robots to Study Human Behavior , 2000, IEEE Intell. Syst..

[48]  L. Siciliano Modelling and Control of Robot Manipulators , 2000 .

[49]  Daniel E. Koditschek,et al.  Stability of coupled hybrid oscillators , 2001, Proceedings 2001 ICRA. IEEE International Conference on Robotics and Automation (Cat. No.01CH37164).

[50]  Tamar Flash,et al.  Computational approaches to motor control , 2001, Current Opinion in Neurobiology.

[51]  Cristina P. Santos,et al.  Control of movement time and sequential action through attractor dynamics: A simulation study demonstrating object interception and coordination , 2001 .

[52]  Aude Billard,et al.  Learning human arm movements by imitation: : Evaluation of a biologically inspired connectionist architecture , 2000, Robotics Auton. Syst..

[53]  Auke Jan Ijspeert,et al.  A connectionist central pattern generator for the aquatic and terrestrial gaits of a simulated salamander , 2001, Biological Cybernetics.

[54]  Jun Nakanishi,et al.  Movement imitation with nonlinear dynamical systems in humanoid robots , 2002, Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No.02CH37292).

[55]  Jun Nakanishi,et al.  Learning rhythmic movements by demonstration using nonlinear oscillators , 2002, IEEE/RSJ International Conference on Intelligent Robots and Systems.

[56]  Yoshiyuki Tanaka,et al.  Bio-mimetic trajectory generation of robots via artificial potential field with time base generator , 2002, IEEE Trans. Syst. Man Cybern. Part C.

[57]  Henry Markram,et al.  Real-Time Computing Without Stable States: A New Framework for Neural Computation Based on Perturbations , 2002, Neural Computation.

[58]  Yoshihiko Nakamura,et al.  Polynomial design of the nonlinear dynamics for the brain-like information processing of whole body motion , 2002, Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No.02CH37292).

[59]  Stefan Schaal,et al.  Computational approaches to motor learning by imitation. , 2003, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[60]  Brett R Fajen,et al.  Behavioral dynamics of steering, obstacle avoidance, and route selection. , 2003, Journal of experimental psychology. Human perception and performance.

[61]  Harald Haas,et al.  Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication , 2004, Science.

[62]  Hiroshi Shimizu,et al.  Self-organized control of bipedal locomotion by neural oscillators in unpredictable environment , 1991, Biological Cybernetics.

[63]  Mitsuo Kawato,et al.  A via-point time optimization algorithm for complex sequential trajectory formation , 2004, Neural Networks.

[64]  E. Todorov Optimality principles in sensorimotor control , 2004, Nature Neuroscience.

[65]  Jun Tani,et al.  Motor primitive and sequence self-organization in a hierarchical recurrent neural network , 2004, Neural Networks.

[66]  G. Schöner,et al.  A dynamic theory of coordination of discrete movement , 1990, Biological Cybernetics.

[67]  Jun Morimoto,et al.  Learning from demonstration and adaptation of biped locomotion , 2004, Robotics Auton. Syst..

[68]  S. Swinnen,et al.  Perception—Action Coupling during Bimanual Coordination: The Role of Visual Perception in the Coalition of Constraints That Govern Bimanual Action , 2004, Journal of motor behavior.

[69]  S. Schaal,et al.  Rhythmic arm movement is not discrete , 2004, Nature Neuroscience.

[70]  Martin A. Giese,et al.  Frequency dependence of the action-perception cycle for postural control in a moving visual environment: relative phase dynamics , 1994, Biological Cybernetics.

[71]  Stefan Schaal,et al.  Rapid synchronization and accurate phase-locking of rhythmic motor primitives , 2005, 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[72]  Christine Chevallereau,et al.  Asymptotically Stable Running for a Five-Link, Four-Actuator, Planar Bipedal Robot , 2005, Int. J. Robotics Res..

[73]  Wolfgang Maass,et al.  Movement Generation with Circuits of Spiking Neurons , 2005, Neural Computation.

[74]  A. Ijspeert,et al.  Dynamic hebbian learning in adaptive frequency oscillators , 2006 .

[75]  Jean-Jacques E. Slotine,et al.  Motion Primitives for Robotic Flight Control , 2006, ArXiv.

[76]  A. Scott Encyclopedia of nonlinear science , 2006 .

[77]  Christopher M. Bishop,et al.  Pattern Recognition and Machine Learning (Information Science and Statistics) , 2006 .

[78]  Ludovic Righetti,et al.  Engineering entrainment and adaptation in limit cycle systems , 2006, Biological Cybernetics.

[79]  Radford M. Neal Pattern Recognition and Machine Learning , 2007, Technometrics.

[80]  Stefan Schaal,et al.  Dynamics systems vs. optimal control--a unifying view. , 2007, Progress in brain research.

[81]  M. Doble Green chemistry and engineering , 2007 .

[82]  Auke Jan Ijspeert,et al.  Central pattern generators for locomotion control in animals and robots: A review , 2008, Neural Networks.

[83]  D. McCrea,et al.  Organization of mammalian locomotor rhythm and pattern generation , 2008, Brain Research Reviews.

[84]  Stefan Schaal,et al.  Robot Programming by Demonstration , 2009, Springer Handbook of Robotics.

[85]  Stefan Schaal,et al.  2008 Special Issue: Reinforcement learning of motor skills with policy gradients , 2008 .

[86]  Jan Peters,et al.  Learning motor primitives for robotics , 2009, 2009 IEEE International Conference on Robotics and Automation.

[87]  Stefan Schaal,et al.  Learning and generalization of motor skills by learning from demonstration , 2009, 2009 IEEE International Conference on Robotics and Automation.

[88]  Henk Nijmeijer,et al.  Robot Programming by Demonstration , 2010, SIMPAR.

[89]  Benjamin Schrauwen,et al.  Design of a Central Pattern Generator Using Reservoir Computing for Learning Human Motion , 2009, 2009 Advanced Technologies for Enhanced Quality of Life.

[90]  Stefan Schaal,et al.  Biologically-inspired dynamical systems for movement generation: Automatic real-time goal adaptation and obstacle avoidance , 2009, 2009 IEEE International Conference on Robotics and Automation.

[91]  A. Billard,et al.  Learning the Nonlinear Multivariate Dynamics of Motion of Robotic Manipulators , 2009 .

[92]  Andrej Gams,et al.  On-line learning and modulation of periodic movements with nonlinear dynamical systems , 2009, Auton. Robots.

[93]  Aude Billard,et al.  Imitation learning of globally stable non-linear point-to-point robot motions using nonlinear programming , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[94]  Jun Morimoto,et al.  Task-Specific Generalization of Discrete and Periodic Dynamic Movement Primitives , 2010, IEEE Transactions on Robotics.

[95]  Stefan Schaal,et al.  A Generalized Path Integral Control Approach to Reinforcement Learning , 2010, J. Mach. Learn. Res..

[96]  Aude Billard,et al.  Learning Non-linear Multivariate Dynamics of Motion in Robotic Manipulators , 2011, Int. J. Robotics Res..

[97]  Minija Tamosiunaite,et al.  Joining Movement Sequences: Modified Dynamic Movement Primitives for Robotics Applications Exemplified on Handwriting , 2012, IEEE Transactions on Robotics.

[98]  Ales Ude,et al.  Efficient sensorimotor learning from multiple demonstrations , 2013, Adv. Robotics.

[99]  Benjamin Schrauwen,et al.  Frequency modulation of large oscillatory neural networks , 2014, Biological Cybernetics.

[100]  A. Ijspeert,et al.  A general family of morphed nonlinear phase oscillators with arbitrary limit cycle shape , 2013 .

[101]  Corey B. Hart,et al.  Motor primitives and synergies in the spinal cord and after injury—the current state of play , 2013, Annals of the New York Academy of Sciences.

[102]  S. Schaal,et al.  Robotics and Neuroscience , 2014, Current Biology.

[103]  Klaus Neumann,et al.  Neural learning of vector fields for encoding stable dynamical systems , 2014, Neurocomputing.

[104]  Olivier Sigaud,et al.  Multiple task optimization using dynamical movement primitives for whole-body reactive control , 2014, 2014 IEEE-RAS International Conference on Humanoid Robots.

[105]  L. F. Abbott,et al.  Hierarchical Control Using Networks Trained with Higher-Level Forward Models , 2014, Neural Computation.

[106]  Vítor M. F. Santos,et al.  Using Kinect for Robot Gesture Imitation , 2014 .

[107]  Auke Jan Ijspeert,et al.  Kinematic primitives for walking and trotting gaits of a quadruped robot with compliant legs , 2014, Front. Comput. Neurosci..

[108]  Vítor Matos,et al.  Towards goal-directed biped locomotion: Combining CPGs and motion primitives , 2014, Robotics Auton. Syst..

[109]  Elsa Andrea Kirchner,et al.  Velocity-Based Multiple Change-Point Inference for Unsupervised Segmentation of Human Movement Behavior , 2014, 2014 22nd International Conference on Pattern Recognition.

[110]  Carme Torras,et al.  Dimensionality reduction and motion coordination in learning trajectories with Dynamic Movement Primitives , 2014, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[111]  Ales Ude,et al.  Solving peg-in-hole tasks by human demonstration and exception strategies , 2014 .

[112]  Darwin G. Caldwell,et al.  Learning reactive robot behavior for autonomous valve turning , 2014, 2014 IEEE-RAS International Conference on Humanoid Robots.

[113]  Freek Stulp,et al.  Simultaneous on-line Discovery and Improvement of Robotic Skill options , 2014, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[114]  Pierre-Yves Oudeyer,et al.  Reusing motor commands to learn object interaction , 2014, 4th International Conference on Development and Learning and on Epigenetic Robotics.

[115]  Darwin G. Caldwell,et al.  A task-parameterized probabilistic model with minimal intervention control , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[116]  Xihong Wu,et al.  Visual gesture recognition for human robot interaction using dynamic movement primitives , 2014, 2014 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[117]  W. Wildman,et al.  Theoretical Neuroscience , 2014 .

[118]  Andrej Gams,et al.  Coupling Movement Primitives: Interaction With the Environment and Bimanual Tasks , 2014, IEEE Transactions on Robotics.

[119]  Naveen Kuppuswamy,et al.  Do muscle synergies reduce the dimensionality of behavior? , 2014, Front. Comput. Neurosci..

[120]  Henrik Gordon Petersen,et al.  Technologies for the Fast Set-Up of Automated Assembly Processes , 2014, KI - Künstliche Intelligenz.

[121]  Mehmet Karamanoglu,et al.  Implementing gait pattern control and transition for legged locomotion , 2014 .

[122]  Dong-Soo Kwon,et al.  Passivity mimicking control for a XENMA gait rehabilitation robot with rimless wheel model , 2014, 2014 11th International Conference on Ubiquitous Robots and Ambient Intelligence (URAI).

[123]  Ales Ude,et al.  Speed profile optimization through directed explorative learning , 2014, 2014 IEEE-RAS International Conference on Humanoid Robots.

[124]  Darwin G. Caldwell,et al.  Learning by imitation with the STIFF-FLOP surgical robot: a biomimetic approach inspired by octopus movements , 2014, ROBIO 2014.

[125]  Matteo Saveriano,et al.  Learning motion and impedance behaviors from human demonstrations , 2014, 2014 11th International Conference on Ubiquitous Robots and Ambient Intelligence (URAI).

[126]  Olivier Sigaud,et al.  Learning a repertoire of actions with deep neural networks , 2014, 4th International Conference on Development and Learning and on Epigenetic Robotics.

[127]  Andrej Gams,et al.  Adapting periodic motion primitives to external feedback: Modulating and changing the motion , 2014, 2014 23rd International Conference on Robotics in Alpe-Adria-Danube Region (RAAD).

[128]  Ludovic Marin,et al.  The importance of studying all subgoals at once , 2014, Front. Hum. Neurosci..

[129]  Jochen J. Steil,et al.  Efficient policy search in low-dimensional embedding spaces by generalizing motion primitives with a parameterized skill memory , 2015, Auton. Robots.

[130]  Siddhartha S. Srinivasa,et al.  Movement primitives via optimization , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[131]  P. N. Kugler,et al.  Information, Natural Law, and the Self-Assembly of Rhythmic Movement , 2015 .

[132]  D. Lathrop Nonlinear Dynamics and Chaos: With Applications to Physics, Biology, Chemistry, and Engineering , 2015 .

[133]  Dimitar Dimitrov,et al.  Model Predictive Motion Control based on Generalized Dynamical Movement Primitives , 2015, J. Intell. Robotic Syst..