An assistive decision-and-control architecture for force-sensitive hand–arm systems driven by human–machine interfaces

Fully autonomous applications of modern robotic systems are still constrained by limitations in sensory data processing, scene interpretation, and automated reasoning. However, their use as assistive devices for people with upper-limb disabilities has become possible with recent advances in “soft robotics”, that is, interaction control, physical human–robot interaction, and reflex planning. In this context, impedance and reflex-based control has generally been understood to be a promising approach to safe interaction robotics. To create semi-autonomous assistive devices, we propose a decision-and-control architecture for hand–arm systems with “soft robotics” capabilities, which can then be used via human–machine interfaces (HMIs). We validated the functionality of our approach within the BrainGate2 clinical trial, in which an individual with tetraplegia used our architecture to control a robotic hand–arm system under neural control via a multi-electrode array implanted in the motor cortex. The neuroscience results of this research have previously been published by Hochberg et al. In this paper we present our assistive decision-and-control architecture and demonstrate how the semi-autonomous assistive behavior can help the user. In our framework the robot is controlled through a multi-priority Cartesian impedance controller and its behavior is extended with collision detection and reflex reaction. Furthermore, virtual workspaces are added to ensure safety. On top of this we employ a decision-and-control architecture that uses sensory information available from the robotic system to evaluate the current state of task execution. Based on a set of available assistive skills, our architecture provides support in object interaction and manipulation and thereby enhances the usability of the robotic system for use with HMIs. The goal of our development is to provide an easy-to-use robotic system for people with physical disabilities and thereby enable them to perform simple tasks of daily living. In an exemplary real-world task, the participant was able to serve herself a beverage autonomously for the first time since her brainstem stroke, which she suffered approximately 14 years prior to this research.

[1]  J. A. Wilson,et al.  Two-dimensional movement control using electrocorticographic signals in humans , 2008, Journal of neural engineering.

[2]  Hong Liu,et al.  Multisensory five-finger dexterous hand: The DLR/HIT Hand II , 2008, 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[3]  Patrick van der Smagt,et al.  Surface EMG in advanced hand prosthetics , 2008, Biological Cybernetics.

[4]  Darius Burschka,et al.  A pilot study in vision-based augmented telemanipulation for remote assembly over high-latency networks , 2013, 2013 IEEE International Conference on Robotics and Automation.

[5]  Justin Bayer,et al.  Continuous robot control using surface electromyography of atrophic muscles , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[6]  Francisco Velasco-Álvarez,et al.  First study towards linear control of an upper-limb neuroprosthesis with an EEG-based Brain-Computer Interface , 2012, 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[7]  Alin Albu-Schäffer,et al.  Modular state-based behavior control for safe human-robot interaction: A lightweight control architecture for a lightweight robot , 2011, 2011 IEEE International Conference on Robotics and Automation.

[8]  Francois Routhier,et al.  Evaluation of the JACO robotic arm: Clinico-economic study for powered wheelchair users with upper-extremity disabilities , 2011, 2011 IEEE International Conference on Rehabilitation Robotics.

[9]  Cuntai Guan,et al.  Temporal classification of multichannel near-infrared spectroscopy signals of motor imagery for developing a brain–computer interface , 2007, NeuroImage.

[10]  Michael A. Arbib,et al.  Neurorobotics: From Vision to Action , 2008, Springer Handbook of Robotics.

[11]  Michael J. Black,et al.  Multi-state decoding of point-and-click control signals from motor cortical activity in a human with tetraplegia , 2007, 2007 3rd International IEEE/EMBS Conference on Neural Engineering.

[12]  Tommy Strandvall,et al.  Eye Tracking in Human-Computer Interaction and Usability Research , 2009, INTERACT.

[13]  Robin C. Ashmore,et al.  An Electrocorticographic Brain Interface in an Individual with Tetraplegia , 2013, PloS one.

[14]  Luca Citi,et al.  Restoring Natural Sensory Feedback in Real-Time Bidirectional Hand Prostheses , 2014, Science Translational Medicine.

[15]  Gert Willem Römer,et al.  Processes for obtaining a Manus (ARM) robot within the Netherlands , 2004 .

[16]  Berthold Bäuml,et al.  When hard realtime matters: Software for complex mechatronic systems , 2008, Robotics Auton. Syst..

[17]  Wolfgang Rosenstiel,et al.  An MEG-based brain–computer interface (BCI) , 2007, NeuroImage.

[18]  Wolfgang Grodd,et al.  Principles of a brain-computer interface (BCI) based on real-time functional magnetic resonance imaging (fMRI) , 2004, IEEE Transactions on Biomedical Engineering.

[19]  Michael J. Black,et al.  Neural control of computer cursor velocity by decoding motor cortical spiking activity in humans with tetraplegia , 2008, Journal of neural engineering.

[20]  Alin Albu-Schäffer,et al.  Towards the Robotic Co-Worker , 2009, ISRR.

[21]  Jacob Rosen,et al.  A myosignal-based powered exoskeleton system , 2001, IEEE Trans. Syst. Man Cybern. Part A.

[22]  Rajesh P. N. Rao,et al.  Real-time functional brain mapping using electrocorticography , 2007, NeuroImage.

[23]  Tim Hornyak Thinking of child's play. , 2006, Scientific American.

[24]  Mehring Carsten,et al.  An Online Brain-Machine Interface Using Decoding Of Movement Direction From The Human Electrocorticogram , 2010 .

[25]  A. Schwartz,et al.  High-performance neuroprosthetic control by an individual with tetraplegia , 2013, The Lancet.

[26]  W. A. Sarnacki,et al.  Electroencephalographic (EEG) control of three-dimensional movement , 2010, Journal of neural engineering.

[27]  T. Kim,et al.  The effect of delayed visual feedback on telerobotic surgery , 2005, Surgical Endoscopy And Other Interventional Techniques.

[28]  Donald H. House,et al.  Measuring gaze depth with an eye tracker during stereoscopic display , 2011, APGV '11.

[29]  Van Nguyen The Synthesis of Stable Force-Closure Grasps , 1986 .

[30]  Alin Albu-Schäffer,et al.  Holistic design and analysis for the human-friendly robotic co-worker , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[31]  S. Petersen,et al.  Characterizing the Hemodynamic Response: Effects of Presentation Rate, Sampling Procedure, and the Possibility of Ordering Brain Activity Based on Relative Timing , 2000, NeuroImage.

[32]  Nicolas Y. Masse,et al.  Advantages of closed-loop calibration in intracortical brain–computer interfaces for people with tetraplegia , 2013, Journal of neural engineering.

[33]  J. Wolpaw,et al.  A P300-based brain–computer interface for people with amyotrophic lateral sclerosis , 2008, Clinical Neurophysiology.

[34]  Panagiotis K. Artemiadis,et al.  EMG-Based Control of a Robot Arm Using Low-Dimensional Embeddings , 2010, IEEE Transactions on Robotics.

[35]  Trent J. Bradberry,et al.  Reconstructing Three-Dimensional Hand Movements from Noninvasive Electroencephalographic Signals , 2010, The Journal of Neuroscience.

[36]  J. Bradshaw,et al.  Mechanisms underlying embodiment, disembodiment and loss of embodiment , 2008, Neuroscience & Biobehavioral Reviews.

[37]  Alessandro De Luca,et al.  Collision detection and reaction: A contribution to safe physical Human-Robot Interaction , 2008, 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[38]  Bernd Freisleben,et al.  HaWCoS: the "hands-free" wheelchair control system , 2002, ASSETS.

[39]  G. Gini,et al.  An EMG-controlled exoskeleton for hand rehabilitation , 2005, 9th International Conference on Rehabilitation Robotics, 2005. ICORR 2005..

[40]  Eric E. Thomson,et al.  Perceiving Invisible Light through a Somatosensory Cortical Prosthesis , 2013, Nature Communications.

[41]  Nitish V. Thakor,et al.  Continuous decoding of finger position from surface EMG signals for the control of powered prostheses , 2008, 2008 30th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[42]  Nicolas Y. Masse,et al.  Reach and grasp by people with tetraplegia using a neurally controlled robotic arm , 2012, Nature.

[43]  Wei Wu,et al.  Bayesian Population Decoding of Motor Cortical Activity Using a Kalman Filter , 2006, Neural Computation.

[44]  Roel Vertegaal A Fitts Law comparison of eye tracking and manual input in the selection of visual targets , 2008, ICMI '08.

[45]  S. Gielen,et al.  The brain–computer interface cycle , 2009, Journal of neural engineering.

[46]  Gabriel Robles-De-La-Torre,et al.  The importance of the sense of touch in virtual and real environments , 2006, IEEE MultiMedia.

[47]  Dima Damen,et al.  Egocentric Real-time Workspace Monitoring using an RGB-D camera , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[48]  Michael J. Black,et al.  Inferring Hand Motion from Multi-Cell Recordings in Motor Cortex using a Kalman Filter , 2002 .

[49]  Maysam Ghovanloo,et al.  Tongue drive: a tongue operated magnetic sensor based wireless assistive technology for people with severe disabilities , 2006, 2006 IEEE International Symposium on Circuits and Systems.

[50]  Jon A. Mukand,et al.  Neuronal ensemble control of prosthetic devices by a human with tetraplegia , 2006, Nature.

[51]  Michael J. Black,et al.  Neural control of cursor trajectory and click by a human with tetraplegia 1000 days after implant of an intracortical microelectrode array , 2011 .

[52]  Xu Cui,et al.  Speeded Near Infrared Spectroscopy (NIRS) Response Detection , 2010, PloS one.

[53]  D.S. Andreasen,et al.  Exoskeleton with EMG based active assistance for rehabilitation , 2005, 9th International Conference on Rehabilitation Robotics, 2005. ICORR 2005..

[54]  Patrick van der Smagt,et al.  EMG-based teleoperation and manipulation with the DLR LWR-III , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[55]  Patrick van der Smagt,et al.  Learning EMG control of a robotic hand: towards active prostheses , 2006, Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006..

[56]  A P Georgopoulos,et al.  On the relations between the direction of two-dimensional arm movements and cell discharge in primate motor cortex , 1982, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[57]  Alin Albu-Schäffer,et al.  The DLR lightweight robot: design and control concepts for robots in human environments , 2007, Ind. Robot.

[58]  A B Schwartz,et al.  Motor cortical representation of speed and direction during reaching. , 1999, Journal of neurophysiology.