Model-Augmented Haptic Telemanipulation: Concept, Retrospective Overview, and Current Use Cases

Certain telerobotic applications, including telerobotics in space, pose particularly demanding challenges to both technology and humans. Traditional bilateral telemanipulation approaches often cannot be used in such applications due to technical and physical limitations such as long and varying delays, packet loss, and limited bandwidth, as well as high reliability, precision, and task duration requirements. In order to close this gap, we research model-augmented haptic telemanipulation (MATM) that uses two kinds of models: a remote model that enables shared autonomous functionality of the teleoperated robot, and a local model that aims to generate assistive augmented haptic feedback for the human operator. Several technological methods that form the backbone of the MATM approach have already been successfully demonstrated in accomplished telerobotic space missions. On this basis, we have applied our approach in more recent research to applications in the fields of orbital robotics, telesurgery, caregiving, and telenavigation. In the course of this work, we have advanced specific aspects of the approach that were of particular importance for each respective application, especially shared autonomy, and haptic augmentation. This overview paper discusses the MATM approach in detail, presents the latest research results of the various technologies encompassed within this approach, provides a retrospective of DLR's telerobotic space missions, demonstrates the broad application potential of MATM based on the aforementioned use cases, and outlines lessons learned and open challenges.

[1]  Klaus Landzettel,et al.  A Universal Task-Level Ground Control and Programming System for Space Robot Applications , 1999 .

[2]  Zoltan-Csaba Marton,et al.  Implicit 3D Orientation Learning for 6D Object Detection from RGB Images , 2018, ECCV.

[3]  Alin Albu-Schäffer,et al.  Safe Interactions and Kinesthetic Feedback in High Performance Earth-To-Moon Teleoperation , 2020, 2020 IEEE Aerospace Conference.

[4]  Mikel Sagardia,et al.  A New Fast and Robust Collision Detection and Force Computation Algorithm Applied to the Physics Engine Bullet: Method, Integration, and Evaluation , 2014, EuroVR.

[5]  Nikhil Gupta,et al.  2D Push Recovery and Balancing of the EVER3 - a Humanoid Robot with Wheel-Base, using Model Predictive Control and Gain Scheduling , 2019, 2019 IEEE-RAS 19th International Conference on Humanoid Robots (Humanoids).

[6]  Jordi Artigas Esclusa,et al.  Shared control for robotic on-orbit servicing , 2016 .

[7]  Thomas Hulin,et al.  Space Factory 4.0 - New processes for the robotic assembly of modular satellites on an in-orbit platform based on „Industrie 4.0” approach , 2018 .

[8]  Klaus Landzettel,et al.  KONTUR-2 MISSION: THE DLR FORCE FEEDBACK JOYSTICK FOR SPACE TELEMANIPULATION FROM THE ISS , 2016 .

[9]  Thomas Hulin,et al.  Model Mediated Teleoperation with a Hand-Arm Exoskeleton in Long Time Delays Using Reinforcement Learning , 2020, 2020 29th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN).

[10]  Jee-Hwan Ryu,et al.  Multilateral control for delayed teleoperation , 2013, 2013 16th International Conference on Advanced Robotics (ICAR).

[11]  Toshiyuki Inagaki,et al.  Adaptive Automation: Sharing and Trading of Control , 2001 .

[12]  Bernhard Rebele,et al.  LRU – Lightweight Rover Unit , 2015 .

[13]  Michael A. Goodrich,et al.  Teleoperation and Beyond for Assistive Humanoid Robots , 2013 .

[14]  Michael Suppa,et al.  Stereo-vision based obstacle mapping for indoor/outdoor SLAM , 2014, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[15]  Cagdas D. Onal,et al.  A Scaled Bilateral Control System for Experimental One-dimensional Teleoperated Nanomanipulation , 2009, Int. J. Robotics Res..

[16]  Florian Schmidt,et al.  Rollin' Justin - Mobile platform with variable base , 2009, 2009 IEEE International Conference on Robotics and Automation.

[17]  Harsimran Singh,et al.  Enhancing the Rate-Hardness of Haptic Interaction: Successive Force Augmentation Approach , 2020, IEEE Transactions on Industrial Electronics.

[18]  Nadine Eberhardt,et al.  Human Factors Applications In Teleoperator Design And Operation , 2016 .

[19]  Eckehard G. Steinbach,et al.  Model-Mediated Teleoperation: Toward Stable and Transparent Teleoperation Systems , 2016, IEEE Access.

[20]  Probal Mitra,et al.  Model-mediated Telemanipulation , 2008, Int. J. Robotics Res..

[21]  Daniel Leidner,et al.  Probabilistic Effect Prediction through Semantic Augmentation and Physical Simulation , 2020, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[22]  Harsimran Singh,et al.  Enhancing the Force Transparency of Time Domain Passivity Approach: Observer-Based Gradient Controller , 2019, 2019 International Conference on Robotics and Automation (ICRA).

[23]  Michael Panzirsch,et al.  Teleoperation for on-orbit servicing missions through the ASTRA geostationary satellite , 2016, 2016 IEEE Aerospace Conference.

[24]  Berthold Färber,et al.  Bi-modal assistance functions and their effect on user perception and movement coordination with telesurgery systems , 2012, 2012 IEEE International Workshop on Haptic Audio Visual Environments and Games (HAVE 2012) Proceedings.

[25]  Jee-Hwan Ryu,et al.  Closing the Force Loop to Enhance Transparency in Time-delayed Teleoperation , 2020, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[26]  Bernhard Weber,et al.  Sensorimotor performance and haptic support in simulated weightlessness , 2020, Experimental Brain Research.

[27]  Sandra Hirche,et al.  Control sharing in human-robot team interaction , 2017, Annu. Rev. Control..

[28]  Ribin Balachandran,et al.  Kontur-3: Human Machine Interfaces for Telenavigation and Manipulation of Robots from ISS , 2020, 2020 IEEE Aerospace Conference.

[29]  Eckehard G. Steinbach,et al.  Passivity-based model updating for Model-mediated Teleoperation , 2015, 2015 IEEE International Conference on Multimedia & Expo Workshops (ICMEW).

[30]  Christian Ott,et al.  Proxy-Based Approach for Position Synchronization of Delayed Robot Coupling Without Sacrificing Performance , 2020, IEEE Robotics and Automation Letters.

[31]  Blake Hannaford,et al.  A design framework for teleoperators with kinesthetic feedback , 1989, IEEE Trans. Robotics Autom..

[32]  Paolo Serafini,et al.  On Theory and Practice of Robots and Manipulators , 1974 .

[34]  Daniel Sebastian Leidner,et al.  Cognitive Reasoning for Compliant Robot Manipulation , 2018, Springer Tracts in Advanced Robotics.

[35]  Daniel Leidner,et al.  Inferring Semantic State Transitions During Telerobotic Manipulation , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[36]  Shahram Payandeh,et al.  On application of virtual fixtures as an aid for telemanipulation and training , 2002, Proceedings 10th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. HAPTICS 2002.

[37]  Louis B. Rosenberg,et al.  Virtual fixtures: Perceptual tools for telerobotic manipulation , 1993, Proceedings of IEEE Virtual Reality Annual International Symposium.

[38]  Michael Panzirsch,et al.  Haptic Augmentation for Teleoperation through Virtual Grasping Points , 2018, IEEE Transactions on Haptics.

[39]  Konstantin Kondak,et al.  Smoother Position-Drift Compensation for Time Domain Passivity Approach Based Teleoperation , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[40]  Patrick Helmer,et al.  A disturbance observer for the sigma.7 haptic device , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[41]  Alin Albu-Schäffer,et al.  The OOS-SIM: An on-ground simulation facility for on-orbit servicing robotic operations , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[42]  Raja Parasuraman,et al.  Humans and Automation: Use, Misuse, Disuse, Abuse , 1997, Hum. Factors.

[43]  Jordi Artigas,et al.  A passive bilateral control scheme for a teleoperator with time-varying communication delay , 2010 .

[44]  Neal Y. Lii,et al.  Deployment of the SOLEX Environment for Analog Space Telerobotics Validation , 2019 .

[45]  Daniel Leidner,et al.  A knowledge-driven shared autonomy human-robot interface for tablet computers , 2014, 2014 IEEE-RAS International Conference on Humanoid Robots.

[46]  Harsimran Singh,et al.  Investigating the Influence of Haptic Feedback in Rover Navigation with Communication Delay , 2020, EuroHaptics.

[47]  Laurel D. Riek,et al.  Healthcare robotics , 2017, Commun. ACM.

[48]  Louis B. Rosenberg,et al.  The Use of Virtual Fixtures as Perceptual Overlays to Enhance Operator Performance in Remote Environments. , 1992 .

[49]  Boeing Phantom,et al.  Voxel-Based 6-DOF Haptic Rendering Improvements , 2006 .

[50]  Klaus Landzettel,et al.  Robotics Component Verification on ISS ROKVISS - Preliminary Results for Telepresence , 2006, 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[51]  M. Furneaux,et al.  Pricing of Information. , 1980 .

[52]  Alin Albu-Schäffer,et al.  The sigma.7 haptic interface for MiroSurge: A new bi-manual surgical console , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[53]  Alin Albu-Schäffer,et al.  DLR MiroSurge: a versatile system for research in endoscopic telesurgery , 2010, International Journal of Computer Assisted Radiology and Surgery.

[54]  Christian Ott,et al.  Adaptive Authority Allocation in Shared Control of Robots Using Bayesian Filters , 2020, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[55]  Robert J. Anderson Teleoperation with Virtual Force Feedback , 1993, ISER.

[56]  Thomas Hulin,et al.  Optimal control for haptic rendering: Fast energy dissipation and minimum overshoot , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[57]  Ève Coste-Manière,et al.  Haptically augmented teleoperation , 2000, Proceedings 2001 ICRA. IEEE International Conference on Robotics and Automation (Cat. No.01CH37164).

[58]  Daniel Leidner,et al.  Knowledge Driven Orbit-to-Ground Teleoperation of a Robot Coworker , 2020, IEEE Robotics and Automation Letters.

[59]  Konstantin Kondak,et al.  The AEROARMS Project: Aerial Robots with Advanced Manipulation Capabilities for Inspection and Maintenance , 2018, IEEE Robotics & Automation Magazine.

[60]  Alin Albu-Schäffer,et al.  A humanoid upper body system for two-handed manipulation , 2007, Proceedings 2007 IEEE International Conference on Robotics and Automation.

[61]  Thomas F. Ewing,et al.  Visually and Haptically Augmented Teleoperation in D&D Tasks Using Virtual Fixtures , 2004 .

[62]  Brian L. Davies,et al.  Active Constraints/Virtual Fixtures: A Survey , 2014, IEEE Transactions on Robotics.

[63]  Freek Stulp,et al.  Teleoperating Robots from the International Space Station: Microgravity Effects on Performance with Force Feedback , 2019, 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[64]  James J. Troy,et al.  Advances in voxel-based 6-DOF haptic rendering , 2005, SIGGRAPH Courses.

[65]  Martin Buss,et al.  Model-Mediated Teleoperation for multi-operator multi-robot systems , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[66]  Michael Goldfarb Dimensional analysis and selective distortion in scaled bilateral telemanipulation , 1998, Proceedings. 1998 IEEE International Conference on Robotics and Automation (Cat. No.98CH36146).

[67]  Daniel Leidner,et al.  Shared Control Templates for Assistive Robotics , 2020, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[68]  Thomas B. Sheridan,et al.  Supervisory control of remote manipulation , 1967, IEEE Spectrum.

[69]  Bernhard Weber,et al.  The Effects of Force Feedback on Surgical Task Performance: A Meta-Analytical Integration , 2014, EuroHaptics.

[70]  Konstantin Kondak,et al.  Whole-Body Teleoperation and Shared Control of Redundant Robots with Applications to Aerial Manipulation , 2021, J. Intell. Robotic Syst..

[71]  Tetsuya Tanioka,et al.  Intelligent humanoid robots expressing artificial humanlike empathy in nursing situations. , 2020, Nursing philosophy : an international journal for healthcare professionals.

[72]  Thomas B. Sheridan,et al.  Telerobotics, Automation, and Human Supervisory Control , 2003 .

[73]  Konstantin Kondak,et al.  Whole-Body Bilateral Teleoperation of a Redundant Aerial Manipulator , 2020, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[74]  Peter Kazanzides,et al.  Augmented reality environment with virtual fixtures for robotic telemanipulation in space , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[75]  Katharina Hertkorn,et al.  Shared Grasping: a Combination of Telepresence and Grasp Planning , 2016 .

[76]  Angelika Peer,et al.  Enhancing the Command-Following Bandwidth for Transparent Bilateral Teleoperation , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[77]  Harsimran Singh,et al.  Extended Predictive Model-Mediated Teleoperation of Mobile Robots through Multilateral Control , 2018, 2018 IEEE Intelligent Vehicles Symposium (IV).

[78]  Alin Albu-Schäffer,et al.  KONTUR-2: Force-feedback teleoperation from the international space station , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[79]  Terrence Fong,et al.  A Safeguarded Teleoperation Controller , 2001 .

[80]  Craig A. Knoblock,et al.  PDDL-the planning domain definition language , 1998 .

[81]  Blake Hannaford,et al.  Time-domain passivity control of haptic interfaces , 2001, IEEE Trans. Robotics Autom..

[82]  Danica Kragic,et al.  Adaptive Virtual Fixtures for Machine-Assisted Teleoperation Tasks , 2005, Proceedings of the 2005 IEEE International Conference on Robotics and Automation.

[83]  Lio-A Personal Robot Assistant for Human-Robot Interaction and Care Applications , 2020, Ieee Robotics and Automation Letters.

[84]  Dale A. Lawrence Stability and transparency in bilateral teleoperation , 1993, IEEE Trans. Robotics Autom..

[85]  Christian Ott,et al.  Hierarchical Impedance-Based Tracking Control of Kinematically Redundant Robots , 2020, IEEE Transactions on Robotics.

[86]  William R. Ferrell,et al.  Remote manipulation with transmission delay. , 1965 .

[87]  Christian Ott,et al.  The 6-DoF Implementation of the Energy-Reflection Based Time Domain Passivity Approach With Preservation of Physical Coupling Behavior , 2020, IEEE Robotics and Automation Letters.

[88]  Mikel Sagardia Erasun Virtual Manipulations with Force Feedback in Complex Interaction Scenarios , 2019 .

[89]  Alin Albu-Schäffer,et al.  Direct force reflecting teleoperation with a flexible joint robot , 2012, 2012 IEEE International Conference on Robotics and Automation.

[90]  Andreas Tobergte,et al.  Human Performance and Workload Evaluation of Input Modalities for Telesurgery , 2013 .

[91]  Alin Albu-Schäffer,et al.  Haptic intention augmentation for cooperative teleoperation , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[92]  Konstantin Kondak,et al.  Development of SAM: cable-Suspended Aerial Manipulator* , 2019, 2019 International Conference on Robotics and Automation (ICRA).

[93]  Thomas Hulin,et al.  An Ecosystem for Heterogeneous Robotic Assistants in Caregiving: Core Functionalities and Use Cases , 2021, IEEE Robotics & Automation Magazine.

[94]  G. Hirzinger,et al.  [Robotic assistance systems for surgery : Current developments and focus of research]. , 2020, Der Chirurg; Zeitschrift fur alle Gebiete der operativen Medizen.

[95]  Gerd Hirzinger,et al.  Sensor-based space robotics-ROTEX and its telerobotic features , 1993, IEEE Trans. Robotics Autom..