Sign language for telemanipulation

Literal teleoperation doesn't work very well. Limited bandwidth, long latencies, non- anthropomorphic mappings all make the effort of teleoperation tedious at best and ineffective at worst. Instead, users of teleoperated and semi-autonomous systems want their robots to `just do it for them,' without sacrificing the operator's intent. Our goal is to maximize human strategic control in teleoperator assisted robotics. In our teleassisted regime, the human operator provides high-level contexts for low-level autonomous robot behaviors. The operator wears an EXOS hand master to communicate via a natural sign language, such as pointing to objects and adopting a grasp preshape. Each sign indicates intention: e.g., reaching or grasping; and, where applicable, a spatial context: e.g., the pointing axis or preshape frame. The robot, a Utah/MIT hand on a Puma arm, acts under local servo control within the proscribed contexts. This paper extends earlier work [Pook & Ballard 1994a] by adding remote visual sensors to the teleassistance repertoire. To view the robot site, the operator wears a virtual research helmet that is coupled to binocular cameras mounted on a second Puma 760. The combined hand-head sensors allows teleassistance to be performed remotely. The example task is to open a door. We also demonstrate the flexibility of the teleassistance model by bootstrapping a `pick and place' task from the door opening task.