Applying Heuristic Evaluation to Human-Robot Interaction Systems

Though attention to evaluating human-robot interfaces has increased in recent years, there are relatively few reports of using evaluation tools during the development of human-robot interaction (HRI) systems to gauge and improve their designs— possibly due to a shortage of suitable evaluation techniques. Heuristic evaluation is a technique suitable for such applications that has become popular in the human-computer interaction (HCI) community. However, it requires usability heuristics applicable to the system environment. This work contributes a set of heuristics appropriate for use with HRI systems, derived from a variety of sources both in and out of the HRI field. Evaluators have successfully used the heuristics on an HRI system, demonstrating their effectiveness against standard measures of heuristic effectiveness.

[1]  Jean Scholtz,et al.  Awareness in human-robot interactions , 2003, SMC'03 Conference Proceedings. 2003 IEEE International Conference on Systems, Man and Cybernetics. Conference Theme - System Security and Assurance (Cat. No.03CH37483).

[2]  Russell H. Taylor,et al.  A Steady-Hand Robotic System for Microsurgical Augmentation , 1999, Int. J. Robotics Res..

[3]  Robin Jeffries,et al.  User interface evaluation in the real world: a comparison of four techniques , 1991, CHI.

[4]  Julie A. Adams,et al.  Critical Considerations for Human-Robot Interface Development , 2002 .

[5]  Alan D. Christiansen,et al.  Command and Control of Robot Teams , 2003 .

[6]  Anind K. Dey,et al.  Heuristic evaluation of ambient displays , 2003, CHI '03.

[7]  Carl Gutwin,et al.  Empirical development of a heuristic evaluation methodology for shared workspace groupware , 2002, CSCW '02.

[8]  Jean Scholtz,et al.  Theory and evaluation of human robot interactions , 2003, 36th Annual Hawaii International Conference on System Sciences, 2003. Proceedings of the.

[9]  Terrence Fong,et al.  Collaboration, Dialogue, and Human-Robot Interaction , 2001 .

[10]  Ronald C. Arkin,et al.  Usability evaluation of high-level user assistance for robot mission specification , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[11]  Jean Scholtz,et al.  Beyond Usability Evaluation: Analysis of Human-Robot Interaction at a Major Robotics Competition , 2004, Hum. Comput. Interact..

[12]  Jakob Nielsen,et al.  Improving a human-computer dialogue , 1990, CACM.

[13]  Michael A. Goodrich,et al.  Seven principles of efficient human robot interaction , 2003, SMC'03 Conference Proceedings. 2003 IEEE International Conference on Systems, Man and Cybernetics. Conference Theme - System Security and Assurance (Cat. No.03CH37483).

[14]  Dan R. Olsen,et al.  Metrics for Evaluating Human-Robot Interactions , 2003 .

[15]  Carl Gutwin,et al.  Adapting the Locales Framework for Heuristic Evaluation of Groupware , 2000, Australas. J. Inf. Syst..

[16]  Arvin Agah,et al.  Human interactions with intelligent systems: research taxonomy , 2000, Comput. Electr. Eng..

[17]  Michael R. M. Jenkin,et al.  A taxonomy for multi-agent robotics , 1996, Auton. Robots.

[18]  Jakob Nielsen,et al.  How to Conduct a Heuristic Evaluation , 2006 .

[19]  Ronald C. Arkin,et al.  Multiagent telerobotics: matching systems to tasks , 1999 .

[20]  Robin R. Murphy,et al.  Human-robot interactions during the robot-assisted urban search and rescue response at the World Trade Center , 2003, IEEE Trans. Syst. Man Cybern. Part B.

[21]  Terrence Fong,et al.  Collaboration, Dialogue, Human-Robot Interaction , 2001, ISRR.

[22]  Illah R. Nourbakhsh,et al.  A survey of socially interactive robots , 2003, Robotics Auton. Syst..

[23]  Jean C. Scholtz,et al.  Evaluation Methods for Human-System Performance of Intelligent Systems , 2002 .

[24]  Salvatore P Schipani,et al.  An Evaluation of Operator Workload, During Partially-Autonomous Vehicle Operations , 2003 .

[25]  T. B. Sheridan.  Eight ultimate challenges of human-robot communication , 1997, Proceedings 6th IEEE International Workshop on Robot and Human Communication. RO-MAN'97 SENDAI.

[26]  Vladimir J. Lumelsky,et al.  Final report for the DARPA/NSF interdisciplinary study on human-robot interaction , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[27]  Jakob Nielsen,et al.  Heuristic evaluation of user interfaces , 1990, CHI '90.

[28]  Illah R. Nourbakhsh,et al.  An Affective Mobile Robot Educator with a Full-Time Job , 1999, Artif. Intell..

[29]  Kazuhiko Kawamura,et al.  Evaluation of an enhanced human-robot interface , 2003, SMC'03 Conference Proceedings. 2003 IEEE International Conference on Systems, Man and Cybernetics. Conference Theme - System Security and Assurance (Cat. No.03CH37483).

[30]  Julie A. Adams,et al.  Interface Evaluation for Mobile Robot Teleoperation , 2003 .

[31]  Cynthia Breazeal,et al.  A Motivational System for Regulating Human-Robot Interaction , 1998, AAAI/IAAI.

[32]  Ronald C. Arkin,et al.  Skills Impact Study for Tactical Mobile Robot Operational Units , 2002 .

[33]  H. Yanco,et al.  Automation as Caregiver: A Survey of Issues and Technologies , 2003 .

[34]  Jakob Nielsen,et al.  Enhancing the explanatory power of usability heuristics , 1994, CHI '94.

[35]  Colin Potts,et al.  Design of Everyday Things , 1988 .

[36]  Michael J. McDonald.  Active Research Topics in Human Machine Interfaces , 2000 .

[37]  Holly A. Yanco,et al.  A Taxonomy for Human-Robot Interaction , 2002 .