A Survey of Augmented Reality

This paper surveys the field of augmented reality (AR), in which 3D virtual objects are integrated into a 3D real environment in real time. It describes the medical, manufacturing, visualization, path planning, entertainment, and military applications that have been explored. This paper describes the characteristics of augmented reality systems, including a detailed discussion of the tradeoffs between optical and video blending approaches. Registration and sensing errors are two of the biggest problems in building effective augmented reality systems, so this paper summarizes current efforts to overcome these problems. Future directions and areas requiring further research are discussed. This survey provides a starting point for anyone interested in researching or using augmented reality.

[1]  Henry Fuchs,et al.  Comparison of optical and video see-through, head-mounted displays , 1995, Other Conferences.

[2]  Steven K. Feiner,et al.  Knowledge-based augmented reality , 1993, CACM.

[3]  Takeo Kanade,et al.  Vision-Based Object Registration for Real-Time Image Overlay , 1995, CVRMed.

[4]  Ming Ouhyoung,et al.  A 3D tracking experiment on latency and its compensation methods in virtual environments , 1995, UIST '95.

[5]  Henry Sowizral,et al.  Tracking position and orientation in a large volume , 1993, Proceedings of IEEE Virtual Reality Annual International Symposium.

[6]  Frank Biocca,et al.  A Survey of Position Trackers , 1992, Presence: Teleoperators & Virtual Environments.

[7]  Randy F. Pausch,et al.  A Literature Survey for Virtual Environments: Military Flight Simulator Visual Systems and Simulator Sickness , 1992, Presence: Teleoperators & Virtual Environments.

[8]  Dave Sims,et al.  New realities in aircraft design and manufacture , 1994, IEEE Computer Graphics and Applications.

[9]  Alain Tanguy,et al.  Virtual Reality as an Operative Tool During Scoliosis Surgery , 1995, CVRMed.

[10]  Michael Gervautz,et al.  CCD‐Camera Based Optical Beacon Tracking for Virtual and Augmented Reality , 1996, Comput. Graph. Forum.

[11]  Frank J. Ferrin.  Survey of helmet tracking technologies , 1991, Medical Imaging.

[12]  Steve T. Bryson.  Measurement and calibration of static distortion of position data from 3D trackers , 1992, Electronic Imaging.

[13]  Steven K. Feiner,et al.  A touring machine: Prototyping 3D mobile augmented reality systems for exploring the urban environment , 1997, Digest of Papers. First International Symposium on Wearable Computers.

[14]  Michael J. Griffin,et al.  Compensating Lags in Head-Coupled Displays Using Head Position Prediction and Image Deflection , 1992 .

[15]  R. Sharma,et al.  Role of computer vision in augmented virtual reality , 1995, Electronic Imaging.

[16]  Susumu Tachi,et al.  Experimental Study on Remote Manipulation Using Virtual Reality , 1993, Presence: Teleoperators & Virtual Environments.

[17]  Ronald Azuma,et al.  A demonstrated optical tracker with scalable work area for head-mounted display systems , 1992, I3D '92.

[18]  Thomas A. DeFanti,et al.  Ultrasonic calibration of a magnetic tracker in a virtual reality space , 1995, Proceedings Virtual Reality Annual International Symposium '95.

[19]  Stephen R. Ellis,et al.  Judged Distance to Virtual Objects in the near Visual Field , 1995 .

[20]  T. P. Caudell,et al.  Augmented reality: an application of heads-up display technology to manual manufacturing processes , 1992, Proceedings of the Twenty-Fifth Hawaii International Conference on System Sciences.

[21]  Ronald Azuma,et al.  Predictive tracking for augmented reality , 1995 .

[22]  Lance Williams,et al.  View Interpolation for Image Synthesis , 1993, SIGGRAPH.

[23]  Mary C. Whitton,et al.  Technologies for augmented reality systems: realizing ultrasound-guided needle biopsies , 1996, SIGGRAPH.

[24]  Fumio Kishino,et al.  Augmented reality: a class of displays on the reality-virtuality continuum , 1995, Other Conferences.

[25]  Shumin Zhai,et al.  ARGOS: a display system for augmenting reality , 1993, CHI '93.

[26]  V. Rich.  Personal communication , 1989, Nature.

[27]  David E. Breen,et al.  Interactive Occlusion and Automatic Object Placement for Augmented Reality , 1996, Comput. Graph. Forum.

[28]  Shumin Zhai,et al.  Applications of augmented reality for human-robot communication , 1993, Proceedings of 1993 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS '93).

[29]  Nicholas Ayache,et al.  A New Framework for Fusing Stereo Images with Volumetric Medical Images , 1995, CVRMed.

[30]  Lance Williams,et al.  View interpolation for image synthesis , 1998 .

[31]  S. Ellis,et al.  Distance Perception of Stereoscopically Presented Virtual Objects Optically Superimposed on Physical Objects by a Head-Mounted See-Through Display , 1994 .

[32]  Frank Biocca,et al.  Quantification of adaptation to virtual-eye location in see-thru head-mounted displays , 1995, Proceedings Virtual Reality Annual International Symposium '95.

[33]  Steve Mann,et al.  Wearable Computing: A First Step Toward Personal Imaging , 1997, Computer.

[34]  Bernard D. Adelstein,et al.  Factors influencing operator interaction with virtual objects viewed via head-mounted see-through displays: viewing conditions and rendering latency , 1997, Proceedings of IEEE 1997 Annual International Symposium on Virtual Reality.

[35]  Gregory B. Newby,et al.  Virtual reality: Scientific and technological challenges , 1996 .

[36]  Mark A. Livingston,et al.  Superior augmented reality registration by integrating landmark tracking and magnetic tracking , 1996, SIGGRAPH.

[37]  T Kanade,et al.  Techniques for fast and accurate intrasurgical registration. , 1995, Journal of image guided surgery.

[38]  Roberta L. Klatzky,et al.  Personal guidance system for the visually impaired , 1994, ASSETS.

[39]  Ivan E. Sutherland,et al.  A head-mounted three dimensional display , 1968, AFIPS Fall Joint Computing Conference.

[40]  David E. Breen,et al.  Annotating Real-World Objects Using Augmented Reality , 1995, Computer Graphics.

[41]  Henry Sowizral,et al.  Fusion of absolute and incremental position and orientation sensors , 1995, Other Conferences.

[42]  David E. Breen,et al.  Calibration Requirements and Procedures for a Monitor-Based Augmented Reality System , 1995, IEEE Trans. Vis. Comput. Graph..

[43]  W. Eric L. Grimson,et al.  Evaluating and Validating an Automated Registration System for Enhanced Reality Visualization in Surgery , 1995, CVRMed.

[44]  Won S. Kim.  Advanced Teleoperation, Graphics Aids, and Application to Time Delay Environments , 1993 .

[45]  Henry Sowizral,et al.  Note on dynamics of human head motions and on predictive filtering of head-set orientations , 1995, Other Conferences.

[46]  James H. Aylor,et al.  Computer for the 21st Century , 1999, Computer.

[47]  Jun Rekimoto,et al.  NaviCam:A Magnifying Glass Approach to Augmented Reality , 1997, Presence: Teleoperators & Virtual Environments.

[48]  Leonard McMillan,et al.  Head-tracked stereoscopic display using image warping , 1995, Electronic Imaging.

[49]  Jun Rekimoto.  A Magnifying Glass Approach to Augmented Reality Systems , 1997, Presence Teleoperators Virtual Environ..

[50]  Ulrich Neumann,et al.  A self-tracking augmented reality system , 1996, VRST.

[51]  Ronald Azuma,et al.  Tracking requirements for augmented reality , 1993, CACM.

[52]  P. Milgram,et al.  A Taxonomy of Mixed Reality Visual Displays , 1994 .

[53]  Kiriakos N. Kutulakos,et al.  Affine object representations for calibration-free augmented reality , 1996, Proceedings of the IEEE 1996 Virtual Reality Annual International Symposium.

[54]  Ronald Azuma,et al.  A frequency-domain analysis of head-motion prediction , 1995, SIGGRAPH.

[55]  Matthias M. Wloka,et al.  Resolving occlusion in augmented reality , 1995, I3D '95.

[56]  Daniel G. Aliaga.  Virtual objects in the real world , 1997, CACM.

[57]  Susumu Tachi,et al.  Compensation of time lag between actual and virtual spaces by multi-sensor integration , 1994, Proceedings of 1994 IEEE International Conference on MFI '94. Multisensor Fusion and Integration for Intelligent Systems.

[58]  Wendy E. Mackay,et al.  Augmenting reality: adding computational dimensions to paper , 1993, CACM.

[59]  Mark R. Mine.  Characterization of End-to-End Delays in Head-Mounted Display Systems , 1993 .

[60]  Khoi Nguyen,et al.  Computer-vision-based registration techniques for augmented reality , 1996, Other Conferences.

[61]  R. Welch.  Perceptual Modification: Adapting to Altered Sensory Environments , 1978 .

[62]  Steven K. Feiner,et al.  Architectural Anatomy , 1995, Presence: Teleoperators & Virtual Environments.

[63]  Daniel G. Aliaga.  Virtual and real object collisions in a merged environment , 1994 .

[64]  Ryutarou Ohbuchi,et al.  Merging virtual objects with the real world: seeing ultrasound imagery within the patient , 1992, SIGGRAPH.

[65]  Ronald Pose,et al.  Priority rendering with a virtual reality address recalculation pipeline , 1994, SIGGRAPH.

[66]  Frederick P. Brooks,et al.  The computer scientist as toolsmith II , 1996, CACM.

[67]  David E. Breen,et al.  Object Calibration for Augmented Reality , 1995, Comput. Graph. Forum.

[68]  Takeo Kanade,et al.  Vision-Based Object Registration for Real-Time Image Overlay , 1995, CVRMed.

[69]  Alain Fournier.  Illumination Problems in Computer Augmented Reality , 1994 .

[70]  Anil K. Jain.  Fundamentals of Digital Image Processing , 2018, Control of Color Imaging Systems.

[71]  Thomas P. Caudell,et al.  Calibration of head-mounted displays for augmented reality applications , 1993, Proceedings of IEEE Virtual Reality Annual International Symposium.

[72]  Paul Milgram,et al.  Positioning accuracy of a virtual stereographic pointer in a real stereoscopic video world , 1991, Electronic Imaging.

[73]  Ronald Azuma,et al.  The Challenge of Making Augmented Reality Work Outdoors , 1999 .

[74]  Susan G. Mair.  Preliminary Report on SIGGRAPH in the 21st Century: Rediscovering Our Fire , 1994 .

[75]  Michael Gleicher,et al.  Through-the-lens camera control , 1992, SIGGRAPH.

[76]  Woodrow Barfield,et al.  Augmented-reality displays , 1995 .

[77]  Steven K. Feiner,et al.  Computer Graphics - Principles and Practice, 3rd Edition , 1990 .

[78]  George W. Fitzmaurice,et al.  Situated information spaces and spatially aware palmtop computers , 1993, CACM.

[79]  .. McGlone,et al.  Vision-Based Object Registration for Real-Time Image Overlay , 1995 .

[80]  Hiroshi Ishii,et al.  Tangible bits: towards seamless interfaces between people, bits and atoms , 1997, CHI.

[81]  Warren Robinett,et al.  A Computational Model for the Stereoscopic Optics of a Head-Mounted Display , 1991, Presence: Teleoperators & Virtual Environments.

[82]  Jannick P. Rolland,et al.  Video see-through design for merging of real and virtual environments , 1993, Proceedings of IEEE Virtual Reality Annual International Symposium.

[83]  William E. Lorensen,et al.  Enhancing reality in the operating room , 1993, Proceedings Visualization '93.

[84]  Jerry Bowskill,et al.  Extending the capabilities of the human visual system: an introduction to enhanced reality , 1995, COMG.

[85]  Eric Foxlin,et al.  Inertial head-tracker sensor fusion by a complementary separate-bias Kalman filter , 1996, Proceedings of the IEEE 1996 Virtual Reality Annual International Symposium.

[86]  W. Eric L. Grimson,et al.  An automatic registration method for frameless stereotaxy, image guided surgery, and enhanced reality visualization , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[87]  Douglas E. Holmgren.  Design and Construction of a 30-Degree See-Through Head-Mounted Display , 1992 .

[88]  Warren Robinson,et al.  Synthetic Experience: A Proposed Taxonomy , 1992, Presence Teleoperators Virtual Environ..

[89]  Woodrow Barfield,et al.  Virtual environments and advanced interface design , 1995 .

[90]  Katashi Nagao,et al.  The world through the computer: computer augmented interaction with real world environments , 1995, UIST '95.

[91]  Grigore C. Burdea,et al.  A Novel Virtual Reality Tool for Teaching Dynamic 3D Anatomy , 1995, CVRMed.

[92]  Won S. Kim,et al.  Virtual Reality Calibration and Preview/Predictive Displays for Telerobotics , 1996, Presence: Teleoperators & Virtual Environments.

[93]  Steven K. Feiner,et al.  Windows on the world: 2D windows for 3D augmented reality , 1993, UIST '93.

[94]  Terry S. Yoo,et al.  Instant Hole: Windows Onto Reality , 1993 .

[95]  J. P. Mellor.  Realtime Camera Calibration for Enhanced Reality Visualization , 1995, CVRMed.

[96]  Hong Chen,et al.  Observing a volume rendered fetus within a pregnant patient , 1994, Proceedings Visualization '94.

[97]  Pattie Maes,et al.  Artificial life meets entertainment: lifelike autonomous agents , 1995, CACM.

[98]  Fumio Kishino,et al.  Effects of fuzziness in perception of stereoscopically presented virtual object location , 1995, Other Conferences.

[99]  Thomas Ertl,et al.  Computer Graphics - Principles and Practice, 3rd Edition , 2014 .

[100]  M. Carter.  Computer graphics: Principles and practice , 1997 .

[101]  Gregory K. Tharp,et al.  Virtual window telepresence system for telerobotic inspection , 1995, Other Conferences.

[102]  Karel Zikan,et al.  Videometric head tracker for augmented reality applications , 1995, Other Conferences.

[103]  P. M. Murray,et al.  Hardware Improvements To The Helmet Mounted Projector On The Visual Display Research Tool (VDRT) At The Naval Training Systems Center , 1989, Defense, Security, and Sensing.

[104]  Roger Y. Tsai,et al.  Techniques for Calibration of the Scale Factor and Image Center for High Accuracy 3-D Machine Vision Metrology , 1988, IEEE Trans. Pattern Anal. Mach. Intell..

[105]  J. P. Mellor,et al.  Enhanced Reality Visualization in a Surgical Environment , 1995 .

[106]  Ulrich Neumann,et al.  Dynamic registration correction in video-based augmented reality systems , 1995, IEEE Computer Graphics and Applications.

[107]  David J. Hawkes,et al.  Neurosurgical Guidance Using the Stereo Microscope , 1995, CVRMed.

[108]  Siu-Leong Iu,et al.  Registering perspective contours with 3-D objects without correspondence, using orthogonal polynomials , 1996, Proceedings of the IEEE 1996 Virtual Reality Annual International Symposium.

[109]  David E. Breen,et al.  Distributed Augmented Reality for Collaborative Design Applications , 1995, Comput. Graph. Forum.

[110]  Benjamin Watson,et al.  Using texture maps to correct for optical distortion in head-mounted displays , 1995, Proceedings Virtual Reality Annual International Symposium '95.

[111]  Frank Biocca,et al.  Virtual Eyes Can Rearrange Your Body: Adaptation to Visual Displacement in See-Through, Head-Mounted Displays , 1998, Presence.

[112]  Warren Robinett,et al.  A Computational Model for the Stereoscopic Optics of a Head-Mounted Display , 1993, Virtual Reality Systems.

[113]  Jannick P. Rolland,et al.  A Method of Computational Correction for Optical Distortioin in Head-Mounted Displays , 1993 .

[114]  Mike Bajura.  Camera Calibration for Video See-Through Head-Mounted Display , 1993 .

[115]  Joseph M. Rosen,et al.  EVOLUTION OF VIRTUAL REALITY : FROM PLANNING TO PERFORMING SURGERY , 1996 .

[116]  David E. Breen,et al.  Calibration Requirements and Procedures for Augmented Reality , 1997 .

[117]  Stephen R. Ellis,et al.  The Relationship of Binocular Convergence and Errors in Judged Distance to Virtual Objects , 1995 .

[118]  Pierre David Wellner,et al.  Interacting with paper on the DigitalDesk , 1993, CACM.

[119]  Matthias M. Wloka.  Lag in Multiprocessor Virtual Reality , 1995, Presence: Teleoperators & Virtual Environments.

[120]  Ronald Azuma,et al.  Improving static and dynamic registration in an optical see-through HMD , 1994, SIGGRAPH.

[121]  Greg Welch,et al.  Spatially augmented reality , 1999 .

[122]  Michael Deering,et al.  High resolution virtual reality , 1992, SIGGRAPH.

[123]  Susumu Tachi,et al.  Methods to Calibrate Projection Transformation Parameters for See-Through Head-Mounted Displays , 1996, Presence: Teleoperators & Virtual Environments.