Free-viewpoint video of human actors

In free-viewpoint video, the viewer can interactively choose his viewpoint in 3-D space to observe the action of a dynamic real-world scene from arbitrary perspectives. The human body and its motion plays a central role in most visual media and its structure can be exploited for robust motion estimation and efficient visualization. This paper describes a system that uses multi-view synchronized video footage of an actor's performance to estimate motion parameters and to interactively re-render the actor's appearance from any viewpoint.The actor's silhouettes are extracted from synchronized video frames via background segmentation and then used to determine a sequence of poses for a 3D human body model. By employing multi-view texturing during rendering, time-dependent changes in the body surface are reproduced in high detail. The motion capture subsystem runs offline, is non-intrusive, yields robust motion parameter estimates, and can cope with a broad range of motion. The rendering subsystem runs at real-time frame rates using ubiquous graphics hardware, yielding a highly naturalistic impression of the actor. The actor can be placed in virtual environments to create composite dynamic scenes. Free-viewpoint video allows the creation of camera fly-throughs or viewing the action interactively from arbitrary perspectives.

[1]  Ramesh Raskar,et al.  Image-based visual hulls , 2000, SIGGRAPH.

[2]  Geovanni Martinez,et al.  3D Motion Estimation of Articulated Objects for Object–Based Analysis–Synthesis Coding (OBASC) , 2002 .

[3]  CurlessBrian,et al.  Articulated body deformation from range scan data , 2002 .

[4]  Pascal Fua,et al.  Local and Global Skeleton Fitting Techniques for Optical Motion Capture , 1998, CAPTECH.

[5]  Larry S. Davis,et al.  3-D model-based tracking of humans in action: a multi-view approach , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[6]  Rangachar Kasturi,et al.  Machine vision , 1995 .

[7]  Jitendra Malik,et al.  Image-based modeling and rendering of architecture with interactive photogrammetry and view-dependent texture mapping , 1998, ISCAS '98. Proceedings of the 1998 IEEE International Symposium on Circuits and Systems (Cat. No.98CH36187).

[8]  Wojciech Matusik,et al.  Polyhedral Visual Hulls for Real-Time Rendering , 2001, Rendering Techniques.

[9]  Zoran Popovic,et al.  Articulated body deformation from range scan data , 2002, SIGGRAPH.

[10]  Yee-Hong Yang,et al.  First Sight: A Human Body Outline Labeling System , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[11]  Larry S. Davis,et al.  A distributed system for real-time volume reconstruction , 2000, Proceedings Fifth IEEE International Workshop on Computer Architectures for Machine Perception.

[12]  Dariu Gavrila,et al.  The Visual Analysis of Human Movement: A Survey , 1999, Comput. Vis. Image Underst..

[13]  Takeo Kanade,et al.  A real time system for robust 3D voxel reconstruction of human motions , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[14]  William H. Press,et al.  Numerical recipes , 1990 .

[15]  Rin-ichiro Taniguchi,et al.  Real-time human motion analysis and IK-based human figure control , 2000, Proceedings Workshop on Human Motion.

[16]  Takeo Kanade,et al.  Spatio-Temporal View Interpolation , 2002, Rendering Techniques.

[17]  Olivier D. Faugeras,et al.  3D articulated models and multi-view tracking with silhouettes , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[18]  Jitendra Malik,et al.  Tracking people with twists and exponential maps , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[19]  Hans-Peter Seidel,et al.  A Silhouette-Based Algorithm for Texture Registration and Stitching , 2001, Graph. Model..

[20]  A. Laurentini,et al.  The Visual Hull Concept for Silhouette-Based Image Understanding , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[21]  Michael Bosse,et al.  Unstructured lumigraph rendering , 2001, SIGGRAPH.

[22]  Kok-Lim Low,et al.  Blending multiple views , 2002, 10th Pacific Conference on Computer Graphics and Applications, 2002. Proceedings..

[23]  Jason P. Luck,et al.  RealTime Markerless Motion Tracking Using Linked Kinematic Chains , 2002, JCIS.

[24]  Kostas Daniilidis,et al.  View-independent scene acquisition for tele-presence , 2000, Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000).

[25]  Pascal Fua,et al.  Tracking and Modeling People in Video Sequences , 2001, Comput. Vis. Image Underst..

[26]  Alberto Menache,et al.  Understanding Motion Capture for Computer Animation and Video Games , 1999 .

[27]  Mohan M. Trivedi,et al.  Articulated body posture estimation from multi-camera voxel data , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[28]  Markus H. Gross,et al.  3D video recorder , 2002, 10th Pacific Conference on Computer Graphics and Applications, 2002. Proceedings..

[29]  Karl Rohr,et al.  Incremental recognition of pedestrians from image sequences , 1993, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[30]  Takeo Kanade,et al.  Constructing virtual worlds using dense stereo , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[31]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[32]  R. Y. Tsai,et al.  An Efficient and Accurate Camera Calibration Technique for 3D Machine Vision , 1986, CVPR 1986.

[33]  Shree K. Nayar,et al.  Reflectance and texture of real-world surfaces , 1999, TOGS.

[34]  Saied Moezzi,et al.  Virtual View Generation for 3D Digital Video , 1997, IEEE Multim..

[35]  Takashi Matsuyama,et al.  Generation, visualization, and editing of 3D video , 2002, Proceedings. First International Symposium on 3D Data Processing Visualization and Transmission.

[36]  Andrea Bottino,et al.  A Silhouette Based Technique for the Reconstruction of Human Movement , 2001, Comput. Vis. Image Underst..

[37]  Nikolaos Grammalidis,et al.  Estimating body animation parameters from depth images using analysis by synthesis , 2001, Proceedings Second International Workshop on Digital and Computational Video.

[38]  Alex Pentland,et al.  Pfinder: Real-Time Tracking of the Human Body , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[39]  Hans-Peter Seidel,et al.  Combining 2d Feature Tracking And Volume Reconstruction For Online Video-Based Human Motion Capture , 2004, Int. J. Image Graph..