Three-dimensional motion computation and object segmentation in a long sequence of stereo frames

We address the problem of computing the three-dimensional motions of objects in a long sequence of stereo frames. Our approach is bottom-up and consists of two levels. The first level deals with the tracking of 3D tokens from frame to frame and the estimation of their kinematics. The processing is completely parallel for each token. The second level groups tokens into objects based on their kinematic parameters, controls the processing at the low level to cope with problems such as occlusion, disappearance, and appearance of tokens, and provides information to other components of the system. We have implemented this approach using 3D line segments obtained from stereo as the tokens. We use classical kinematics and derive closed-form solutions for some special, but useful, cases of motions. The motion computation problem is then formulated as a tracking problem in order to apply the extended Kalman filter. The tracking is performed in a prediction-matching-update loop in which multiple matches can be handled. Tokens are labeled by a number called its support of existence which measures their adequation to the measurements. If this number goes beyond a threshold, the token disappears. The individual line segments can be grouped into rigid objects according to the similarity of their kinematic parameters. Experiments using synthetic and real data have been carried out and the results found to be quite good.

[1]  W. Eric L. Grimson,et al.  Computational Experiments with a Feature Based Stereo Algorithm , 1985, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  Thomas S. Huang,et al.  A linear algorithm for motion estimation using straight line correspondences , 1988, Comput. Vis. Graph. Image Process..

[3]  Andrea J. van Doorn,et al.  Invariant Properties of the Motion Parallax Field due to the Movement of Rigid Bodies Relative to an Observer , 1975 .

[4]  Olivier Faugeras,et al.  Maintaining representations of the environment of a mobile robot , 1988, IEEE Trans. Robotics Autom..

[5]  Xinhua Zhuang,et al.  Two view motion analysis under a small perturbation , 1985, ICASSP '85. IEEE International Conference on Acoustics, Speech, and Signal Processing.

[6]  Olivier D. Faugeras,et al.  Representing Stereo Data with the Delaunay Triangulation , 1990, Artif. Intell..

[7]  H. C. Longuet-Higgins,et al.  A computer algorithm for reconstructing a scene from two projections , 1981, Nature.

[8]  Olivier D. Faugeras,et al.  On the motion of 3D curves and its relationship to optical flow , 1990, ECCV.

[9]  H. K. Nishihara,et al.  PRISM: A Practical Mealtime Imaging Stereo Matcher , 1984, Other Conferences.

[10]  Hugh Durrant-Whyte,et al.  Integration, coordination, and control of multi-sensor robot systems , 1987 .

[11]  David G. Kirkpatrick,et al.  A Linear Algorithm for Determining the Separation of Convex Polyhedra , 1985, J. Algorithms.

[12]  Narendra Ahuja,et al.  3-D Motion Estimation, Understanding, and Prediction from Noisy Image Sequences , 1987, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Olivier D. Faugeras,et al.  Determining motion from 3D line segment matches: a comparative study , 1991, Image Vis. Comput..

[14]  J.-P. Gambotto Tracking points and line segments in image sequences , 1989, [1989] Proceedings. Workshop on Visual Motion.

[15]  Ishwar K. Sethi,et al.  Finding Trajectories of Feature Points in a Monocular Image Sequence , 1987, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  N. Ayache Construction et fusion de représentations visuelles (3D) : applications à la robotique mobile , 1988 .

[17]  Thomas S. Huang,et al.  DETERMINING 3-D MOTION/STRUCTURE OF A RIGID BODY OVER 3 FRAMES USING STRAIGHT LINE CORRESPONDENCES. , 1983 .

[18]  E. D. Dickmanns,et al.  4 D-dynamic scene analysis with integral spatio-temporal models , 1988 .

[19]  R. W. Smith Problems in Optics , 1975 .

[20]  O. Faugeras,et al.  Motion from point matches: Multiplicity of solutions , 1989, [1989] Proceedings. Workshop on Visual Motion.

[21]  John K. Tsotsos,et al.  Applying temporal constraints to the dynamic stereo problem , 1986, Comput. Vis. Graph. Image Process..

[22]  Tomaso Poggio,et al.  Cooperative computation of stereo disparity , 1988 .

[23]  Thomas S. Huang,et al.  Image Sequence Analysis: Motion Estimation , 1981 .

[24]  T. Poggio,et al.  A computational theory of human stereo vision , 1979, Proceedings of the Royal Society of London. Series B. Biological Sciences.

[25]  W. Eric L. Grimson,et al.  From images to surfaces , 1981 .

[26]  Volker Graefe,et al.  Applications of dynamic monocular machine vision , 1988, Machine Vision and Applications.

[27]  V. S. Hwang,et al.  Tracking feature points in time-varying images using an opportunistic selection approach , 1989, Pattern Recognit..

[28]  Ellen C. Hildreth,et al.  Measurement of Visual Motion , 1984 .

[29]  Thomas S. Huang,et al.  Estimating three-dimensional motion parameters of a rigid planar patch , 1981 .

[30]  J. Koenderink Optic flow , 1986, Vision Research.

[31]  Y. Bar-Shalom Tracking and data association , 1988 .

[32]  Thomas S. Huang,et al.  Image sequence analysis , 1981 .

[33]  Jake K. Aggarwal,et al.  Structure from Motion of Rigid and Jointed Objects , 1981, Artif. Intell..

[34]  Rachid Deriche,et al.  Depth and Motion Analysis: The Machine Being Developed within Esprit Project 940 (Invited) , 1988, MVA.

[35]  Olivier D. Faugeras,et al.  Tracking and Motion Estimation in a Sequence of Stereo Frames , 1990, ECAI.

[36]  Michael Ian Shamos,et al.  Computational geometry: an introduction , 1985 .

[37]  Donald B. Gennery,et al.  Tracking Known Three-Dimensional Objects , 1982, AAAI.

[38]  G. Gordon,et al.  On the tracking of featureless objects with occlusion , 1989, [1989] Proceedings. Workshop on Visual Motion.

[39]  Takeo Kanade,et al.  Stereo by Intra- and Inter-Scanline Search Using Dynamic Programming , 1985, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[40]  Olivier Faugeras,et al.  Building a 3D world model with a mobile robot: 3D line segment representation and integration , 1990, [1990] Proceedings. 10th International Conference on Pattern Recognition.

[41]  Yoshifumi Kitamura,et al.  Three-dimensional data acquisition by trinocular vision , 1989, Adv. Robotics.

[42]  Olivier D. Faugeras,et al.  Analysis Of A Sequence Of Stereo Scenes Containing Multiple Moving Objects Using Rigidity Constraints , 1988, [1988 Proceedings] Second International Conference on Computer Vision.

[43]  Olivier D. Faugeras,et al.  Building, Registrating, and Fusing Noisy Visual Maps , 1988, Int. J. Robotics Res..

[44]  Jake K. Aggarwal,et al.  Determining object motion in a sequence of stereo images , 1987, IEEE Journal on Robotics and Automation.

[45]  Zhengyou Zhang,et al.  Estimation of Displacements from Two 3-D Frames Obtained From Stereo , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[46]  Olivier D. Faugeras,et al.  Determining motion from 3D line segment matches: a comparative study , 1990, BMVC.

[47]  Rachid Deriche,et al.  Tracking line segments , 1990, Image Vis. Comput..

[48]  G.-S. Young,et al.  3-D motion estimation using a sequence of noisy stereo images , 1988, Proceedings CVPR '88: The Computer Society Conference on Computer Vision and Pattern Recognition.

[49]  R. Hetherington The Perception of the Visual World , 1952 .

[50]  Hans-Hellmut Nagel,et al.  Image Sequences - Ten (Octal) Years - from phenomenology towards a Theoretical Foundation , 1988, Int. J. Pattern Recognit. Artif. Intell..

[51]  Jake K. Aggarwal,et al.  Computer Tracking of Objects Moving in Space , 1979, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[52]  Berthold K. P. Horn,et al.  Determining Optical Flow , 1981, Other Conferences.

[53]  Thomas O. Binford,et al.  Depth from Edge and Intensity Based Stereo , 1981, IJCAI.

[54]  D Marr,et al.  Cooperative computation of stereo disparity. , 1976, Science.

[55]  S. Ullman The Interpretation of Visual Motion , 1979 .

[56]  Kenneth S. Roberts,et al.  A new representation for a line , 1988, Proceedings CVPR '88: The Computer Society Conference on Computer Vision and Pattern Recognition.

[57]  Hans-Hellmut Nagel,et al.  Displacement vectors derived from second-order intensity variations in image sequences , 1983, Comput. Vis. Graph. Image Process..

[58]  Olivier Faugeras,et al.  Three-Dimensional Computer Vision , 1993 .

[59]  Jake K. Aggarwal,et al.  Analysis of a sequence of images using point and line correspondences , 1987, Proceedings. 1987 IEEE International Conference on Robotics and Automation.

[60]  James L. Crowley,et al.  Measuring Image Flow By Tracking Edge-lines , 1988, [1988 Proceedings] Second International Conference on Computer Vision.

[61]  O. Faugeras,et al.  A preliminary investigation of the problem of determining ego- and object motions from stereo , 1988, [1988 Proceedings] 9th International Conference on Pattern Recognition.

[62]  Bruce Lowerre,et al.  The Harpy speech understanding system , 1990 .

[63]  R. Chellappa,et al.  Experiments and uniqueness results on object structure and kinematics from a sequence of monocular images , 1989, [1989] Proceedings. Workshop on Visual Motion.

[64]  D Marr,et al.  A computational theory of human stereo vision. , 1979, Proceedings of the Royal Society of London. Series B, Biological sciences.

[65]  Thomas S. Huang,et al.  Uniqueness and Estimation of Three-Dimensional Motion Parameters of Rigid Objects with Curved Surfaces , 1984, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[66]  Volker Graefe,et al.  Dynamic monocular machine vision , 1988, Machine Vision and Applications.

[67]  Thomas S. Huang,et al.  Estimation of rigid body motion using straight line correspondences , 1986, Comput. Vis. Graph. Image Process..

[68]  J P Frisby,et al.  PMF: A Stereo Correspondence Algorithm Using a Disparity Gradient Limit , 1985, Perception.