OpenDR: An Approximate Differentiable Renderer

Inverse graphics attempts to take sensor data and infer 3D geometry, illumination, materials, and motions such that a graphics renderer could realistically reproduce the observed scene. Renderers, however, are designed to solve the forward process of image synthesis. To go in the other direction, we propose an approximate differentiable renderer (DR) that explicitly models the relationship between changes in model parameters and image observations. We describe a publicly available OpenDR framework that makes it easy to express a forward graphics model and then automatically obtain derivatives with respect to the model parameters and to optimize over them. Built on a new auto-differentiation package and OpenGL, OpenDR provides a local optimization method that can be incorporated into probabilistic programming frameworks. We demonstrate the power and simplicity of programming with OpenDR by using it to solve the problem of estimating human body shape from Kinect depth and RGB data.

[1]  Bruce G. Baumgart,et al.  Geometric modeling for computer vision. , 1974 .

[2]  Berthold K. P. Horn.  Understanding Image Intensities , 1977, Artif. Intell..

[3]  Ulf Grenander.  Metric Pattern Theory , 1981 .

[4]  Demetri Terzopoulos,et al.  Regularization of Inverse Visual Problems Involving Discontinuities , 1986, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[5]  M. Bertero,et al.  Ill-posed problems in early vision , 1988, Proc. IEEE.

[6]  James F. Blinn,et al.  Physically-based modeling: past, present, and future , 1989, SIGGRAPH '89.

[7]  Joel L. Davis,et al.  Large-Scale Neuronal Theories of the Brain , 1994 .

[8]  David Mumford,et al.  Neuronal Architectures for Pattern-theoretic Problems , 1995 .

[9]  Michael J. Jones,et al.  Model-Based Matching by Linear Combinations of Prototypes , 1996 .

[10]  Janne Heikkilä,et al.  A four-step camera calibration procedure with implicit image correction , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[11]  Daniel Kersten,et al.  Inverse 3-D graphics: A metaphor for visual perception , 1997 .

[12]  Matthew Turk,et al.  A Morphable Model For The Synthesis Of 3D Faces , 1999, SIGGRAPH.

[13]  Andreas Griewank,et al.  Evaluating derivatives - principles and techniques of algorithmic differentiation, Second Edition , 2000, Frontiers in applied mathematics.

[14]  Eric Jones,et al.  SciPy: Open Source Scientific Tools for Python , 2001 .

[15]  Cristian Sminchisescu,et al.  A Framework for Generic State Estimation in Computer Vision Applications , 2001, ICVS.

[16]  Mads Nielsen,et al.  Computer Vision — ECCV 2002 , 2002, Lecture Notes in Computer Science.

[17]  David A. Maluf,et al.  Dramatic Improvements to Feature Based Stereo , 2002, ECCV.

[18]  Kathleen M. Robinette,et al.  Civilian American and European Surface Anthropometry Resource (CAESAR), Final Report. Volume 1. Summary , 2002 .

[19]  Cristian Sminchisescu,et al.  Estimation algorithms for ambiguous visual models : Three Dimensional Human Modeling and Motion Reconstruction in Monocular Video Sequences. (Algorithmes d'estimation pour des modèles visuels ambigus : Modélisation Humaine Tridimensionnelle et Reconstruction du Mouvement dans des Séquences Vidéo Mon , 2002 .

[20]  Zoran Popovic,et al.  The space of human body shapes: reconstruction and parameterization from range scans , 2003, ACM Trans. Graph..

[21]  Gustavo Patow,et al.  A Survey of Inverse Rendering Problems , 2003, Comput. Graph. Forum.

[22]  John C. Stutz,et al.  Modeling Images of Natural 3D Surfaces: Overview and Potential Applications , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[23]  Travis E. Oliphant,et al.  Python for Scientific Computing , 2007, Computing in Science & Engineering.

[24]  J. Bastian.  Reconstructing 3D geometry from multiple images via inverse rendering. , 2008 .

[25]  David J. Fleet,et al.  Model-based hand tracking with texture, shading and self-occlusions , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[26]  Joshua B. Tenenbaum,et al.  Church: a language for generative models , 2008, UAI.

[27]  Razvan Pascanu,et al.  Theano: A CPU and GPU Math Compiler in Python , 2010, SciPy.

[28]  Michael J. Black,et al.  Home 3D body scans from noisy image and range data , 2011, 2011 International Conference on Computer Vision.

[29]  Michael J. Black,et al.  Coregistration: Simultaneous Alignment and Modeling of Articulated 3D Shape , 2012, ECCV.

[30]  Andrew W. Fitzgibbon,et al.  A unifying resolution-independent formulation for early vision , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[31]  Matthieu Guillaumin,et al.  Segmentation Propagation in ImageNet , 2012, ECCV.

[32]  Erol Gelenbe,et al.  Natural Computation , 2012, Comput. J..

[33]  Joshua B. Tenenbaum,et al.  Approximate Bayesian Image Interpretation using Generative Probabilistic Graphics Programs , 2013, NIPS.

[34]  Frank D. Wood,et al.  A New Approach to Probabilistic Programming Inference , 2014, AISTATS.

[35]  Adam Binch,et al.  Perception as Bayesian Inference , 2014 .

[36]  Sebastian Nowozin,et al.  The informed sampler: A discriminative approach to Bayesian inference in generative computer vision models , 2014, Comput. Vis. Image Underst..