Using Programming Language Theory to Make Automatic Differentiation Sound and Efficient

This paper discusses a new automatic differentiation (AD) system that correctly and automatically accepts nested and dynamic use of the AD operators, without any manual intervention. The system is based on a new formulation of AD as highly generalized first-class citizens in a λ-calculus, which is briefly described. Because the λ-calculus is the basis for modern programming-language implementation techniques, integration of AD into the λ-calculus allows AD to be integrated into an aggressive compiler. We exhibit a research compiler which does this integration. Using novel analysis techniques, it accepts source code involving free use of a first-class forward AD operator and generates object code which attains numerical performance comparable to, or better than, the most aggressive existing AD systems.

[1]  A. Church.  The calculi of lambda-conversion , 1941 .

[2]  Barak A. Pearlmutter,et al.  Using Polyvariant Union-Free Flow Analysis to Compile aHigher-Order Functional-Programming Language with aFirst-Class Derivative Operator to Efficient Fortran-like Code , 2008 .

[3]  Brian Campbell,et al.  Amortised Memory Analysis Using the Depth of Data Structures , 2009, ESOP.

[4]  Joyce L. Vedral,et al.  Functional Programming Languages and Computer Architecture , 1989, Lecture Notes in Computer Science.

[5]  John Hughes,et al.  Why Functional Programming Matters , 1989, Comput. J..

[6]  Barak A. Pearlmutter,et al.  Reverse-mode AD in a functional framework: Lambda the ultimate backpropagator , 2008, TOPL.

[7]  Barak A. Pearlmutter,et al.  Putting the Automatic Back into AD:Part I, What’s Wrong (CVS: 1.1) , 2008 .

[8]  Flemming Nielson,et al.  International Workshop on Principles of Program Analysis , 1999 .

[9]  Dafydd Gibbon,et al.  1 User’s guide , 1998 .

[10]  Flemming Nielson,et al.  Principles of Program Analysis , 1999, Springer Berlin Heidelberg.

[11]  Philip Wadler,et al.  Comprehending monads , 1990, Mathematical Structures in Computer Science.

[12]  Thomas Johnsson,et al.  Lambda Lifting: Treansforming Programs to Recursive Equations , 1985, FPCA.

[13]  Barak A. Pearlmutter,et al.  Nesting forward-mode AD in a functional framework , 2008, High. Order Symb. Comput..

[14]  Laurent Hascoët,et al.  TAPENADE 2.1 user's guide , 2004 .

[15]  Barak A. Pearlmutter,et al.  First-class nonstandard interpretations by opening closures , 2007, POPL '07.