DiffSharp: An AD Library for .NET Languages

DiffSharp is an algorithmic differentiation or automatic differentiation (AD) library for the .NET ecosystem, which is targeted by the C# and F# languages, among others. The library has been designed with machine learning applications in mind, allowing very succinct implementations of models and optimization routines. DiffSharp is implemented in F# and exposes forward and reverse AD operators as general nestable higher-order functions, usable by any .NET language. It provides high-performance linear algebra primitives---scalars, vectors, and matrices, with a generalization to tensors underway---that are fully supported by all the AD operators, and which use a BLAS/LAPACK backend via the highly optimized OpenBLAS library. DiffSharp currently uses operator overloading, but we are developing a transformation-based version of the library using F#'s "code quotation" metaprogramming facility. Work on a CUDA-based GPU backend is also underway.

[1]  B. Christianson.  Reverse accumulation and attractive fixed points , 1994 .

[2]  Uwe Naumann,et al.  GPU-accelerated adjoint algorithmic differentiation , 2016, Comput. Phys. Commun..

[3]  Barak A. Pearlmutter,et al.  Nesting forward-mode AD in a functional framework , 2008, High. Order Symb. Comput..

[4]  R. E. Wengert,et al.  A simple automatic derivative evaluation program , 1964, Commun. ACM.

[5]  Andrea Walther,et al.  Differentiating Fixed Point Iterations with ADOL-C: Gradient Calculation for Fluid Dynamics , 2006, HPSC.

[6]  Barak A. Pearlmutter,et al.  Automatic differentiation in machine learning: a survey , 2015, J. Mach. Learn. Res..

[7]  Ryan P. Adams,et al.  Gradient-based Hyperparameter Optimization through Reversible Learning , 2015, ICML.

[8]  Barak A. Pearlmutter,et al.  DiffSharp: Automatic Differentiation Library , 2015, ArXiv.

[9]  Yoshua Bengio,et al.  Deep Learning of Representations: Looking Forward , 2013, SLSP.

[10]  Andreas Griewank,et al.  Evaluating derivatives - principles and techniques of algorithmic differentiation, Second Edition , 2000, Frontiers in applied mathematics.

[11]  Barak A. Pearlmutter,et al.  Perturbation Confusion and Referential Transparency:Correct Functional Implementation of Forward-Mode AD , 2005 .

[12]  Barak A. Pearlmutter,et al.  Reverse-mode AD in a functional framework: Lambda the ultimate backpropagator , 2008, TOPL.

[13]  M. Giles.  Collected Matrix Derivative Results for Forward and Reverse Mode Algorithmic Differentiation , 2008 .

[14]  Don Syme.  Leveraging .NET meta-programming components from F#: integrated queries and interoperable heterogeneous execution , 2006, ML '06.