Automatic Differentiation and Ubiquitous Adaptive Computing

Computer science has worked for decades to realize a sort of pantheistic dream, in which the inanimate objects that surround us are endowed with a vital computational essence and become our magical servants. The hardware is coming along nicely, but the vital essence is static: despite their remarkable abilities, our servants are bullheaded and unchanging. In order to make such complex systems adaptive, in a variety of senses, we need for adaptation to be compositional and automatic. As a step in that direction, we’ve tried to push a simple adaptive mechanism—gradient optimization—into fully general settings, complete with iteration and nesting and Turing equivalence. People now call this ‘differentiable programming’. I’ll describe some of the automatic differentiation underpinnings that seem poised to help make the computers that surround us a bit less annoying. Copyright © 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).