BFGS Optimization for Faster and Automated Supervised Learning

Standard back-propagation learning (BP) is known to have slow convergence properties. Furthermore no general prescription is given for selecting the appropriate learning rate, so success is dependent on a trial and error process. In this work a well known optimization technique (the Broyden-Fletcher-Goldfarb-Shanno memoryless quasi-Newton method) is employed to speed up convergence and to select parameters. The strict locality requirement is relaxed but parallelism of computation is maintained, allowing efficient use of concurrent computation. While requiring only limited changes to BP, this method yields a speed-up factor of 100 – 500 for the medium-size networks considered.