Exploiting Domain-Speci c Properties: Compiling Parallel Dynamic Neural Network Algorithms into E cient Code

Domain-speci c constraints can be exploited to implement compiler optimizations that are not otherwise feasible: Compilers for neural network learning algorithms can achieve near-optimal co-locality of data and processes and near-optimal balancing of load over processors, even for dynamically irregular problems. This is impossible for general programs, but restricting programs to the neural algorithm domain allows for the exploitation of domain-speci c properties: The operations performed by neural algorithms are broadcasts, reductions, and object-local operations only; the load distribution is regular with respect to the (perhaps irregular) network topology; changes of network topology occur only from time to time. A language, compilation techniques, and a compiler implementation on the MasPar MP-1 are described; quantitative results for the e ects of various optimizations used in the compiler are shown. Conservative experiments with weight pruning algorithms yield performance improvements of 27% due to load balancing; 195% improvement is achieved due to data locality, both compared to unoptimized versions. Two other optimizations, connection allocation and selecting the number of replicates, speed programs up by about 50% or 100%, respectively. This work can be viewed as a case study in exploiting domain-speci c information; some of the principles presented here may apply to other domains as well.