Fast neural net simulation with a DSP processor array

This paper describes the implementation of a fast neural net simulator on a novel parallel distributed-memory computer. A 60-processor system, named MUSIC (multiprocessor system with intelligent communication), is operational and runs the backpropagation algorithm at a speed of 330 million connection updates per second (continuous weight update) using 32-b floating-point precision. This is equal to 1.4 Gflops sustained performance. The complete system with 3.8 Gflops peak performance consumes less than 800 W of electrical power and fits into a 19-in rack. While reaching the speed of modern supercomputers, MUSIC still can be used as a personal desktop computer at a researcher's own disposal. In neural net simulation, this gives a computing performance to a single user which was unthinkable before. The system's real-time interfaces make it especially useful for embedded applications.

[1]  G. Amdhal,et al.  Validity of the single processor approach to achieving large scale computing capabilities , 1967, AFIPS '67 (Spring).

[2]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[3]  H. T. Kung,et al.  The Warp Computer: Architecture, Implementation, and Performance , 1987, IEEE Transactions on Computers.

[4]  Terrence J. Sejnowski,et al.  Parallel Networks that Learn to Pronounce English Text , 1987, Complex Syst..

[5]  D. S. Touretzky,et al.  Neural network simulation at Warp speed: how we got 17 million connections per second , 1988, IEEE 1988 International Conference on Neural Networks.

[6]  Yann LeCun,et al.  Generalization and network design strategies , 1989 .

[7]  Yann LeCun,et al.  Improving the convergence of back-propagation learning with second-order methods , 1989 .

[8]  Jill P. Mesirov,et al.  An Efficient Implementation of the Back-propagation Algorithm on the Connection Machine CM-2 , 1989, NIPS.

[9]  Michael J. Witbrock,et al.  An implementation of backpropagation learning on GF11, a large SIMD parallel computer , 1990, Parallel Comput..

[10]  Hal McCartor,et al.  Back Propagation Implementation on the Adaptive Solutions CNAPS Neurocomputer Chip , 1990, NIPS 1990.

[11]  Seungryoul Maeng,et al.  Parallel simulation of multilayered neural networks on distributed-memory multiprocessors , 1990 .

[12]  Soheil Shams,et al.  Implementation of Multilayer Neural Networks on Parallel Programmable Digital Computers , 1991 .

[13]  Jeff A. Bilmes,et al.  Software for ANN Training on a Ring Array Processor , 1991, NIPS.

[14]  Anton Gunzinger,et al.  Achieving supercomputer performane for neural net simulation with an array of digital signal processors , 1992, IEEE Micro.

[15]  Nazif Tepedelenlioglu,et al.  A fast new algorithm for training feedforward neural networks , 1992, IEEE Trans. Signal Process..

[16]  R. R. Shively,et al.  Application and packaging of the AT&T DSP3 parallel signal processor , 1992, [1992] Proceedings of the International Conference on Application Specific Array Processors.

[17]  Jeff A. Bilmes,et al.  The Ring Array Processor: A Multiprocessing Peripheral for Connection Applications , 1992, J. Parallel Distributed Comput..

[18]  Anton Gunzinger,et al.  Achieving super computer performance with a DSP array processor , 1992, Proceedings Supercomputing '92.

[19]  Jean-François Leber.  The recognition of acoustical signals using neural networks and an open simulator , 1992 .

[20]  Anton Gunzinger,et al.  Architecture and realization of a multi signal processor system , 1992, [1992] Proceedings of the International Conference on Application Specific Array Processors.

[21]  Christian Halloy,et al.  Neural Network Simulations on Massively Parallel Computers: Applications in Chemical Physics , 1993, IWANN.

[22]  K. Asakawa,et al.  Highly parallel architecture for back-propagation using a ring-register data path , 1993 .

[23]  Xiao Liu,et al.  Benchmarking of the CM-5 and the Cray machines with a very large backpropagation neural network , 1994, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94).