Algorithmic Mapping of Neural Network Models onto Parallel SIMD Machines

Implementations of neural networks on programmable massively parallel computers are addressed. The methods are based on a graph theoretic approach and are applicable to a large class of networks in which the computations can be described by means of matrix and vector operations. A detailed characterization of the target machine is provided. Two mappings are presented. The first is designed for a processor array consisting of a very large number of small processing units. The neurons and the nonzero synaptic weights are assigned to the processors in a predetermined order, one per processor. The data transfers between processors containing neurons and weights are implemented using a novel routing algorithm. The second mapping is designed for the data array of size N*N and a smaller processor array of size P*P, P >

[1]  Charles Clos,et al.  A study of non-blocking switching networks , 1953 .

[2]  V. Benes.  On rearrangeable three-stage connecting networks , 1962 .

[3]  Sartaj Sahni,et al.  Data broadcasting in SIMD computers , 1981, IEEE Transactions on Computers.

[4]  Sartaj Sahni,et al.  Optimal BPC Permutations on a Cube Connected SIMD Computer , 1982, IEEE Transactions on Computers.

[5]  W. Daniel Hillis,et al.  The connection machine , 1985 .

[6]  Jerry L. Potter.  The Massively Parallel Processor , 1985 .

[7]  Harold M. Hastings,et al.  Neural nets on the MPP , 1987 .

[8]  Geoffrey E. Hinton,et al.  Connectionist Architectures for Artificial Intelligence , 1990, Computer.

[9]  Guy E. Blelloch,et al.  Network Learning on the Connection Machine , 1987, IJCAI.

[10]  U. Schwiegelshohn.  A shortperiodic two-dimensional systolic sorting algorithm , 1988, [1988] Proceedings. International Conference on Systolic Arrays.

[11]  S. Y. Kung,et al.  Parallel architectures for artificial neural nets , 1988, IEEE 1988 International Conference on Neural Networks.

[12]  D. S. Touretzky,et al.  Neural network simulation at Warp speed: how we got 17 million connections per second , 1988, IEEE 1988 International Conference on Neural Networks.

[13]  S. Tomboulian.  Overview and extensions of a system for routing directed graphs on SIMD architectures , 1988, Proceedings., 2nd Symposium on the Frontiers of Massively Parallel Computation.

[14]  Jan Grinberg,et al.  The 3-D Computer: An Integrated Stack of WSI Wafers , 1989 .

[15]  Jenq-Neng Hwang,et al.  A Unified Systolic Architecture for Artificial Neural Networks , 1989, J. Parallel Distributed Comput..

[16]  Yann LeCun,et al.  Optimal Brain Damage , 1989, NIPS.

[17]  K. W. Przytula,et al.  Mapping of neural networks onto programmable parallel machines , 1990, IEEE International Symposium on Circuits and Systems.

[18]  Viktor K. Prasanna,et al.  Algorithmic mapping of neural network models onto parallel SIMD machines , 1990, [1990] Proceedings of the International Conference on Application Specific Array Processors.

[19]  Nam Ling,et al.  Systolic architectures for artificial neural nets , 1991, [Proceedings] 1991 IEEE International Joint Conference on Neural Networks.

[20]  V. K. Prasanna,et al.  Mapping image algorithms onto window architecture , 1991 .