Portable and Scalable Algorithm for Irregular All-to-All Communication

In irregular all-to-all communication, messages are exchanged between every pair of processors. The message sizes vary from processor to processor and are known only at run time. This is a fundamental communication primitive in parallelizing irregularly structured scientific computations. Our algorithm reduces the total number of message start-ups. It also reduces node contention by smoothing out the lengths of the messages communicated. As compared to the earlier approaches, our algorithm provides deterministic performance and also reduces the buffer space at the nodes during message passing. The performance of the algorithm is characterised using a simple communication model of high-performance computing (HPC) platforms. We show the implementation on T3D and SP2 using C and the message passing interface standard. These can be easily ported to other HPC platforms. The results show the effectiveness of the proposed technique as well as the interplay among the machine size, the variance in message length, and the network interface.

[1]  Viktor K. Prasanna,et al.  Scalable Data Parallel Implementations of Object Recognition Using Geometric Hashing , 1994, J. Parallel Distributed Comput..

[2]  W. Gropp,et al.  Using MPI-2nd Edition , 1999 .

[3]  Geoffrey C. Fox,et al.  Supporting irregular distributions using data-parallel languages , 1995, IEEE Parallel Distributed Technol. Syst. Appl..

[4]  Thorsten von Eicken,et al.  U-Net: a user-level network interface for parallel and distributed computing , 1995, SOSP.

[5]  Charles L. Seitz,et al.  Myrinet: A Gigabit-per-Second Local Area Network , 1995, IEEE Micro.

[6]  M. Guptay,et al.  Compile-Time Estimation of Communication Costs ofPrograms , 1994 .

[7]  Shahid H. Bokhari,et al.  Optimal Multiphase Complete Exchange on Circuit-Switched Hypercube Architectures , 1994, SIGMETRICS.

[8]  David A. Bader,et al.  Practical parallel algorithms for personalized communication and integer sorting , 1996, JEAL.

[9]  Viktor K. Prasanna,et al.  Portable and scalable algorithms for irregular all-to-all communication , 1996, Proceedings of 16th International Conference on Distributed Computing Systems.

[10]  Sanjay Ranka,et al.  Personalized Communication Avoiding Node Contention on Distributed Memory Systems , 1993, 1993 International Conference on Parallel Processing - ICPP'93.

[11]  Marina del Rey,et al.  Improving PVM Performance Using ATOMIC User-Level Protocol , 1995 .

[12]  Nada Golmie,et al.  Study of interoperability between EFCI and ER switch mechanisms for ABR traffic in an ATM network , 1995, Proceedings of Fourth International Conference on Computer Communications and Networks - IC3N'95.

[13]  Sanjay Ranka,et al.  Many-to-many personalized communication with bounded traffic , 1995, Proceedings Frontiers '95. The Fifth Symposium on the Frontiers of Massively Parallel Computation.

[14]  James Demmel,et al.  ScaLAPACK: A Portable Linear Algebra Library for Distributed Memory Computers - Design Issues and Performance , 1995, Proceedings of the 1996 ACM/IEEE Conference on Supercomputing.

[15]  Viktor K. Prasanna,et al.  Efficient Algorithms for Block-Cyclic Redistribution of Arrays , 1999, Algorithmica.

[16]  Sean W. Smith,et al.  Parallelizing a global atmospheric chemical tracer model , 1994, Proceedings of IEEE Scalable High Performance Computing Conference.

[17]  Andrew A. Chien,et al.  A comparison of architectural support for messaging in the TMC CM-5 and the Cray T3D , 1995, Proceedings 22nd Annual International Symposium on Computer Architecture.

[18]  Jang Sun Lee,et al.  Communication-Efficient and Memory-Bounded External Redistribution , 1995 .

[19]  Yi Liu,et al.  Scalable S-To-P Broadcasting on Message-Passing MPPs , 1998, IEEE Trans. Parallel Distributed Syst..

[20]  David E. Culler,et al.  A Case for NOW (Networks Of Workstations) , 1995, IEEE Micro.

[21]  Yi Liu,et al.  Scalable S-to-P broadcasting on message-passing MPPs , 1996, Proceedings of the 1996 ICPP Workshop on Challenges for Parallel Processing.

[22]  R. Sarnath,et al.  Proceedings of the International Conference on Parallel Processing , 1992 .

[23]  Surajit Chaudhuri,et al.  Avoiding Retrieval Contention for Composite Multimedia Objects , 1998 .

[24]  Geoffrey C. Fox,et al.  Static and Run-Time Algorithms for All-to-Many Personalized Communication on Permutation Networks , 1994, IEEE Trans. Parallel Distributed Syst..

[25]  Shahid H. Bokhari,et al.  Multiphase complete exchange on Paragon, SP2, and CS-2 , 1996, IEEE Parallel Distributed Technol. Syst. Appl..

[26]  Viktor K. Prasanna,et al.  Parallelization of perceptual grouping on distributed memory machines , 1995, Proceedings of Conference on Computer Architectures for Machine Perception.

[27]  R. V. Shankar,et al.  Many-to-many Personalized Communication with Bounded Traac 1 Basic Communication Primitives , 2007 .

[28]  Gregory G. Finn,et al.  Atomic: A High-Speed Local Communication Architecture , 1994, J. High Speed Networks.

[29]  Susanne E. Hambrusch,et al.  Communication Operations on Coarse-Grained Mesh Architectures , 1995, Parallel Comput..

[30]  George Karypis,et al.  Introduction to Parallel Computing , 1994 .