Hi Bill, Is there a paper or anything on the hierarchical collectives ? A.Chan On Wed, 26 Mar 2008, William Gropp wrote:

We've done some work with MPI-1 where this information was made available through attributes on communicators (with Nick Karonis et al on hierarchical collectives). It would be nice to be able to make use of that again.

Bill

On Mar 25, 2008, at 9:28 PM, Anthony Chan wrote:

...

This question has come up on and off for a while: Is it a good idea for MPI to provide topological information of underlying hardware ? If the answer is yes, what is the correct way of providing the info ? For applications like PetSc, they just want to send/recv message to/from processes on the same node to take advantage of shared memory in nemesis, or may be even on same nodeboard or same rack on P. Then predefined MPI_COMM_SAME_NODE, or MPI_COMM_SAME_RACK with appropriate abstraction for deeper hierarchy will be good. Another example of the how useful of this topological info is profiling tools, slogX could use this info, e.g. geometric coordinate of the network like those in BG's MPI_Get_processor_name, it would be nice for MPI to provide that piece info so that it is portable.

I bring up the question so PMI2 or may be Hydra can be designed to provide the topology info if MPI thinks it is approproiate to provide that piece of info.

A.Chan

William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

On Wed, Mar 26, 2008 at 09:28:45AM -0500, Anthony Chan wrote:

Hi Bill,

Is there a paper or anything on the hierarchical collectives ?

http://citeseer.ist.psu.edu/601735.html ? ==rob -- Rob Latham Mathematics and Computer Science Division A215 0178 EA2D B059 8CDF Argonne National Lab, IL USA B29D F333 664A 4280 315B

The argument for the MPI topological routine is more for MPI users not MPI implementions. The same reason that MPI_Cart_create() exists, so more like a convenience function. There are user algorithms that could easily take advantage of locality in terms of network distance. Knowing which rank is closer, existing MPI app could be further exteneded to take advantage of shared memory. Let's say MPI has MPI_Get_topology() which could return NULL for sock but returns more meaning info that exposes shared memory hierarchy with nemesis. A.Chan On Wed, 26 Mar 2008, Rusty Lusk wrote:

In my opinion, this should not be part of the MPI standard, but rather a separate library that an MPI implementation can use to obtain this information to guide its collectives and topology routines. A while back, I think, MPICH exported the information to the user through attributes, but I don't think all MPI's should be required to do so. I think we did develop the interface for such a library.

Rusty

On Mar 25, 2008, at 9:28 PM, Anthony Chan wrote:

...

This question has come up on and off for a while: Is it a good idea for MPI to provide topological information of underlying hardware ? If the answer is yes, what is the correct way of providing the info ? For applications like PetSc, they just want to send/recv message to/from processes on the same node to take advantage of shared memory in nemesis, or may be even on same nodeboard or same rack on P. Then predefined MPI_COMM_SAME_NODE, or MPI_COMM_SAME_RACK with appropriate abstraction for deeper hierarchy will be good. Another example of the how useful of this topological info is profiling tools, slogX could use this info, e.g. geometric coordinate of the network like those in BG's MPI_Get_processor_name, it would be nice for MPI to provide that piece info so that it is portable.

I bring up the question so PMI2 or may be Hydra can be designed to provide the topology info if MPI thinks it is approproiate to provide that piece of info.

A.Chan