Hi Pavan, Thanks very much for your reply and for working on a patch. I tried it again with your suggested "-hosts" option, as follows, and indeed version 1.4.1p1 does work: Hello from rank 0 of 6, getpid()==21949 Hello from rank 1 of 6, getpid()==21950 Hello from rank 2 of 6, getpid()==21951 Hello from rank 3 of 6, getpid()==21945 Hello from rank 4 of 6, getpid()==21946 Hello from rank 5 of 6, getpid()==21947 mpir_proctable_t::create: extracting hostname/execname/pids for 6 processes mpir_proctable_t::create: MPIR_proctable[0]: host_name(0x005641c0)="127.0.0.2", executable_name(0x00575640)="./mpichbug", pid=21949 mpir_proctable_t::create: MPIR_proctable[1]: host_name(0x00575620)="127.0.0.2", executable_name(0x005757a0)="./mpichbug", pid=21950 mpir_proctable_t::create: MPIR_proctable[2]: host_name(0x005643a0)="127.0.0.2", executable_name(0x00575740)="./mpichbug", pid=21951 mpir_proctable_t::create: MPIR_proctable[3]: host_name(0x00564110)="127.0.0.3", executable_name(0x00575410)="./mpichbug", pid=21945 mpir_proctable_t::create: MPIR_proctable[4]: host_name(0x00575430)="127.0.0.3", executable_name(0x00575450)="./mpichbug", pid=21946 mpir_proctable_t::create: MPIR_proctable[5]: host_name(0x00575470)="127.0.0.3", executable_name(0x00575490)="./mpichbug", pid=21947 I look forward to your fix for this problem. On a different topic, the above output shows a potential performance and scalability problem with the way the MPIR_proctable[] is filled out. The MPIR_PROCDESC contain two pointer to null-terminated string members, as described in the MPIR spec: typedef struct { char *host_name; char *executable_name; int pid; } MPIR_PROCDESC; As an "advice to implementors" kind-of suggestion, the MPIR spec says: "The MPI implementation should share the host and executable name character strings across multiple process descriptor entries whenever possible. For example, if all of the MPI processes are executing �/path/a.out�, then the executable name field in each process descriptor should point to the same null-terminated character string. Sharing the strings enhances the tools scalability by allowing it to cache data from the starter process and avoid reading redundant character strings." In the output above, we can see that there are exactly three unique strings: "127.0.0.2", "127.0.0.3", and "./mpichbug". However, each string is allocated at a unique address. This arrangement not only causes the MPIR proctable entries to require more space in the MPI starter process, it requires the tool to read each string individually, thus defeating any data-caching the tool might do, and slowing the tool down when reading the MPIR proctable. If possible, it would be good is MPICH could be changed to share the host and executable name character strings across multiple process descriptor entries. Cheers, John D. Pavan Balaji wrote:

John,

This does seem like a bug. Specifically, this is a problem with the wrap-around of hosts. For example, I don't expect this problem to show up when you do:

mpiexec -hosts 127.0.0.2:3,127.0.0.3:3 -n 6 ./mpichbug

This should only show up when the number of cores is not sufficient in the first round and mpiexec has to wrap around to the first host again:

mpiexec -hosts 127.0.0.2,127.0.0.3 -n 6 ./mpichbug

I'm working on a patch. I'll commit it in shortly.

-- Pavan

On 03/11/2013 02:59 PM US Central Time, John DelSignore wrote:

...

Hi,

I'm pretty sure this is a MPICH Hydra bug, but I wanted to ask this group before I go through the trouble of figuring out how to file a MPICH bug report, which I think requires me to create an MPICH Trac account, which I don't know how to do.

As far as I can tell, Hydra fills in the MPIR_proctable[] incorrectly with multiple processes per node. The index into the MPIR_proctable[] is supposed to be the MPI process's rank in MPI_COMM_WORLD. To demonstrate this problem. I created a simple MPI "hello world" program where each MPI process prints out its rank and pid; I attached it to this email.

This is the version of MPICH I am using:

shell% /home/mware/argonne/mpich2/1.4.1p1/x86_64-linux/bin/mpirun --version|head -3 HYDRA build details: Version: 1.4.1p1 Release Date: Thu Sep 1 13:53:02 CDT 2011 shell%

I ran the code under TotalView using 2 nodes and 6 processes (3 per node). I enabled logging so that TotalView would output the contents of the MPIR_proctable[] as it extracted it from the mpirun process. Here is the output of the run:

shell% tv8cli \ -verbosity errors \ -x15 \ -parallel_stop no \ -debug_file debug.log \ -args \ /home/mware/argonne/mpich2/1.4.1p1/x86_64-linux/bin/mpirun \ -hosts 127.0.0.2,127.0.0.3 \ -n 6 \ ./mpichbug d1.<> dcont Hello from rank 0 of 6, getpid()==2691 Hello from rank 1 of 6, getpid()==2729 Hello from rank 2 of 6, getpid()==2693 Hello from rank 3 of 6, getpid()==2730 Hello from rank 4 of 6, getpid()==2694 Hello from rank 5 of 6, getpid()==2734 d1.<> quit -force shell%

Grepping for "proctable" in the debugger's log file shows the contents of the MPIR_proctable[]:

shell% grep proctable debug.log mpir_proctable_t::create: extracting hostname/execname/pids for 6 processes mpir_proctable_t::create: MPIR_proctable[0]: host_name(0x0056be20)="127.0.0.2", executable_name(0x0056be80)="./mpichbug", pid=2691 mpir_proctable_t::create: MPIR_proctable[1]: host_name(0x0056be00)="127.0.0.2", executable_name(0x0056bde0)="./mpichbug", pid=2693 mpir_proctable_t::create: MPIR_proctable[2]: host_name(0x005859a0)="127.0.0.2", executable_name(0x0056c080)="./mpichbug", pid=2694 mpir_proctable_t::create: MPIR_proctable[3]: host_name(0x005856b0)="127.0.0.3", executable_name(0x005856d0)="./mpichbug", pid=2729 mpir_proctable_t::create: MPIR_proctable[4]: host_name(0x005856f0)="127.0.0.3", executable_name(0x00585710)="./mpichbug", pid=2730 mpir_proctable_t::create: MPIR_proctable[5]: host_name(0x00585730)="127.0.0.3", executable_name(0x00585750)="./mpichbug", pid=2734 shell%

Matching up the pid values shows that the MPIR_proctable[rank] does not match the rank returned to the program by MPI_Comm_rank() for some of the MPI processes. Here's the MPI rank to MPIR_proctable rank mapping: 0 => 0 1 => 3 2 => 1 3 => 4 4 => 2 5 => 5

Do you agree that this is an MPICH Hydra bug?

Any advice on how to create an MPICH Trac account so that I can report the bug?

Thanks, John D.

_______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss