- discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Persistent Communication using MPI_SEND_INIT, MPI_RECV_INIT etc.
by Timothy Stitt 26 Mar '13

26 Mar '13

Thanks for the quick reply Jeff. That information is valuable. I'll follow up on your pointers. Much appreciated, Tim. On Mar 26, 2013, at 12:32 PM, Jeff Hammond <jhammond(a)alcf.anl.gov<mailto:[email protected]>> wrote: You might want to look at neighborhood collectives, which are discussed in Chapter 7 of MPI-3. This is a new feature so it may not be implemented in all MPI implementations, but MPICH supports it. I guess MVAPICH will support it soon enough if not already. When persistent MPI send/recv is discussed at the MPI Forum, it is often described as an inadequate solution because it does not specify a full channel and thus some important optimizations, e.g. for RDMA, may not be feasible. If you can use MPI-3 RMA, that is probably going to be a good idea, although high-quality support for RMA varies. MPICH-derived implementations usually do a good job though. Best, Jeff On Tue, Mar 26, 2013 at 11:25 AM, Timothy Stitt <Timothy.Stitt.9(a)nd.edu<mailto:[email protected]>> wrote: Hi all, I've been asking this question around various MPI boards to try and get a consensus before I decide to rewrite some MPI code. I am grateful for any advice that you can give. I've inherited a MPI code that was written ~8-10 years ago and it predominately uses MPI persistent communication routines for data transfers e.g. MPI_SEND_INIT, MPI_RECV_INIT, MPI_START etc. (which I am not familiar with and don't normally hear much discussion about). I was just wondering if using persistent communication calls is still regarded as the most efficient/scalable way to perform communication when the communication pattern is known and fixed amongst neighborhood processes? We regularly run the code across an IB network so would there be a benefit to rewrite the code using another approach (e.g. MPI one-sided communication) or should I leave it as it is? The code currently scales up to 10K cores and I want to push it even further and thus was wondering if there is any benefit in tinkering with this persistent MPI communication approach? Thanks in advance for any advice. Tim. _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov<mailto:[email protected]> / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss Tim Stitt PhD User Support Manager (CRC) Research Assistant Professor (Computer Science & Engineering) Room 108, Center for Research Computing | University of Notre Dame | P.O. Box 539, Notre Dame, IN 46556 | Phone: 574-631-5287 | Email: tstitt(a)nd.edu<mailto:[email protected]>

1 0

Re: [mpich-discuss] Persistent Communication using MPI_SEND_INIT, MPI_RECV_INIT etc.
by Jeff Hammond 26 Mar '13

26 Mar '13

You might want to look at neighborhood collectives, which are discussed in Chapter 7 of MPI-3. This is a new feature so it may not be implemented in all MPI implementations, but MPICH supports it. I guess MVAPICH will support it soon enough if not already. When persistent MPI send/recv is discussed at the MPI Forum, it is often described as an inadequate solution because it does not specify a full channel and thus some important optimizations, e.g. for RDMA, may not be feasible. If you can use MPI-3 RMA, that is probably going to be a good idea, although high-quality support for RMA varies. MPICH-derived implementations usually do a good job though. Best, Jeff On Tue, Mar 26, 2013 at 11:25 AM, Timothy Stitt <Timothy.Stitt.9(a)nd.edu> wrote: > Hi all, > > I've been asking this question around various MPI boards to try and get a consensus before I decide to rewrite some MPI code. I am grateful for any advice that you can give. > > I've inherited a MPI code that was written ~8-10 years ago and it predominately uses MPI persistent communication routines for data transfers e.g. MPI_SEND_INIT, MPI_RECV_INIT, MPI_START etc. (which I am not familiar with and don't normally hear much discussion about). I was just wondering if using persistent communication calls is still regarded as the most efficient/scalable way to perform communication when the communication pattern is known and fixed amongst neighborhood processes? We regularly run the code across an IB network so would there be a benefit to rewrite the code using another approach (e.g. MPI one-sided communication) or should I leave it as it is? The code currently scales up to 10K cores and I want to push it even further and thus was wondering if there is any benefit in tinkering with this persistent MPI communication approach? > > Thanks in advance for any advice. > > Tim. > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Persistent Communication using MPI_SEND_INIT, MPI_RECV_INIT etc.
by Timothy Stitt 26 Mar '13

26 Mar '13

Hi all, I've been asking this question around various MPI boards to try and get a consensus before I decide to rewrite some MPI code. I am grateful for any advice that you can give. I've inherited a MPI code that was written ~8-10 years ago and it predominately uses MPI persistent communication routines for data transfers e.g. MPI_SEND_INIT, MPI_RECV_INIT, MPI_START etc. (which I am not familiar with and don't normally hear much discussion about). I was just wondering if using persistent communication calls is still regarded as the most efficient/scalable way to perform communication when the communication pattern is known and fixed amongst neighborhood processes? We regularly run the code across an IB network so would there be a benefit to rewrite the code using another approach (e.g. MPI one-sided communication) or should I leave it as it is? The code currently scales up to 10K cores and I want to push it even further and thus was wondering if there is any benefit in tinkering with this persistent MPI communication approach? Thanks in advance for any advice. Tim.

1 0

Re: [mpich-discuss] MPI_Comm_Spawn causing zombies of hydra_pmi_proxy
by Silvan Brändli 26 Mar '13

26 Mar '13

Just for the record: I guess my zombie problem was caused by ignoring return codes unequal zero by using the workaround as described in http://lists.mpich.org/pipermail/discuss/2013-February/000429.html . A dirty fix which allows to ignore non-zero return codes and get rid of zombies (caused by spawning processes) is to add something like int waitstatus, child_pid; while ((child_pid = waitpid(-1, &waitstatus, WNOHANG)) > 0) { printf("HYDRA: child of pid %d finished\n",child_pid); } at the beginning of function HYDU_create_process in src/pm/hydra/utils/launch/launch.c . Best regards Silvan On 21.03.2013 12:31, Silvan Brï¿½ndli wrote: > PS: Just some additional information on compilers: so far I used icpc, > when changing it to gcc-4.7.2 I still get the zombies. The only change > in the linked libraries is an additional > libdl.so.2 => /lib64/libdl.so.2 > > Any hint / idea? > Best regards > Silvan > > On 19.03.2013 10:47, Silvan Brï¿½ndli wrote: >> Dear Pavan, >> >> in the attached example I use mpich3, but still I get the zombies. Is >> there something wrong with >> - my MPI function calls? (disconnect, finalize) >> - my linked libraries? (see below) >> >> Thanks in advance! >> Best regards >> Silvan >> >> ldd_hi >> linux-vdso.so.1 (0x00007fffa01ff000) >> libmpichcxx.so.10 => /opt/mpich3/lib/libmpichcxx.so.10 >> (0x00007fdfec0e4000) >> libmpich.so.10 => /opt/mpich3/lib/libmpich.so.10 >> (0x00007fdfebc6f000) >> libopa.so.1 => /opt/mpich3/lib/libopa.so.1 (0x00007fdfeba6d000) >> libmpl.so.1 => /opt/mpich3/lib/libmpl.so.1 (0x00007fdfeb868000) >> libaio.so.1 => /lib64/libaio.so.1 (0x00007fdfeb666000) >> libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fdfeb449000) >> libm.so.6 => /lib64/libm.so.6 (0x00007fdfeb14e000) >> libstdc++.so.6 => >> /usr/lib/gcc/x86_64-pc-linux-gnu/4.7.2/libstdc++.so.6 >> (0x00007fdfeae47000) >> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fdfeac30000) >> libc.so.6 => /lib64/libc.so.6 (0x00007fdfea882000) >> libdl.so.2 => /lib64/libdl.so.2 (0x00007fdfea67e000) >> libgfortran.so.3 => >> /usr/lib/gcc/x86_64-pc-linux-gnu/4.7.2/libgfortran.so.3 >> (0x00007fdfea361000) >> libquadmath.so.0 => >> /usr/lib/gcc/x86_64-pc-linux-gnu/4.7.2/libquadmath.so.0 >> (0x00007fdfea12b000) >> /lib64/ld-linux-x86-64.so.2 (0x00007fdfec308000) >> librt.so.1 => /lib64/librt.so.1 (0x00007fdfe9f23000) >> ldd_main >> linux-vdso.so.1 (0x00007fffeb1ff000) >> libmpichcxx.so.10 => /opt/mpich3/lib/libmpichcxx.so.10 >> (0x00007f6fc797e000) >> libmpich.so.10 => /opt/mpich3/lib/libmpich.so.10 >> (0x00007f6fc7509000) >> libopa.so.1 => /opt/mpich3/lib/libopa.so.1 (0x00007f6fc7307000) >> libmpl.so.1 => /opt/mpich3/lib/libmpl.so.1 (0x00007f6fc7102000) >> libaio.so.1 => /lib64/libaio.so.1 (0x00007f6fc6f00000) >> libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f6fc6ce3000) >> libm.so.6 => /lib64/libm.so.6 (0x00007f6fc69e8000) >> libstdc++.so.6 => >> /usr/lib/gcc/x86_64-pc-linux-gnu/4.7.2/libstdc++.so.6 >> (0x00007f6fc66e1000) >> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f6fc64ca000) >> libc.so.6 => /lib64/libc.so.6 (0x00007f6fc611c000) >> libdl.so.2 => /lib64/libdl.so.2 (0x00007f6fc5f18000) >> libgfortran.so.3 => >> /usr/lib/gcc/x86_64-pc-linux-gnu/4.7.2/libgfortran.so.3 >> (0x00007f6fc5bfb000) >> libquadmath.so.0 => >> /usr/lib/gcc/x86_64-pc-linux-gnu/4.7.2/libquadmath.so.0 >> (0x00007f6fc59c5000) >> /lib64/ld-linux-x86-64.so.2 (0x00007f6fc7ba2000) >> librt.so.1 => /lib64/librt.so.1 (0x00007f6fc57bd000) >> >> On 07.03.2013 17:23, Pavan Balaji wrote: >>> >>> I just tried this with the mpich master and it seems to work correctly, >>> and there are no zombie processes (though I reduced the number of >>> iterations to 10000, instead of 200000). This was a problem in mpich >>> once upon a time, but that was a few years ago. Are you using the >>> latest version of mpich (3.0.2)? >>> >>> -- Pavan >>> >>> On 03/07/2013 09:44 AM US Central Time, Silvan Brï¿½ndli wrote: >>>> PS: The attached programs are a simplification of my code. They >>>> reproduce the zombie problem. Waiting for the 32k zombies takes a >>>> while... but I expect the same behaviour as with my original code. >>>> >>>> Am I missing something when finishing the called program? I just use >>>> MPI_Comm_disconnect and MPI_Finalize. >>>> >>>> Best regards >>>> Silvan >>>> >>>> main.cpp >>>> >>>> #include <mpi.h>; >>>> >>>> int main(int argc, char *argv[]) >>>> { >>>> int myrank; >>>> int spawnerror; >>>> int value = 123; >>>> void *buf = &value; >>>> MPI_Comm child_comm; >>>> >>>> if (MPI_Init(&argc,&argv)!=MPI_SUCCESS) >>>> { >>>> printf("MPI_Init failed"); >>>> } >>>> >>>> MPI_Comm_rank(MPI_COMM_WORLD, &myrank); >>>> >>>> char* hiargv[] = {"23",NULL}; >>>> for(int i = 1; i <= 200000; i++) >>>> { >>>> value = i; >>>> printf("Main before spawn %d\n",i); >>>> MPI_Comm_spawn("./hi",hiargv, 1, MPI_INFO_NULL, myrank, >>>> MPI_COMM_SELF, &child_comm, &spawnerror); >>>> MPI_Send(buf, 1, MPI_INTEGER, 0, 1, child_comm); >>>> MPI_Comm_disconnect(&child_comm); >>>> } >>>> >>>> MPI_Finalize(); >>>> return 0; >>>> } >>>> >>>> hi.cpp: >>>> >>>> #include <mpi.h>; >>>> >>>> int main(int argc, char** argv) { >>>> MPI_Comm parent; >>>> MPI_Status status; >>>> int err; >>>> int value = -1; >>>> void* buf= &value; >>>> >>>> if (MPI_Init(&argc,&argv)!=MPI_SUCCESS) >>>> { >>>> printf("MPI_Init failed"); >>>> } >>>> MPI_Comm_get_parent(&parent); >>>> if (parent == MPI_COMM_NULL) printf("No parent!"); >>>> >>>> MPI_Recv(buf, 1, MPI_INTEGER, 0, MPI_ANY_TAG, parent, &status); >>>> MPI_Comm_disconnect(&parent); >>>> err = MPI_Finalize(); >>>> printf("hi finalized %d %d \n",err, value); >>>> return 0; >>>> } >>>> >>>> >>>> >>>> On 07.03.2013 12:38, Silvan Brï¿½ndli wrote: >>>>> Dear all, >>>>> >>>>> again I have a question related to spawning processes. I understand >>>>> the >>>>> situation as follows: >>>>> >>>>> My program A spawns program B. Program B spawns program C1, C2 ... >>>>> C10000 ... >>>>> Program Cx terminates correctly before Cx+1 is called, however >>>>> returning >>>>> 1 to mpiexec. To handle this I use the workaround as described in >>>>> http://lists.mpich.org/pipermail/discuss/2013-February/000429.html >>>>> >>>>> Now it looks like with every Spawn a "hydra_pmi_proxy" is started, the >>>>> calling program is mpiexec. When the program Cx is finished this >>>>> "hydra_pmi_proxy" remains as a zombie until the programs A, B and >>>>> mpiexec are finished. When approx. 32k of those "hydra_pmi_proxy" >>>>> exist >>>>> I get some problems (too many processes or something similar). >>>>> >>>>> What can I do to finish "hydra_pmi_proxy" while my programs A, B and >>>>> mpiexec are still running? >>>>> >>>>> I'm glad about every hint. >>>>> >>>>> Best regards >>>>> Silvan >>>>> >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> >>> >> >> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > -- Dipl.-Ing. Silvan Brï¿½ndli Numerische Strukturanalyse mit Anwendungen in der Schiffstechnik (M-10) Technische Universitï¿½t Hamburg-Harburg Schwarzenbergstraï¿½e 95c 21073 Hamburg Tel. : +49 (0)40 42878 - 6187 Fax. : +49 (0)40 42878 - 6090 e-mail: silvan.braendli(a)tuhh.de www : http://www.tuhh.de/skf 5th GACM Colloquium on Computational Mechanics http://www.tu-harburg.de/gacm2013

1 0

Re: [mpich-discuss] merging intracommuncators
by Rajeev Thakur 25 Mar '13

25 Mar '13

Try doing the second spawn with intracomm instead of MPI_COMM_WORLD as the communicator. However, it would be collective over the first set of children as well, i.e., both the parent and the first set of children would need to call the second spawn in this case. Rajeev On Mar 25, 2013, at 5:13 PM, Ali Bouaricha wrote: > Hello, > > I'm using MPI_Comm_Spawn to spawn processes dynamically. I spawn > few processes, then I execute something, then I spawn few more processes. > I want to merge intracommunicators in such way that the master > Processes including all the spawn children processes (the first and > Second time) would have rank > 0, 1, 2, 3, .... p-1. > > My problem is that I cannot merge intracommunicators. Here is what I mean: > > Parent code: > =========== > > error = MPI_Comm_spawn("child", MPI_ARGV_NULL, 2, > info, 0, MPI_COMM_WORLD, &intercomm, errcodes); > > MPI_Intercomm_merge(intercomm, 0, &intracomm); > ..... > > do something > > .... > error = MPI_Comm_spawn("child", MPI_ARGV_NULL, 3, > info, 0, MPI_COMM_WORLD, &intercomm, errcodes); > > MPI_Intercomm_merge(intercomm, 0, &intracomm); > > > > The child code: > =============== > > MPI_Comm_get_parent(&intercomm); > > MPI_Comm_size(intercomm,&nprocs); > printf(" child nprocs = %d\n", nprocs); > > MPI_Intercomm_merge(intercomm, 0, &intracomm); > > MPI_Comm_rank(intracomm, &rank); > printf(" child rank = %d\n", rank); > > ===== > > The outptut after executing: mpiexec -n 1 ./parent > > > Gives: child rank 1, 2 (for the first spawn) > Child rank 1, 2, 3 (for the second spawn) > > The question is that how I can merge so that my processes would have ranks > 1, 2, 3, 4, 5? > > Thanks a lot, > > Ali > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

merging intracommuncators
by Ali Bouaricha 25 Mar '13

25 Mar '13

Hello, I'm using MPI_Comm_Spawn to spawn processes dynamically. I spawn few processes, then I execute something, then I spawn few more processes. I want to merge intracommunicators in such way that the master Processes including all the spawn children processes (the first and Second time) would have rank 0, 1, 2, 3, .... p-1. My problem is that I cannot merge intracommunicators. Here is what I mean: Parent code: =========== error = MPI_Comm_spawn("child", MPI_ARGV_NULL, 2, info, 0, MPI_COMM_WORLD, &intercomm, errcodes); MPI_Intercomm_merge(intercomm, 0, &intracomm); ..... do something .... error = MPI_Comm_spawn("child", MPI_ARGV_NULL, 3, info, 0, MPI_COMM_WORLD, &intercomm, errcodes); MPI_Intercomm_merge(intercomm, 0, &intracomm); The child code: =============== MPI_Comm_get_parent(&intercomm); MPI_Comm_size(intercomm,&nprocs); printf(" child nprocs = %d\n", nprocs); MPI_Intercomm_merge(intercomm, 0, &intracomm); MPI_Comm_rank(intracomm, &rank); printf(" child rank = %d\n", rank); ===== The outptut after executing: mpiexec -n 1 ./parent Gives: child rank 1, 2 (for the first spawn) Child rank 1, 2, 3 (for the second spawn) The question is that how I can merge so that my processes would have ranks 1, 2, 3, 4, 5? Thanks a lot, Ali

1 0

Re: [mpich-discuss] Problem in the step of "make"
by Eric A. Borisch 22 Mar '13

22 Mar '13

On Fri, Mar 22, 2013 at 2:48 PM, yuanp <peng.yuan(a)student.ecp.fr> wrote: > Yeah, I thought it was successful and did a make install and setted the > PATH. But when I ran some tests I met some problems and I thought it was > due to the step make. > I copied the code from my terminal > > dhcp-27-100:mpich-install holmes_conan$ ssh othermachine date > ssh: Could not resolve hostname othermachine: nodename nor servname > provided, or not known > dhcp-27-100:mpich-install holmes_conan$ rsh othermachine date > rcmd: getaddrinfo: nodename nor servname provided, or not known > This *looks* like examples of testing connectivity: 'ssh othermachine date' should try to execute 'date' on the machine named 'othermachine'; you likely need to replace othermachine with the name of the other machine you're connecting to; or localhost if you're just using one machine (ie testing that sshd is running & accepting connections, likely trying to make sure it's working passwordless.) The two commands above and their responses have nothing to do with mpich's installation state. -- Eric

1 0

Re: [mpich-discuss] Problem in the step of "make"
by yuanp 22 Mar '13

22 Mar '13

Yeah, I thought it was successful and did a make install and setted the PATH. But when I ran some tests I met some problems and I thought it was due to the step make. I copied the code from my terminal dhcp-27-100:mpich-install holmes_conan$ ssh othermachine date ssh: Could not resolve hostname othermachine: nodename nor servname provided, or not known dhcp-27-100:mpich-install holmes_conan$ rsh othermachine date rcmd: getaddrinfo: nodename nor servname provided, or not known I will then try macport On 13-3-22 下午5:59, Rajeev Thakur wrote: > The make seems to have gone through successfully. What problem do you see? You just need to do a make install now. > > On Mar 22, 2013, at 10:35 AM, yuanp wrote: > >> Hello, >> >> I am currently a student at Ecole Central Paris in Paris following the cours of High Performance Computing which demand the installation of mpich2. Possessing a macbook air with OSX Lion version 10.7.5, I have met a problem when install mpich2 at the step "make". I don't know what to do now. >> I have enclosed the c.txt, m.txt, mi.txt files for you. Thank you for help. >> >> Best regards >> >> Peng YUAN >> <Archive.zip>_______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] Problem in the step of "make"
by Dave Goodell 22 Mar '13

22 Mar '13

On Mar 22, 2013, at 11:25 AM CDT, Eric A. Borisch <eborisch(a)ieee.org> wrote: > On Fri, Mar 22, 2013 at 10:35 AM, yuanp <peng.yuan(a)student.ecp.fr> wrote: > Hello, > > I am currently a student at Ecole Central Paris in Paris following the cours of High Performance Computing which demand the installation of mpich2. Possessing a macbook air with OSX Lion version 10.7.5, I have met a problem when install mpich2 at the step "make". I don't know what to do now. > I have enclosed the c.txt, m.txt, mi.txt files for you. Thank you for help. > > Any reason you can't use MacPorts to install mpich? Or homebrew, if you don't prefer MacPorts. Eric maintains a nice package in MacPorts specifically to make this process easy, and I do the same for homebrew: "sudo port install mpich" OR "brew install mpich" -Dave

1 0

Re: [mpich-discuss] Problem in the step of "make"
by Rajeev Thakur 22 Mar '13

22 Mar '13

The make seems to have gone through successfully. What problem do you see? You just need to do a make install now. On Mar 22, 2013, at 10:35 AM, yuanp wrote: > Hello, > > I am currently a student at Ecole Central Paris in Paris following the cours of High Performance Computing which demand the installation of mpich2. Possessing a macbook air with OSX Lion version 10.7.5, I have met a problem when install mpich2 at the step "make". I don't know what to do now. > I have enclosed the c.txt, m.txt, mi.txt files for you. Thank you for help. > > Best regards > > Peng YUAN > <Archive.zip>_______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0