March 2013 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Persistent Communication using MPI_SEND_INIT, MPI_RECV_INIT etc.
by Pavan Balaji 26 Mar '13

26 Mar '13

Hi Tim, I didn't mean for you to give me a 1-paragraph summary of the communication model. My first guess is a neighborhood collective (in MPI-3), which Jeff already suggested, but you'll need to look through the capabilities of the different MPI operations and decide. With respect to MPI implementations, I guess most MPICH derivatives (including MVAPICH, Cray MPI, IBM MPI, Intel MPI, etc.) should have MPI-3 capability this year or early next year, though we don't have an official date from most of them. I believe the latest release of MVAPICH already has MPI-3 support. -- Pavan On 03/26/2013 01:46 PM US Central Time, Timothy Stitt wrote: > Pavan...thanks for the comments. > > Our finite element simulation decomposes the problem domain into a 2D grid of sub-domains with a 1-1 mapping between compute cores and sub-domains. In each sub-domain we apply a ghost zone between direct neighbors and communicate the relevant data between neighbors during each timestep. > > In general we use MVAPICH2 on the various systems we have access to. When using a Cray we use their tuned MPI library which I believe is derived from MPICH2. If we primarily are MPICH-based (and with a basic idea of our communication pattern as given above) would you recommend looking into some other MPI(-3) functionality to achieve better scalability and performance? > > Cheers, > > Tim. > > On Mar 26, 2013, at 12:59 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > >> Tim, >> >> It's hard for us to decide which MPI functionality is better for you >> without knowing your algorithmic model. For some algorithms persistent >> send/recv is great. For some, RMA is great. Of course, each one has >> its shortcomings. Please don't jump to one or the other based on a >> short email discussion :-). >> >> Also note that some MPI implementations do optimize persistent >> communication, particularly for memory registration. So even if you >> don't see a benefit on some platforms, it doesn't mean that other MPI >> implementations cannot take advantage of it. >> >> -- Pavan >> >> On 03/26/2013 11:36 AM US Central Time, Timothy Stitt wrote: >>> Thanks for the quick reply Jeff. That information is valuable. I'll >>> follow up on your pointers. >>> >>> Much appreciated, >>> >>> Tim. >>> >>> On Mar 26, 2013, at 12:32 PM, Jeff Hammond <jhammond(a)alcf.anl.gov >>> <mailto:[email protected]>> wrote: >>> >>>> You might want to look at neighborhood collectives, which are >>>> discussed in Chapter 7 of MPI-3. This is a new feature so it may not >>>> be implemented in all MPI implementations, but MPICH supports it. I >>>> guess MVAPICH will support it soon enough if not already. >>>> >>>> When persistent MPI send/recv is discussed at the MPI Forum, it is >>>> often described as an inadequate solution because it does not specify >>>> a full channel and thus some important optimizations, e.g. for RDMA, >>>> may not be feasible. >>>> >>>> If you can use MPI-3 RMA, that is probably going to be a good idea, >>>> although high-quality support for RMA varies. MPICH-derived >>>> implementations usually do a good job though. >>>> >>>> Best, >>>> >>>> Jeff >>>> >>>> On Tue, Mar 26, 2013 at 11:25 AM, Timothy Stitt >>>> <Timothy.Stitt.9(a)nd.edu <mailto:[email protected]>> wrote: >>>>> Hi all, >>>>> >>>>> I've been asking this question around various MPI boards to try and >>>>> get a consensus before I decide to rewrite some MPI code. I am >>>>> grateful for any advice that you can give. >>>>> >>>>> I've inherited a MPI code that was written ~8-10 years ago and it >>>>> predominately uses MPI persistent communication routines for data >>>>> transfers e.g. MPI_SEND_INIT, MPI_RECV_INIT, MPI_START etc. (which I >>>>> am not familiar with and don't normally hear much discussion about). >>>>> I was just wondering if using persistent communication calls is still >>>>> regarded as the most efficient/scalable way to perform communication >>>>> when the communication pattern is known and fixed amongst >>>>> neighborhood processes? We regularly run the code across an IB >>>>> network so would there be a benefit to rewrite the code using another >>>>> approach (e.g. MPI one-sided communication) or should I leave it as >>>>> it is? The code currently scales up to 10K cores and I want to push >>>>> it even further and thus was wondering if there is any benefit in >>>>> tinkering with this persistent MPI communication approach? >>>>> >>>>> Thanks in advance for any advice. >>>>> >>>>> Tim. >>>>> >>>>> _______________________________________________ >>>>> discuss mailing list discuss(a)mpich.org <mailto:[email protected]> >>>>> To manage subscription options or unsubscribe: >>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> >>>> >>>> >>>> -- >>>> Jeff Hammond >>>> Argonne Leadership Computing Facility >>>> University of Chicago Computation Institute >>>> jhammond(a)alcf.anl.gov <mailto:[email protected]> / (630) 252-5381 >>>> http://www.linkedin.com/in/jeffhammond >>>> https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>> >>> >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >> >> -- >> Pavan Balaji >> http://www.mcs.anl.gov/~balaji > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] Persistent Communication using MPI_SEND_INIT, MPI_RECV_INIT etc.
by Timothy Stitt 26 Mar '13

26 Mar '13

Pavan...thanks for the comments. Our finite element simulation decomposes the problem domain into a 2D grid of sub-domains with a 1-1 mapping between compute cores and sub-domains. In each sub-domain we apply a ghost zone between direct neighbors and communicate the relevant data between neighbors during each timestep. In general we use MVAPICH2 on the various systems we have access to. When using a Cray we use their tuned MPI library which I believe is derived from MPICH2. If we primarily are MPICH-based (and with a basic idea of our communication pattern as given above) would you recommend looking into some other MPI(-3) functionality to achieve better scalability and performance? Cheers, Tim. On Mar 26, 2013, at 12:59 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > Tim, > > It's hard for us to decide which MPI functionality is better for you > without knowing your algorithmic model. For some algorithms persistent > send/recv is great. For some, RMA is great. Of course, each one has > its shortcomings. Please don't jump to one or the other based on a > short email discussion :-). > > Also note that some MPI implementations do optimize persistent > communication, particularly for memory registration. So even if you > don't see a benefit on some platforms, it doesn't mean that other MPI > implementations cannot take advantage of it. > > -- Pavan > > On 03/26/2013 11:36 AM US Central Time, Timothy Stitt wrote: >> Thanks for the quick reply Jeff. That information is valuable. I'll >> follow up on your pointers. >> >> Much appreciated, >> >> Tim. >> >> On Mar 26, 2013, at 12:32 PM, Jeff Hammond <jhammond(a)alcf.anl.gov >> <mailto:[email protected]>> wrote: >> >>> You might want to look at neighborhood collectives, which are >>> discussed in Chapter 7 of MPI-3. This is a new feature so it may not >>> be implemented in all MPI implementations, but MPICH supports it. I >>> guess MVAPICH will support it soon enough if not already. >>> >>> When persistent MPI send/recv is discussed at the MPI Forum, it is >>> often described as an inadequate solution because it does not specify >>> a full channel and thus some important optimizations, e.g. for RDMA, >>> may not be feasible. >>> >>> If you can use MPI-3 RMA, that is probably going to be a good idea, >>> although high-quality support for RMA varies. MPICH-derived >>> implementations usually do a good job though. >>> >>> Best, >>> >>> Jeff >>> >>> On Tue, Mar 26, 2013 at 11:25 AM, Timothy Stitt >>> <Timothy.Stitt.9(a)nd.edu <mailto:[email protected]>> wrote: >>>> Hi all, >>>> >>>> I've been asking this question around various MPI boards to try and >>>> get a consensus before I decide to rewrite some MPI code. I am >>>> grateful for any advice that you can give. >>>> >>>> I've inherited a MPI code that was written ~8-10 years ago and it >>>> predominately uses MPI persistent communication routines for data >>>> transfers e.g. MPI_SEND_INIT, MPI_RECV_INIT, MPI_START etc. (which I >>>> am not familiar with and don't normally hear much discussion about). >>>> I was just wondering if using persistent communication calls is still >>>> regarded as the most efficient/scalable way to perform communication >>>> when the communication pattern is known and fixed amongst >>>> neighborhood processes? We regularly run the code across an IB >>>> network so would there be a benefit to rewrite the code using another >>>> approach (e.g. MPI one-sided communication) or should I leave it as >>>> it is? The code currently scales up to 10K cores and I want to push >>>> it even further and thus was wondering if there is any benefit in >>>> tinkering with this persistent MPI communication approach? >>>> >>>> Thanks in advance for any advice. >>>> >>>> Tim. >>>> >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org <mailto:[email protected]> >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>> >>> >>> >>> -- >>> Jeff Hammond >>> Argonne Leadership Computing Facility >>> University of Chicago Computation Institute >>> jhammond(a)alcf.anl.gov <mailto:[email protected]> / (630) 252-5381 >>> http://www.linkedin.com/in/jeffhammond >>> https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] Persistent Communication using MPI_SEND_INIT, MPI_RECV_INIT etc.
by Pavan Balaji 26 Mar '13

26 Mar '13

Tim, It's hard for us to decide which MPI functionality is better for you without knowing your algorithmic model. For some algorithms persistent send/recv is great. For some, RMA is great. Of course, each one has its shortcomings. Please don't jump to one or the other based on a short email discussion :-). Also note that some MPI implementations do optimize persistent communication, particularly for memory registration. So even if you don't see a benefit on some platforms, it doesn't mean that other MPI implementations cannot take advantage of it. -- Pavan On 03/26/2013 11:36 AM US Central Time, Timothy Stitt wrote: > Thanks for the quick reply Jeff. That information is valuable. I'll > follow up on your pointers. > > Much appreciated, > > Tim. > > On Mar 26, 2013, at 12:32 PM, Jeff Hammond <jhammond(a)alcf.anl.gov > <mailto:[email protected]>> wrote: > >> You might want to look at neighborhood collectives, which are >> discussed in Chapter 7 of MPI-3. This is a new feature so it may not >> be implemented in all MPI implementations, but MPICH supports it. I >> guess MVAPICH will support it soon enough if not already. >> >> When persistent MPI send/recv is discussed at the MPI Forum, it is >> often described as an inadequate solution because it does not specify >> a full channel and thus some important optimizations, e.g. for RDMA, >> may not be feasible. >> >> If you can use MPI-3 RMA, that is probably going to be a good idea, >> although high-quality support for RMA varies. MPICH-derived >> implementations usually do a good job though. >> >> Best, >> >> Jeff >> >> On Tue, Mar 26, 2013 at 11:25 AM, Timothy Stitt >> <Timothy.Stitt.9(a)nd.edu <mailto:[email protected]>> wrote: >>> Hi all, >>> >>> I've been asking this question around various MPI boards to try and >>> get a consensus before I decide to rewrite some MPI code. I am >>> grateful for any advice that you can give. >>> >>> I've inherited a MPI code that was written ~8-10 years ago and it >>> predominately uses MPI persistent communication routines for data >>> transfers e.g. MPI_SEND_INIT, MPI_RECV_INIT, MPI_START etc. (which I >>> am not familiar with and don't normally hear much discussion about). >>> I was just wondering if using persistent communication calls is still >>> regarded as the most efficient/scalable way to perform communication >>> when the communication pattern is known and fixed amongst >>> neighborhood processes? We regularly run the code across an IB >>> network so would there be a benefit to rewrite the code using another >>> approach (e.g. MPI one-sided communication) or should I leave it as >>> it is? The code currently scales up to 10K cores and I want to push >>> it even further and thus was wondering if there is any benefit in >>> tinkering with this persistent MPI communication approach? >>> >>> Thanks in advance for any advice. >>> >>> Tim. >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org <mailto:[email protected]> >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> >> >> -- >> Jeff Hammond >> Argonne Leadership Computing Facility >> University of Chicago Computation Institute >> jhammond(a)alcf.anl.gov <mailto:[email protected]> / (630) 252-5381 >> http://www.linkedin.com/in/jeffhammond >> https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > *Tim Stitt* PhD > User Support Manager (CRC) > Research Assistant Professor (Computer Science & Engineering) > Room 108, Center for Research Computing | University of Notre Dame | > P.O. Box 539, Notre Dame, IN 46556 | Phone: 574-631-5287 | Email: > tstitt(a)nd.edu <mailto:[email protected]> > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] Persistent Communication using MPI_SEND_INIT, MPI_RECV_INIT etc.
by Timothy Stitt 26 Mar '13

26 Mar '13

Thanks for the quick reply Jeff. That information is valuable. I'll follow up on your pointers. Much appreciated, Tim. On Mar 26, 2013, at 12:32 PM, Jeff Hammond <jhammond(a)alcf.anl.gov<mailto:[email protected]>> wrote: You might want to look at neighborhood collectives, which are discussed in Chapter 7 of MPI-3. This is a new feature so it may not be implemented in all MPI implementations, but MPICH supports it. I guess MVAPICH will support it soon enough if not already. When persistent MPI send/recv is discussed at the MPI Forum, it is often described as an inadequate solution because it does not specify a full channel and thus some important optimizations, e.g. for RDMA, may not be feasible. If you can use MPI-3 RMA, that is probably going to be a good idea, although high-quality support for RMA varies. MPICH-derived implementations usually do a good job though. Best, Jeff On Tue, Mar 26, 2013 at 11:25 AM, Timothy Stitt <Timothy.Stitt.9(a)nd.edu<mailto:[email protected]>> wrote: Hi all, I've been asking this question around various MPI boards to try and get a consensus before I decide to rewrite some MPI code. I am grateful for any advice that you can give. I've inherited a MPI code that was written ~8-10 years ago and it predominately uses MPI persistent communication routines for data transfers e.g. MPI_SEND_INIT, MPI_RECV_INIT, MPI_START etc. (which I am not familiar with and don't normally hear much discussion about). I was just wondering if using persistent communication calls is still regarded as the most efficient/scalable way to perform communication when the communication pattern is known and fixed amongst neighborhood processes? We regularly run the code across an IB network so would there be a benefit to rewrite the code using another approach (e.g. MPI one-sided communication) or should I leave it as it is? The code currently scales up to 10K cores and I want to push it even further and thus was wondering if there is any benefit in tinkering with this persistent MPI communication approach? Thanks in advance for any advice. Tim. _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov<mailto:[email protected]> / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss Tim Stitt PhD User Support Manager (CRC) Research Assistant Professor (Computer Science & Engineering) Room 108, Center for Research Computing | University of Notre Dame | P.O. Box 539, Notre Dame, IN 46556 | Phone: 574-631-5287 | Email: tstitt(a)nd.edu<mailto:[email protected]>

1 0

Re: [mpich-discuss] Persistent Communication using MPI_SEND_INIT, MPI_RECV_INIT etc.
by Jeff Hammond 26 Mar '13

26 Mar '13

You might want to look at neighborhood collectives, which are discussed in Chapter 7 of MPI-3. This is a new feature so it may not be implemented in all MPI implementations, but MPICH supports it. I guess MVAPICH will support it soon enough if not already. When persistent MPI send/recv is discussed at the MPI Forum, it is often described as an inadequate solution because it does not specify a full channel and thus some important optimizations, e.g. for RDMA, may not be feasible. If you can use MPI-3 RMA, that is probably going to be a good idea, although high-quality support for RMA varies. MPICH-derived implementations usually do a good job though. Best, Jeff On Tue, Mar 26, 2013 at 11:25 AM, Timothy Stitt <Timothy.Stitt.9(a)nd.edu> wrote: > Hi all, > > I've been asking this question around various MPI boards to try and get a consensus before I decide to rewrite some MPI code. I am grateful for any advice that you can give. > > I've inherited a MPI code that was written ~8-10 years ago and it predominately uses MPI persistent communication routines for data transfers e.g. MPI_SEND_INIT, MPI_RECV_INIT, MPI_START etc. (which I am not familiar with and don't normally hear much discussion about). I was just wondering if using persistent communication calls is still regarded as the most efficient/scalable way to perform communication when the communication pattern is known and fixed amongst neighborhood processes? We regularly run the code across an IB network so would there be a benefit to rewrite the code using another approach (e.g. MPI one-sided communication) or should I leave it as it is? The code currently scales up to 10K cores and I want to push it even further and thus was wondering if there is any benefit in tinkering with this persistent MPI communication approach? > > Thanks in advance for any advice. > > Tim. > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Persistent Communication using MPI_SEND_INIT, MPI_RECV_INIT etc.
by Timothy Stitt 26 Mar '13

26 Mar '13

Hi all, I've been asking this question around various MPI boards to try and get a consensus before I decide to rewrite some MPI code. I am grateful for any advice that you can give. I've inherited a MPI code that was written ~8-10 years ago and it predominately uses MPI persistent communication routines for data transfers e.g. MPI_SEND_INIT, MPI_RECV_INIT, MPI_START etc. (which I am not familiar with and don't normally hear much discussion about). I was just wondering if using persistent communication calls is still regarded as the most efficient/scalable way to perform communication when the communication pattern is known and fixed amongst neighborhood processes? We regularly run the code across an IB network so would there be a benefit to rewrite the code using another approach (e.g. MPI one-sided communication) or should I leave it as it is? The code currently scales up to 10K cores and I want to push it even further and thus was wondering if there is any benefit in tinkering with this persistent MPI communication approach? Thanks in advance for any advice. Tim.

1 0

Re: [mpich-discuss] MPI_Comm_Spawn causing zombies of hydra_pmi_proxy
by Silvan Brändli 26 Mar '13

26 Mar '13

Just for the record: I guess my zombie problem was caused by ignoring return codes unequal zero by using the workaround as described in http://lists.mpich.org/pipermail/discuss/2013-February/000429.html . A dirty fix which allows to ignore non-zero return codes and get rid of zombies (caused by spawning processes) is to add something like int waitstatus, child_pid; while ((child_pid = waitpid(-1, &waitstatus, WNOHANG)) > 0) { printf("HYDRA: child of pid %d finished\n",child_pid); } at the beginning of function HYDU_create_process in src/pm/hydra/utils/launch/launch.c . Best regards Silvan On 21.03.2013 12:31, Silvan Brï¿½ndli wrote: > PS: Just some additional information on compilers: so far I used icpc, > when changing it to gcc-4.7.2 I still get the zombies. The only change > in the linked libraries is an additional > libdl.so.2 => /lib64/libdl.so.2 > > Any hint / idea? > Best regards > Silvan > > On 19.03.2013 10:47, Silvan Brï¿½ndli wrote: >> Dear Pavan, >> >> in the attached example I use mpich3, but still I get the zombies. Is >> there something wrong with >> - my MPI function calls? (disconnect, finalize) >> - my linked libraries? (see below) >> >> Thanks in advance! >> Best regards >> Silvan >> >> ldd_hi >> linux-vdso.so.1 (0x00007fffa01ff000) >> libmpichcxx.so.10 => /opt/mpich3/lib/libmpichcxx.so.10 >> (0x00007fdfec0e4000) >> libmpich.so.10 => /opt/mpich3/lib/libmpich.so.10 >> (0x00007fdfebc6f000) >> libopa.so.1 => /opt/mpich3/lib/libopa.so.1 (0x00007fdfeba6d000) >> libmpl.so.1 => /opt/mpich3/lib/libmpl.so.1 (0x00007fdfeb868000) >> libaio.so.1 => /lib64/libaio.so.1 (0x00007fdfeb666000) >> libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fdfeb449000) >> libm.so.6 => /lib64/libm.so.6 (0x00007fdfeb14e000) >> libstdc++.so.6 => >> /usr/lib/gcc/x86_64-pc-linux-gnu/4.7.2/libstdc++.so.6 >> (0x00007fdfeae47000) >> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fdfeac30000) >> libc.so.6 => /lib64/libc.so.6 (0x00007fdfea882000) >> libdl.so.2 => /lib64/libdl.so.2 (0x00007fdfea67e000) >> libgfortran.so.3 => >> /usr/lib/gcc/x86_64-pc-linux-gnu/4.7.2/libgfortran.so.3 >> (0x00007fdfea361000) >> libquadmath.so.0 => >> /usr/lib/gcc/x86_64-pc-linux-gnu/4.7.2/libquadmath.so.0 >> (0x00007fdfea12b000) >> /lib64/ld-linux-x86-64.so.2 (0x00007fdfec308000) >> librt.so.1 => /lib64/librt.so.1 (0x00007fdfe9f23000) >> ldd_main >> linux-vdso.so.1 (0x00007fffeb1ff000) >> libmpichcxx.so.10 => /opt/mpich3/lib/libmpichcxx.so.10 >> (0x00007f6fc797e000) >> libmpich.so.10 => /opt/mpich3/lib/libmpich.so.10 >> (0x00007f6fc7509000) >> libopa.so.1 => /opt/mpich3/lib/libopa.so.1 (0x00007f6fc7307000) >> libmpl.so.1 => /opt/mpich3/lib/libmpl.so.1 (0x00007f6fc7102000) >> libaio.so.1 => /lib64/libaio.so.1 (0x00007f6fc6f00000) >> libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f6fc6ce3000) >> libm.so.6 => /lib64/libm.so.6 (0x00007f6fc69e8000) >> libstdc++.so.6 => >> /usr/lib/gcc/x86_64-pc-linux-gnu/4.7.2/libstdc++.so.6 >> (0x00007f6fc66e1000) >> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f6fc64ca000) >> libc.so.6 => /lib64/libc.so.6 (0x00007f6fc611c000) >> libdl.so.2 => /lib64/libdl.so.2 (0x00007f6fc5f18000) >> libgfortran.so.3 => >> /usr/lib/gcc/x86_64-pc-linux-gnu/4.7.2/libgfortran.so.3 >> (0x00007f6fc5bfb000) >> libquadmath.so.0 => >> /usr/lib/gcc/x86_64-pc-linux-gnu/4.7.2/libquadmath.so.0 >> (0x00007f6fc59c5000) >> /lib64/ld-linux-x86-64.so.2 (0x00007f6fc7ba2000) >> librt.so.1 => /lib64/librt.so.1 (0x00007f6fc57bd000) >> >> On 07.03.2013 17:23, Pavan Balaji wrote: >>> >>> I just tried this with the mpich master and it seems to work correctly, >>> and there are no zombie processes (though I reduced the number of >>> iterations to 10000, instead of 200000). This was a problem in mpich >>> once upon a time, but that was a few years ago. Are you using the >>> latest version of mpich (3.0.2)? >>> >>> -- Pavan >>> >>> On 03/07/2013 09:44 AM US Central Time, Silvan Brï¿½ndli wrote: >>>> PS: The attached programs are a simplification of my code. They >>>> reproduce the zombie problem. Waiting for the 32k zombies takes a >>>> while... but I expect the same behaviour as with my original code. >>>> >>>> Am I missing something when finishing the called program? I just use >>>> MPI_Comm_disconnect and MPI_Finalize. >>>> >>>> Best regards >>>> Silvan >>>> >>>> main.cpp >>>> >>>> #include <mpi.h>; >>>> >>>> int main(int argc, char *argv[]) >>>> { >>>> int myrank; >>>> int spawnerror; >>>> int value = 123; >>>> void *buf = &value; >>>> MPI_Comm child_comm; >>>> >>>> if (MPI_Init(&argc,&argv)!=MPI_SUCCESS) >>>> { >>>> printf("MPI_Init failed"); >>>> } >>>> >>>> MPI_Comm_rank(MPI_COMM_WORLD, &myrank); >>>> >>>> char* hiargv[] = {"23",NULL}; >>>> for(int i = 1; i <= 200000; i++) >>>> { >>>> value = i; >>>> printf("Main before spawn %d\n",i); >>>> MPI_Comm_spawn("./hi",hiargv, 1, MPI_INFO_NULL, myrank, >>>> MPI_COMM_SELF, &child_comm, &spawnerror); >>>> MPI_Send(buf, 1, MPI_INTEGER, 0, 1, child_comm); >>>> MPI_Comm_disconnect(&child_comm); >>>> } >>>> >>>> MPI_Finalize(); >>>> return 0; >>>> } >>>> >>>> hi.cpp: >>>> >>>> #include <mpi.h>; >>>> >>>> int main(int argc, char** argv) { >>>> MPI_Comm parent; >>>> MPI_Status status; >>>> int err; >>>> int value = -1; >>>> void* buf= &value; >>>> >>>> if (MPI_Init(&argc,&argv)!=MPI_SUCCESS) >>>> { >>>> printf("MPI_Init failed"); >>>> } >>>> MPI_Comm_get_parent(&parent); >>>> if (parent == MPI_COMM_NULL) printf("No parent!"); >>>> >>>> MPI_Recv(buf, 1, MPI_INTEGER, 0, MPI_ANY_TAG, parent, &status); >>>> MPI_Comm_disconnect(&parent); >>>> err = MPI_Finalize(); >>>> printf("hi finalized %d %d \n",err, value); >>>> return 0; >>>> } >>>> >>>> >>>> >>>> On 07.03.2013 12:38, Silvan Brï¿½ndli wrote: >>>>> Dear all, >>>>> >>>>> again I have a question related to spawning processes. I understand >>>>> the >>>>> situation as follows: >>>>> >>>>> My program A spawns program B. Program B spawns program C1, C2 ... >>>>> C10000 ... >>>>> Program Cx terminates correctly before Cx+1 is called, however >>>>> returning >>>>> 1 to mpiexec. To handle this I use the workaround as described in >>>>> http://lists.mpich.org/pipermail/discuss/2013-February/000429.html >>>>> >>>>> Now it looks like with every Spawn a "hydra_pmi_proxy" is started, the >>>>> calling program is mpiexec. When the program Cx is finished this >>>>> "hydra_pmi_proxy" remains as a zombie until the programs A, B and >>>>> mpiexec are finished. When approx. 32k of those "hydra_pmi_proxy" >>>>> exist >>>>> I get some problems (too many processes or something similar). >>>>> >>>>> What can I do to finish "hydra_pmi_proxy" while my programs A, B and >>>>> mpiexec are still running? >>>>> >>>>> I'm glad about every hint. >>>>> >>>>> Best regards >>>>> Silvan >>>>> >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> >>> >> >> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > -- Dipl.-Ing. Silvan Brï¿½ndli Numerische Strukturanalyse mit Anwendungen in der Schiffstechnik (M-10) Technische Universitï¿½t Hamburg-Harburg Schwarzenbergstraï¿½e 95c 21073 Hamburg Tel. : +49 (0)40 42878 - 6187 Fax. : +49 (0)40 42878 - 6090 e-mail: silvan.braendli(a)tuhh.de www : http://www.tuhh.de/skf 5th GACM Colloquium on Computational Mechanics http://www.tu-harburg.de/gacm2013

1 0

Re: [mpich-discuss] merging intracommuncators
by Rajeev Thakur 25 Mar '13

25 Mar '13

Try doing the second spawn with intracomm instead of MPI_COMM_WORLD as the communicator. However, it would be collective over the first set of children as well, i.e., both the parent and the first set of children would need to call the second spawn in this case. Rajeev On Mar 25, 2013, at 5:13 PM, Ali Bouaricha wrote: > Hello, > > I'm using MPI_Comm_Spawn to spawn processes dynamically. I spawn > few processes, then I execute something, then I spawn few more processes. > I want to merge intracommunicators in such way that the master > Processes including all the spawn children processes (the first and > Second time) would have rank > 0, 1, 2, 3, .... p-1. > > My problem is that I cannot merge intracommunicators. Here is what I mean: > > Parent code: > =========== > > error = MPI_Comm_spawn("child", MPI_ARGV_NULL, 2, > info, 0, MPI_COMM_WORLD, &intercomm, errcodes); > > MPI_Intercomm_merge(intercomm, 0, &intracomm); > ..... > > do something > > .... > error = MPI_Comm_spawn("child", MPI_ARGV_NULL, 3, > info, 0, MPI_COMM_WORLD, &intercomm, errcodes); > > MPI_Intercomm_merge(intercomm, 0, &intracomm); > > > > The child code: > =============== > > MPI_Comm_get_parent(&intercomm); > > MPI_Comm_size(intercomm,&nprocs); > printf(" child nprocs = %d\n", nprocs); > > MPI_Intercomm_merge(intercomm, 0, &intracomm); > > MPI_Comm_rank(intracomm, &rank); > printf(" child rank = %d\n", rank); > > ===== > > The outptut after executing: mpiexec -n 1 ./parent > > > Gives: child rank 1, 2 (for the first spawn) > Child rank 1, 2, 3 (for the second spawn) > > The question is that how I can merge so that my processes would have ranks > 1, 2, 3, 4, 5? > > Thanks a lot, > > Ali > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

merging intracommuncators
by Ali Bouaricha 25 Mar '13

25 Mar '13

Hello, I'm using MPI_Comm_Spawn to spawn processes dynamically. I spawn few processes, then I execute something, then I spawn few more processes. I want to merge intracommunicators in such way that the master Processes including all the spawn children processes (the first and Second time) would have rank 0, 1, 2, 3, .... p-1. My problem is that I cannot merge intracommunicators. Here is what I mean: Parent code: =========== error = MPI_Comm_spawn("child", MPI_ARGV_NULL, 2, info, 0, MPI_COMM_WORLD, &intercomm, errcodes); MPI_Intercomm_merge(intercomm, 0, &intracomm); ..... do something .... error = MPI_Comm_spawn("child", MPI_ARGV_NULL, 3, info, 0, MPI_COMM_WORLD, &intercomm, errcodes); MPI_Intercomm_merge(intercomm, 0, &intracomm); The child code: =============== MPI_Comm_get_parent(&intercomm); MPI_Comm_size(intercomm,&nprocs); printf(" child nprocs = %d\n", nprocs); MPI_Intercomm_merge(intercomm, 0, &intracomm); MPI_Comm_rank(intracomm, &rank); printf(" child rank = %d\n", rank); ===== The outptut after executing: mpiexec -n 1 ./parent Gives: child rank 1, 2 (for the first spawn) Child rank 1, 2, 3 (for the second spawn) The question is that how I can merge so that my processes would have ranks 1, 2, 3, 4, 5? Thanks a lot, Ali

1 0

Re: [mpich-discuss] Problem in the step of "make"
by Eric A. Borisch 22 Mar '13

22 Mar '13

On Fri, Mar 22, 2013 at 2:48 PM, yuanp <peng.yuan(a)student.ecp.fr> wrote: > Yeah, I thought it was successful and did a make install and setted the > PATH. But when I ran some tests I met some problems and I thought it was > due to the step make. > I copied the code from my terminal > > dhcp-27-100:mpich-install holmes_conan$ ssh othermachine date > ssh: Could not resolve hostname othermachine: nodename nor servname > provided, or not known > dhcp-27-100:mpich-install holmes_conan$ rsh othermachine date > rcmd: getaddrinfo: nodename nor servname provided, or not known > This *looks* like examples of testing connectivity: 'ssh othermachine date' should try to execute 'date' on the machine named 'othermachine'; you likely need to replace othermachine with the name of the other machine you're connecting to; or localhost if you're just using one machine (ie testing that sshd is running & accepting connections, likely trying to make sure it's working passwordless.) The two commands above and their responses have nothing to do with mpich's installation state. -- Eric

1 0