- discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Fault tolerance after MPI_Comm_connect/accept
by Matthieu Dorier 05 Mar '13

05 Mar '13

Alright, thanks for the answer (and for the ticket). Cheers, Matthieu ----- Mail original ----- > De: "Jim Dinan" <dinan(a)mcs.anl.gov> > À: discuss(a)mpich.org > Envoyé: Mardi 5 Mars 2013 16:33:31 > Objet: Re: [mpich-discuss] Fault tolerance after MPI_Comm_connect/accept > > Hi Mathieu, > > I created an MPI Forum ticket for this: > > https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/365 > > In terms of what is guaranteed by the standard, the behavior is > undefined. In terms of that MPICH will do, I am not sure, although > my > guess is that current MPICH will be unable to continue working after > such a failure. You may need to do some testing or read the code to > find out. > > Cheers, > ~Jim. > > On 3/4/13 7:56 AM, Matthieu Dorier wrote: > > Hi, > > > > I'm connecting two MPI applications A and B using MPI_Comm_accept > > in A > > and MPI_Comm_connect in B. I would like to know a bit more about > > the > > behavior in case one application stops (say B): will a > > communication > > attempt (e.g. MPI_Send) from a process from A to a process from B > > crash? > > return an error? block? > > Is there a way for application A to notice that B has stopped in > > order > > to avoid communicating with it? > > > > Thanks. > > > > PS: by the way for whom is involved in the MPI forum, an > > MPI_Comm_iaccept in the MPI3 standard would have been useful. > > Something > > to keep in mind for the next version maybe ;) > > > > Matthieu Dorier > > PhD student at ENS Cachan Brittany and IRISA > > http://people.irisa.fr/Matthieu.Dorier > > > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] Fault tolerance after MPI_Comm_connect/accept
by Jim Dinan 05 Mar '13

05 Mar '13

Hi Mathieu, I created an MPI Forum ticket for this: https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/365 In terms of what is guaranteed by the standard, the behavior is undefined. In terms of that MPICH will do, I am not sure, although my guess is that current MPICH will be unable to continue working after such a failure. You may need to do some testing or read the code to find out. Cheers, ~Jim. On 3/4/13 7:56 AM, Matthieu Dorier wrote: > Hi, > > I'm connecting two MPI applications A and B using MPI_Comm_accept in A > and MPI_Comm_connect in B. I would like to know a bit more about the > behavior in case one application stops (say B): will a communication > attempt (e.g. MPI_Send) from a process from A to a process from B crash? > return an error? block? > Is there a way for application A to notice that B has stopped in order > to avoid communicating with it? > > Thanks. > > PS: by the way for whom is involved in the MPI forum, an > MPI_Comm_iaccept in the MPI3 standard would have been useful. Something > to keep in mind for the next version maybe ;) > > Matthieu Dorier > PhD student at ENS Cachan Brittany and IRISA > http://people.irisa.fr/Matthieu.Dorier > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] Maximum number of communicators
by Dave Goodell 04 Mar '13

04 Mar '13

Nick, Re: the linked thread: Nobody should be using MPICH 1.2.7p1 (old MPICH1, not new MPICH >=v3.0). Drawing conclusions about MPI-on-BG/Q behavior from that ancient software is a very bad idea. I would be happy to discuss communicators with you in person, if that would help. The way these limits get hit can be complicated sometimes. Basically, a given process can only be a member of ~2048 communicators simultaneously. Furthermore, this number can go way down depending on which context IDs are free on other processes. With that said, however, the most common case where applications run out of context IDs is still usually a communicator leak of some sort. -Dave On Mar 4, 2013, at 2:03 PM CST, Nichols A. Romero wrote: > Dave et al., > > I am still conversing with the application developer about this issue. It looks like I jumped to the wrong conclusion based on the mpiP data that I collected. > > I won't ask you to read this whole, post. But if you can read the last post by fgygi here: > http://fpmd.ucdavis.edu/qbox-list/viewtopic.php?f=3&t=219&start=10 > > I have asked him to try to create a reduced test case based on MPI only. The code is written on top of ScaLAPACK, there is a for loop that creates a communicator, then destroy it based on its rank on the 2D processor grid. Ultimately, when the loop is over, MPI_Comm_create/Comm_free can be called about 1000 times, but only one of those communicators will continue existing. > > > ----- Original Message ----- >> From: "Nichols A. Romero" <naromero(a)alcf.anl.gov> >> To: discuss(a)mpich.org >> Sent: Monday, February 25, 2013 2:17:04 PM >> Subject: Re: [mpich-discuss] Maximum number of communicators >> >> I cannot reproduce this in a simple test code. I am convinced that >> this is an application error. I linked the application against mpiP >> and I see that the calls to Comm_create >> Comm_free by two to three >> orders of magnitude. >> >> The application is template C++ and I suspect that the C++ class >> destructor is not properly calling Comm_free. >> >> ----- Original Message ----- >>> From: "Dave Goodell" <goodell(a)mcs.anl.gov> >>> To: discuss(a)mpich.org >>> Sent: Sunday, February 24, 2013 3:36:54 PM >>> Subject: Re: [mpich-discuss] Maximum number of communicators >>> >>> Test code indeed would be helpful. >>> >>> You may get better warnings about context ID and/or communicator >>> leaks if you configure a stock MPICH with "--enable-g=all". >>> >>> -Dave >>> >>> On Feb 23, 2013, at 7:05 PM CST, Jeff Hammond wrote: >>> >>>> It's 2048 in all MPICH2-derived implementations that I know >>>> about, >>>> including the ones used on BGP and BGQ. >>>> >>>> Since I know what your issue is already, I think the problem has >>>> something to do with context id leaking. You should post your >>>> test >>>> code. >>>> >>>> Jeff >>>> >>>> On Sat, Feb 23, 2013 at 6:24 PM, Nichols A. Romero >>>> <naromero(a)alcf.anl.gov> wrote: >>>>> Hi, >>>>> >>>>> What is the maximum number of communicators in the MPICH that is >>>>> used on BG/P vs. that used on BG/Q? >>>>> >>>>> Sent from my iPhone >>>>> _______________________________________________ >>>>> discuss mailing list discuss(a)mpich.org >>>>> To manage subscription options or unsubscribe: >>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> >>>> >>>> >>>> -- >>>> Jeff Hammond >>>> Argonne Leadership Computing Facility >>>> University of Chicago Computation Institute >>>> jhammond(a)alcf.anl.gov / (630) 252-5381 >>>> http://www.linkedin.com/in/jeffhammond >>>> https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >> >> -- >> Nichols A. Romero, Ph.D. >> Argonne Leadership Computing Facility >> Argonne National Laboratory >> Building 240 Room 2-127 >> 9700 South Cass Avenue >> Argonne, IL 60490 >> (630) 252-3441 >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > -- > Nichols A. Romero, Ph.D. > Argonne Leadership Computing Facility > Argonne National Laboratory > Building 240 Room 2-127 > 9700 South Cass Avenue > Argonne, IL 60490 > (630) 252-3441 > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Maximum number of communicators
by Nichols A. Romero 04 Mar '13

04 Mar '13

Dave et al., I am still conversing with the application developer about this issue. It looks like I jumped to the wrong conclusion based on the mpiP data that I collected. I won't ask you to read this whole, post. But if you can read the last post by fgygi here: http://fpmd.ucdavis.edu/qbox-list/viewtopic.php?f=3&t=219&start=10 I have asked him to try to create a reduced test case based on MPI only. The code is written on top of ScaLAPACK, there is a for loop that creates a communicator, then destroy it based on its rank on the 2D processor grid. Ultimately, when the loop is over, MPI_Comm_create/Comm_free can be called about 1000 times, but only one of those communicators will continue existing. ----- Original Message ----- > From: "Nichols A. Romero" <naromero(a)alcf.anl.gov> > To: discuss(a)mpich.org > Sent: Monday, February 25, 2013 2:17:04 PM > Subject: Re: [mpich-discuss] Maximum number of communicators > > I cannot reproduce this in a simple test code. I am convinced that > this is an application error. I linked the application against mpiP > and I see that the calls to Comm_create >> Comm_free by two to three > orders of magnitude. > > The application is template C++ and I suspect that the C++ class > destructor is not properly calling Comm_free. > > ----- Original Message ----- > > From: "Dave Goodell" <goodell(a)mcs.anl.gov> > > To: discuss(a)mpich.org > > Sent: Sunday, February 24, 2013 3:36:54 PM > > Subject: Re: [mpich-discuss] Maximum number of communicators > > > > Test code indeed would be helpful. > > > > You may get better warnings about context ID and/or communicator > > leaks if you configure a stock MPICH with "--enable-g=all". > > > > -Dave > > > > On Feb 23, 2013, at 7:05 PM CST, Jeff Hammond wrote: > > > > > It's 2048 in all MPICH2-derived implementations that I know > > > about, > > > including the ones used on BGP and BGQ. > > > > > > Since I know what your issue is already, I think the problem has > > > something to do with context id leaking. You should post your > > > test > > > code. > > > > > > Jeff > > > > > > On Sat, Feb 23, 2013 at 6:24 PM, Nichols A. Romero > > > <naromero(a)alcf.anl.gov> wrote: > > >> Hi, > > >> > > >> What is the maximum number of communicators in the MPICH that is > > >> used on BG/P vs. that used on BG/Q? > > >> > > >> Sent from my iPhone > > >> _______________________________________________ > > >> discuss mailing list discuss(a)mpich.org > > >> To manage subscription options or unsubscribe: > > >> https://lists.mpich.org/mailman/listinfo/discuss > > > > > > > > > > > > -- > > > Jeff Hammond > > > Argonne Leadership Computing Facility > > > University of Chicago Computation Institute > > > jhammond(a)alcf.anl.gov / (630) 252-5381 > > > http://www.linkedin.com/in/jeffhammond > > > https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond > > > _______________________________________________ > > > discuss mailing list discuss(a)mpich.org > > > To manage subscription options or unsubscribe: > > > https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > > -- > Nichols A. Romero, Ph.D. > Argonne Leadership Computing Facility > Argonne National Laboratory > Building 240 Room 2-127 > 9700 South Cass Avenue > Argonne, IL 60490 > (630) 252-3441 > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Nichols A. Romero, Ph.D. Argonne Leadership Computing Facility Argonne National Laboratory Building 240 Room 2-127 9700 South Cass Avenue Argonne, IL 60490 (630) 252-3441

1 0

Re: [mpich-discuss] Hydra launching on CPU+MIC?
by Pavan Balaji 04 Mar '13

04 Mar '13

On 03/04/2013 09:06 AM US Central Time, Jeff Hammond wrote: > And when I run in this mode, I will be using Nemesis shm within the > host and within the MIC while using SCIF between them, right? Sorry, I didn't answer this part of the question, the first time around. It'll use shared memory on the host and on the MIC. Between the two, the default communication is tcp. You can build mpich with --with-device=ch3:nemesis:scif to use scif. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] Hydra launching on CPU+MIC?
by Jeff Hammond 04 Mar '13

04 Mar '13

That's exactly the type of RTM I was looking for :-) I'm spoiled by Hydra awesomeness on production systems where PBS integration and the like make it unnecessary for me to learn such things. Jeff On Mon, Mar 4, 2013 at 10:47 AM, Dave Goodell <goodell(a)mcs.anl.gov> wrote: > Can't you do something like this? > > ----8<---- > host-hostname:16 > mic-hostname:60 > ----8<---- > > http://wiki.mpich.org/mpich/index.php/Using_the_Hydra_Process_Manager > > -Dave > > On Mar 4, 2013, at 10:34 AM CST, Jeff Hammond wrote: > >> Hmm, I was hoping to not have to manually create a hostfile that >> contained 16 copies of the host IP and 60 copies of the MIC IP. >> >> Jeff >> >> On Mon, Mar 4, 2013 at 10:16 AM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: >>> >>> Just give the IP address of the MIC card as one of the hosts. >>> >>> -- Pavan >>> >>> On 03/04/2013 09:06 AM US Central Time, Jeff Hammond wrote: >>>> Is there documentation somewhere for how to launch Nh MPI processes on >>>> the host and Nm MPI processes on the MIC using Hydra? TACC told me >>>> that Intel MPI's Hydra supports this so I guess MPICH's does as well >>>> but I'm not sure how to do it. >>>> >>>> I only care about launching on a single box right now, not a cluster, >>>> so interactions with multiple nodes, IB, etc. don't matter to me. >>>> >>>> And when I run in this mode, I will be using Nemesis shm within the >>>> host and within the MIC while using SCIF between them, right? >>>> >>>> Thanks, >>>> >>>> Jeff >>>> >>> >>> -- >>> Pavan Balaji >>> http://www.mcs.anl.gov/~balaji >> >> >> >> -- >> Jeff Hammond >> Argonne Leadership Computing Facility >> University of Chicago Computation Institute >> jhammond(a)alcf.anl.gov / (630) 252-5381 >> http://www.linkedin.com/in/jeffhammond >> https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Re: [mpich-discuss] Hydra launching on CPU+MIC?
by Dave Goodell 04 Mar '13

04 Mar '13

Can't you do something like this? ----8<---- host-hostname:16 mic-hostname:60 ----8<---- http://wiki.mpich.org/mpich/index.php/Using_the_Hydra_Process_Manager -Dave On Mar 4, 2013, at 10:34 AM CST, Jeff Hammond wrote: > Hmm, I was hoping to not have to manually create a hostfile that > contained 16 copies of the host IP and 60 copies of the MIC IP. > > Jeff > > On Mon, Mar 4, 2013 at 10:16 AM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: >> >> Just give the IP address of the MIC card as one of the hosts. >> >> -- Pavan >> >> On 03/04/2013 09:06 AM US Central Time, Jeff Hammond wrote: >>> Is there documentation somewhere for how to launch Nh MPI processes on >>> the host and Nm MPI processes on the MIC using Hydra? TACC told me >>> that Intel MPI's Hydra supports this so I guess MPICH's does as well >>> but I'm not sure how to do it. >>> >>> I only care about launching on a single box right now, not a cluster, >>> so interactions with multiple nodes, IB, etc. don't matter to me. >>> >>> And when I run in this mode, I will be using Nemesis shm within the >>> host and within the MIC while using SCIF between them, right? >>> >>> Thanks, >>> >>> Jeff >>> >> >> -- >> Pavan Balaji >> http://www.mcs.anl.gov/~balaji > > > > -- > Jeff Hammond > Argonne Leadership Computing Facility > University of Chicago Computation Institute > jhammond(a)alcf.anl.gov / (630) 252-5381 > http://www.linkedin.com/in/jeffhammond > https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Hydra launching on CPU+MIC?
by Jeff Hammond 04 Mar '13

04 Mar '13

Hmm, I was hoping to not have to manually create a hostfile that contained 16 copies of the host IP and 60 copies of the MIC IP. Jeff On Mon, Mar 4, 2013 at 10:16 AM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > > Just give the IP address of the MIC card as one of the hosts. > > -- Pavan > > On 03/04/2013 09:06 AM US Central Time, Jeff Hammond wrote: >> Is there documentation somewhere for how to launch Nh MPI processes on >> the host and Nm MPI processes on the MIC using Hydra? TACC told me >> that Intel MPI's Hydra supports this so I guess MPICH's does as well >> but I'm not sure how to do it. >> >> I only care about launching on a single box right now, not a cluster, >> so interactions with multiple nodes, IB, etc. don't matter to me. >> >> And when I run in this mode, I will be using Nemesis shm within the >> host and within the MIC while using SCIF between them, right? >> >> Thanks, >> >> Jeff >> > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Re: [mpich-discuss] Hydra launching on CPU+MIC?
by Pavan Balaji 04 Mar '13

04 Mar '13

Just give the IP address of the MIC card as one of the hosts. -- Pavan On 03/04/2013 09:06 AM US Central Time, Jeff Hammond wrote: > Is there documentation somewhere for how to launch Nh MPI processes on > the host and Nm MPI processes on the MIC using Hydra? TACC told me > that Intel MPI's Hydra supports this so I guess MPICH's does as well > but I'm not sure how to do it. > > I only care about launching on a single box right now, not a cluster, > so interactions with multiple nodes, IB, etc. don't matter to me. > > And when I run in this mode, I will be using Nemesis shm within the > host and within the MIC while using SCIF between them, right? > > Thanks, > > Jeff > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Hydra launching on CPU+MIC?
by Jeff Hammond 04 Mar '13

04 Mar '13

Is there documentation somewhere for how to launch Nh MPI processes on the host and Nm MPI processes on the MIC using Hydra? TACC told me that Intel MPI's Hydra supports this so I guess MPICH's does as well but I'm not sure how to do it. I only care about launching on a single box right now, not a cluster, so interactions with multiple nodes, IB, etc. don't matter to me. And when I run in this mode, I will be using Nemesis shm within the host and within the MIC while using SCIF between them, right? Thanks, Jeff -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0