March 2013 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Fault tolerance after MPI_Comm_connect/accept
by Pavan Balaji 05 Mar '13

05 Mar '13

On 03/05/2013 10:34 AM US Central Time, Jim Dinan wrote: > You should ignore my comments on MPICH FT. My info is clearly > out-of-date. It sounds like what you're looking for should be fully > supported. :) Well, almost :-). Some things could not be done cleanly while staying within MPI-3. For example, when you do a wildcard receive, it will always return an error if any process in the communicator is dead. The MPI Forum is working on fixing this by allowing the user to "opt in" for this wildcard stuff. I'm a few months behind on the MPI-3.1 fault-tolerance proposal, but I believe this is still present. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] Fault tolerance after MPI_Comm_connect/accept
by Jim Dinan 05 Mar '13

05 Mar '13

Hi Mathieu, You should ignore my comments on MPICH FT. My info is clearly out-of-date. It sounds like what you're looking for should be fully supported. :) ~Jim. On 3/5/13 9:42 AM, Matthieu Dorier wrote: > Alright, thanks for the answer (and for the ticket). > Cheers, > > Matthieu > > ----- Mail original ----- >> De: "Jim Dinan" <dinan(a)mcs.anl.gov> >> ï¿½: discuss(a)mpich.org >> Envoyï¿½: Mardi 5 Mars 2013 16:33:31 >> Objet: Re: [mpich-discuss] Fault tolerance after MPI_Comm_connect/accept >> >> Hi Mathieu, >> >> I created an MPI Forum ticket for this: >> >> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/365 >> >> In terms of what is guaranteed by the standard, the behavior is >> undefined. In terms of that MPICH will do, I am not sure, although >> my >> guess is that current MPICH will be unable to continue working after >> such a failure. You may need to do some testing or read the code to >> find out. >> >> Cheers, >> ~Jim. >> >> On 3/4/13 7:56 AM, Matthieu Dorier wrote: >>> Hi, >>> >>> I'm connecting two MPI applications A and B using MPI_Comm_accept >>> in A >>> and MPI_Comm_connect in B. I would like to know a bit more about >>> the >>> behavior in case one application stops (say B): will a >>> communication >>> attempt (e.g. MPI_Send) from a process from A to a process from B >>> crash? >>> return an error? block? >>> Is there a way for application A to notice that B has stopped in >>> order >>> to avoid communicating with it? >>> >>> Thanks. >>> >>> PS: by the way for whom is involved in the MPI forum, an >>> MPI_Comm_iaccept in the MPI3 standard would have been useful. >>> Something >>> to keep in mind for the next version maybe ;) >>> >>> Matthieu Dorier >>> PhD student at ENS Cachan Brittany and IRISA >>> http://people.irisa.fr/Matthieu.Dorier >>> >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] Fault tolerance after MPI_Comm_connect/accept
by Pavan Balaji 05 Mar '13

05 Mar '13

On 03/05/2013 09:33 AM US Central Time, Jim Dinan wrote: > I created an MPI Forum ticket for this: > > https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/365 Jim: iaccept was a part of Josh's original MPI-3 proposal, but was voted down by the Forum because it's collective over the communicator (though that's true for all nonblocking collectives). I, myself, pointed out several use cases for iaccept (one of which was VOCL, where we have to use a separate thread for comm_accept). It might be worthwhile revisiting it for MPI-3.1 or MPI-4. > In terms of what is guaranteed by the standard, the behavior is > undefined. In terms of that MPICH will do, I am not sure, although my > guess is that current MPICH will be unable to continue working after > such a failure. You may need to do some testing or read the code to > find out. FWIW, the default behavior in mpich is to clean up all processes if something goes wrong. However, you can pass -disable-auto-cleanup to mpiexec to disable this behavior. In this case, if one process dies, the remaining processes are left alone, and will return an error on communication -- you'll obviously need to check the return value of the MPI functions to detect such failures. One small gotcha is the interaction of such failures with MPI-3 nonblocking collectives. Currently, we don't have a way to get these two working together, though that's planned to be fixed in mpich-3.1 to be released in 2014 (preview releases will show up later this year). You can use the environment MPICH_ENABLE_COLL_FT_RET=1 to tell MPICH to not hang, and return an error, if a process fails during a collective operation. But doing this will cause nonblocking collectives to break. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] Fault tolerance after MPI_Comm_connect/accept
by Matthieu Dorier 05 Mar '13

05 Mar '13

Alright, thanks for the answer (and for the ticket). Cheers, Matthieu ----- Mail original ----- > De: "Jim Dinan" <dinan(a)mcs.anl.gov> > À: discuss(a)mpich.org > Envoyé: Mardi 5 Mars 2013 16:33:31 > Objet: Re: [mpich-discuss] Fault tolerance after MPI_Comm_connect/accept > > Hi Mathieu, > > I created an MPI Forum ticket for this: > > https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/365 > > In terms of what is guaranteed by the standard, the behavior is > undefined. In terms of that MPICH will do, I am not sure, although > my > guess is that current MPICH will be unable to continue working after > such a failure. You may need to do some testing or read the code to > find out. > > Cheers, > ~Jim. > > On 3/4/13 7:56 AM, Matthieu Dorier wrote: > > Hi, > > > > I'm connecting two MPI applications A and B using MPI_Comm_accept > > in A > > and MPI_Comm_connect in B. I would like to know a bit more about > > the > > behavior in case one application stops (say B): will a > > communication > > attempt (e.g. MPI_Send) from a process from A to a process from B > > crash? > > return an error? block? > > Is there a way for application A to notice that B has stopped in > > order > > to avoid communicating with it? > > > > Thanks. > > > > PS: by the way for whom is involved in the MPI forum, an > > MPI_Comm_iaccept in the MPI3 standard would have been useful. > > Something > > to keep in mind for the next version maybe ;) > > > > Matthieu Dorier > > PhD student at ENS Cachan Brittany and IRISA > > http://people.irisa.fr/Matthieu.Dorier > > > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] Fault tolerance after MPI_Comm_connect/accept
by Jim Dinan 05 Mar '13

05 Mar '13

Hi Mathieu, I created an MPI Forum ticket for this: https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/365 In terms of what is guaranteed by the standard, the behavior is undefined. In terms of that MPICH will do, I am not sure, although my guess is that current MPICH will be unable to continue working after such a failure. You may need to do some testing or read the code to find out. Cheers, ~Jim. On 3/4/13 7:56 AM, Matthieu Dorier wrote: > Hi, > > I'm connecting two MPI applications A and B using MPI_Comm_accept in A > and MPI_Comm_connect in B. I would like to know a bit more about the > behavior in case one application stops (say B): will a communication > attempt (e.g. MPI_Send) from a process from A to a process from B crash? > return an error? block? > Is there a way for application A to notice that B has stopped in order > to avoid communicating with it? > > Thanks. > > PS: by the way for whom is involved in the MPI forum, an > MPI_Comm_iaccept in the MPI3 standard would have been useful. Something > to keep in mind for the next version maybe ;) > > Matthieu Dorier > PhD student at ENS Cachan Brittany and IRISA > http://people.irisa.fr/Matthieu.Dorier > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] Maximum number of communicators
by Dave Goodell 04 Mar '13

04 Mar '13

Nick, Re: the linked thread: Nobody should be using MPICH 1.2.7p1 (old MPICH1, not new MPICH >=v3.0). Drawing conclusions about MPI-on-BG/Q behavior from that ancient software is a very bad idea. I would be happy to discuss communicators with you in person, if that would help. The way these limits get hit can be complicated sometimes. Basically, a given process can only be a member of ~2048 communicators simultaneously. Furthermore, this number can go way down depending on which context IDs are free on other processes. With that said, however, the most common case where applications run out of context IDs is still usually a communicator leak of some sort. -Dave On Mar 4, 2013, at 2:03 PM CST, Nichols A. Romero wrote: > Dave et al., > > I am still conversing with the application developer about this issue. It looks like I jumped to the wrong conclusion based on the mpiP data that I collected. > > I won't ask you to read this whole, post. But if you can read the last post by fgygi here: > http://fpmd.ucdavis.edu/qbox-list/viewtopic.php?f=3&t=219&start=10 > > I have asked him to try to create a reduced test case based on MPI only. The code is written on top of ScaLAPACK, there is a for loop that creates a communicator, then destroy it based on its rank on the 2D processor grid. Ultimately, when the loop is over, MPI_Comm_create/Comm_free can be called about 1000 times, but only one of those communicators will continue existing. > > > ----- Original Message ----- >> From: "Nichols A. Romero" <naromero(a)alcf.anl.gov> >> To: discuss(a)mpich.org >> Sent: Monday, February 25, 2013 2:17:04 PM >> Subject: Re: [mpich-discuss] Maximum number of communicators >> >> I cannot reproduce this in a simple test code. I am convinced that >> this is an application error. I linked the application against mpiP >> and I see that the calls to Comm_create >> Comm_free by two to three >> orders of magnitude. >> >> The application is template C++ and I suspect that the C++ class >> destructor is not properly calling Comm_free. >> >> ----- Original Message ----- >>> From: "Dave Goodell" <goodell(a)mcs.anl.gov> >>> To: discuss(a)mpich.org >>> Sent: Sunday, February 24, 2013 3:36:54 PM >>> Subject: Re: [mpich-discuss] Maximum number of communicators >>> >>> Test code indeed would be helpful. >>> >>> You may get better warnings about context ID and/or communicator >>> leaks if you configure a stock MPICH with "--enable-g=all". >>> >>> -Dave >>> >>> On Feb 23, 2013, at 7:05 PM CST, Jeff Hammond wrote: >>> >>>> It's 2048 in all MPICH2-derived implementations that I know >>>> about, >>>> including the ones used on BGP and BGQ. >>>> >>>> Since I know what your issue is already, I think the problem has >>>> something to do with context id leaking. You should post your >>>> test >>>> code. >>>> >>>> Jeff >>>> >>>> On Sat, Feb 23, 2013 at 6:24 PM, Nichols A. Romero >>>> <naromero(a)alcf.anl.gov> wrote: >>>>> Hi, >>>>> >>>>> What is the maximum number of communicators in the MPICH that is >>>>> used on BG/P vs. that used on BG/Q? >>>>> >>>>> Sent from my iPhone >>>>> _______________________________________________ >>>>> discuss mailing list discuss(a)mpich.org >>>>> To manage subscription options or unsubscribe: >>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> >>>> >>>> >>>> -- >>>> Jeff Hammond >>>> Argonne Leadership Computing Facility >>>> University of Chicago Computation Institute >>>> jhammond(a)alcf.anl.gov / (630) 252-5381 >>>> http://www.linkedin.com/in/jeffhammond >>>> https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >> >> -- >> Nichols A. Romero, Ph.D. >> Argonne Leadership Computing Facility >> Argonne National Laboratory >> Building 240 Room 2-127 >> 9700 South Cass Avenue >> Argonne, IL 60490 >> (630) 252-3441 >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > -- > Nichols A. Romero, Ph.D. > Argonne Leadership Computing Facility > Argonne National Laboratory > Building 240 Room 2-127 > 9700 South Cass Avenue > Argonne, IL 60490 > (630) 252-3441 > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Maximum number of communicators
by Nichols A. Romero 04 Mar '13

04 Mar '13

Dave et al., I am still conversing with the application developer about this issue. It looks like I jumped to the wrong conclusion based on the mpiP data that I collected. I won't ask you to read this whole, post. But if you can read the last post by fgygi here: http://fpmd.ucdavis.edu/qbox-list/viewtopic.php?f=3&t=219&start=10 I have asked him to try to create a reduced test case based on MPI only. The code is written on top of ScaLAPACK, there is a for loop that creates a communicator, then destroy it based on its rank on the 2D processor grid. Ultimately, when the loop is over, MPI_Comm_create/Comm_free can be called about 1000 times, but only one of those communicators will continue existing. ----- Original Message ----- > From: "Nichols A. Romero" <naromero(a)alcf.anl.gov> > To: discuss(a)mpich.org > Sent: Monday, February 25, 2013 2:17:04 PM > Subject: Re: [mpich-discuss] Maximum number of communicators > > I cannot reproduce this in a simple test code. I am convinced that > this is an application error. I linked the application against mpiP > and I see that the calls to Comm_create >> Comm_free by two to three > orders of magnitude. > > The application is template C++ and I suspect that the C++ class > destructor is not properly calling Comm_free. > > ----- Original Message ----- > > From: "Dave Goodell" <goodell(a)mcs.anl.gov> > > To: discuss(a)mpich.org > > Sent: Sunday, February 24, 2013 3:36:54 PM > > Subject: Re: [mpich-discuss] Maximum number of communicators > > > > Test code indeed would be helpful. > > > > You may get better warnings about context ID and/or communicator > > leaks if you configure a stock MPICH with "--enable-g=all". > > > > -Dave > > > > On Feb 23, 2013, at 7:05 PM CST, Jeff Hammond wrote: > > > > > It's 2048 in all MPICH2-derived implementations that I know > > > about, > > > including the ones used on BGP and BGQ. > > > > > > Since I know what your issue is already, I think the problem has > > > something to do with context id leaking. You should post your > > > test > > > code. > > > > > > Jeff > > > > > > On Sat, Feb 23, 2013 at 6:24 PM, Nichols A. Romero > > > <naromero(a)alcf.anl.gov> wrote: > > >> Hi, > > >> > > >> What is the maximum number of communicators in the MPICH that is > > >> used on BG/P vs. that used on BG/Q? > > >> > > >> Sent from my iPhone > > >> _______________________________________________ > > >> discuss mailing list discuss(a)mpich.org > > >> To manage subscription options or unsubscribe: > > >> https://lists.mpich.org/mailman/listinfo/discuss > > > > > > > > > > > > -- > > > Jeff Hammond > > > Argonne Leadership Computing Facility > > > University of Chicago Computation Institute > > > jhammond(a)alcf.anl.gov / (630) 252-5381 > > > http://www.linkedin.com/in/jeffhammond > > > https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond > > > _______________________________________________ > > > discuss mailing list discuss(a)mpich.org > > > To manage subscription options or unsubscribe: > > > https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > > -- > Nichols A. Romero, Ph.D. > Argonne Leadership Computing Facility > Argonne National Laboratory > Building 240 Room 2-127 > 9700 South Cass Avenue > Argonne, IL 60490 > (630) 252-3441 > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Nichols A. Romero, Ph.D. Argonne Leadership Computing Facility Argonne National Laboratory Building 240 Room 2-127 9700 South Cass Avenue Argonne, IL 60490 (630) 252-3441

1 0

Re: [mpich-discuss] Hydra launching on CPU+MIC?
by Pavan Balaji 04 Mar '13

04 Mar '13

On 03/04/2013 09:06 AM US Central Time, Jeff Hammond wrote: > And when I run in this mode, I will be using Nemesis shm within the > host and within the MIC while using SCIF between them, right? Sorry, I didn't answer this part of the question, the first time around. It'll use shared memory on the host and on the MIC. Between the two, the default communication is tcp. You can build mpich with --with-device=ch3:nemesis:scif to use scif. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] Hydra launching on CPU+MIC?
by Jeff Hammond 04 Mar '13

04 Mar '13

That's exactly the type of RTM I was looking for :-) I'm spoiled by Hydra awesomeness on production systems where PBS integration and the like make it unnecessary for me to learn such things. Jeff On Mon, Mar 4, 2013 at 10:47 AM, Dave Goodell <goodell(a)mcs.anl.gov> wrote: > Can't you do something like this? > > ----8<---- > host-hostname:16 > mic-hostname:60 > ----8<---- > > http://wiki.mpich.org/mpich/index.php/Using_the_Hydra_Process_Manager > > -Dave > > On Mar 4, 2013, at 10:34 AM CST, Jeff Hammond wrote: > >> Hmm, I was hoping to not have to manually create a hostfile that >> contained 16 copies of the host IP and 60 copies of the MIC IP. >> >> Jeff >> >> On Mon, Mar 4, 2013 at 10:16 AM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: >>> >>> Just give the IP address of the MIC card as one of the hosts. >>> >>> -- Pavan >>> >>> On 03/04/2013 09:06 AM US Central Time, Jeff Hammond wrote: >>>> Is there documentation somewhere for how to launch Nh MPI processes on >>>> the host and Nm MPI processes on the MIC using Hydra? TACC told me >>>> that Intel MPI's Hydra supports this so I guess MPICH's does as well >>>> but I'm not sure how to do it. >>>> >>>> I only care about launching on a single box right now, not a cluster, >>>> so interactions with multiple nodes, IB, etc. don't matter to me. >>>> >>>> And when I run in this mode, I will be using Nemesis shm within the >>>> host and within the MIC while using SCIF between them, right? >>>> >>>> Thanks, >>>> >>>> Jeff >>>> >>> >>> -- >>> Pavan Balaji >>> http://www.mcs.anl.gov/~balaji >> >> >> >> -- >> Jeff Hammond >> Argonne Leadership Computing Facility >> University of Chicago Computation Institute >> jhammond(a)alcf.anl.gov / (630) 252-5381 >> http://www.linkedin.com/in/jeffhammond >> https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Re: [mpich-discuss] Hydra launching on CPU+MIC?
by Dave Goodell 04 Mar '13

04 Mar '13

Can't you do something like this? ----8<---- host-hostname:16 mic-hostname:60 ----8<---- http://wiki.mpich.org/mpich/index.php/Using_the_Hydra_Process_Manager -Dave On Mar 4, 2013, at 10:34 AM CST, Jeff Hammond wrote: > Hmm, I was hoping to not have to manually create a hostfile that > contained 16 copies of the host IP and 60 copies of the MIC IP. > > Jeff > > On Mon, Mar 4, 2013 at 10:16 AM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: >> >> Just give the IP address of the MIC card as one of the hosts. >> >> -- Pavan >> >> On 03/04/2013 09:06 AM US Central Time, Jeff Hammond wrote: >>> Is there documentation somewhere for how to launch Nh MPI processes on >>> the host and Nm MPI processes on the MIC using Hydra? TACC told me >>> that Intel MPI's Hydra supports this so I guess MPICH's does as well >>> but I'm not sure how to do it. >>> >>> I only care about launching on a single box right now, not a cluster, >>> so interactions with multiple nodes, IB, etc. don't matter to me. >>> >>> And when I run in this mode, I will be using Nemesis shm within the >>> host and within the MIC while using SCIF between them, right? >>> >>> Thanks, >>> >>> Jeff >>> >> >> -- >> Pavan Balaji >> http://www.mcs.anl.gov/~balaji > > > > -- > Jeff Hammond > Argonne Leadership Computing Facility > University of Chicago Computation Institute > jhammond(a)alcf.anl.gov / (630) 252-5381 > http://www.linkedin.com/in/jeffhammond > https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0