Correct, there is no MPI-level query mechanism without the ULFM extensions. Ken On 08/31/2017 11:49 AM, Zaak Beekman wrote:

Ken,

Thanks so much for your concise response!

Is my understanding correct that: If one MPI rank, or all the ranks on a given node are to fail, from, say, a hardware issue, there is no means of determining that this has happened unless the MPI library supports the user level failure mitigation (ULFM) features? So if one were to pass --disable-auto-cleanup when the runtime didn't have support for ULFM, and processes on remote MPI ranks died, there'd be no way to query them and call MPI_abort from a different, still running rank?

Thanks again, Zaak

On Thu, Aug 31, 2017 at 12:19 PM <[email protected] <mailto:[email protected]>> wrote:

Send discuss mailing list submissions to [email protected] <mailto:[email protected]>

To subscribe or unsubscribe via the World Wide Web, visit https://lists.mpich.org/mailman/listinfo/discuss or, via email, send a message with subject or body 'help' to [email protected] <mailto:[email protected]>

You can reach the person managing the list at [email protected] <mailto:[email protected]>

When replying, please edit your Subject line so it is more specific than "Re: Contents of discuss digest..."

Today's Topics:

1. question about -disable-auto-cleanup (Zaak Beekman) 2. Re: Torque MPICH jobs stuck (Souparno Adhikary) 3. Re: question about -disable-auto-cleanup (Kenneth Raffenetti)

----------------------------------------------------------------------

Message: 1 Date: Wed, 30 Aug 2017 17:29:11 +0000 From: Zaak Beekman <[email protected] <mailto:[email protected]>> To: [email protected] <mailto:[email protected]> Subject: [mpich-discuss] question about -disable-auto-cleanup Message-ID:

<CAAbnBwb8L2VtXqsbbGjZvSOHpTkd7mK6SeQuhHKTEG0brOPSBQ@mail.gmail.com <mailto:CAAbnBwb8L2VtXqsbbGjZvSOHpTkd7mK6SeQuhHKTEG0brOPSBQ@mail.gmail.com>> Content-Type: text/plain; charset="utf-8"

OK, since there were no responses here to my previous email, perhaps a better question would be:

What is a good resource to learn about the impact of passing `--disable-auto-cleanup` at runtime?

Some google searches bring up discussions of what appear to be bugs in the standard and/or implementation, but I'm not sure where to look to find out about even the intended runtime semantics.

Any and all help pointing me in the right direction would be much appreciated.

Thanks, Zaak

On Wed, Aug 30, 2017 at 1:00 PM <[email protected] <mailto:[email protected]>> wrote:

> Send discuss mailing list submissions to > [email protected] <mailto:[email protected]> > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.mpich.org/mailman/listinfo/discuss > or, via email, send a message with subject or body 'help' to > [email protected] <mailto:[email protected]> > > You can reach the person managing the list at > [email protected] <mailto:[email protected]> > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of discuss digest..." > > > Today's Topics: > > 1. question about -disable-auto-cleanup (Zaak Beekman) > 2. Torque MPICH jobs stuck (Souparno Adhikary) > 3. Re: Torque MPICH jobs stuck (Halim Amer) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 29 Aug 2017 21:22:49 +0000 > From: Zaak Beekman <[email protected] <mailto:[email protected]>> > To: [email protected] <mailto:[email protected]> > Subject: [mpich-discuss] question about -disable-auto-cleanup > Message-ID: > < > CAAbnBwZrQ03YmmmayhcHEywh8bEFMZ_AycBydOqZFB023KeJZQ@mail.gmail.com <mailto:CAAbnBwZrQ03YmmmayhcHEywh8bEFMZ_AycBydOqZFB023KeJZQ@mail.gmail.com>> > Content-Type: text/plain; charset="utf-8" > > I know that --disable-auto-cleanup is required to enable the fault-tolerant > MPI features, but are there downsides to passing this? Performance > implications? > > I ask, because over at https://github.com/sourceryinstitute/OpenCoarrays > we've > implemented much of the Fortran 2015 failed images feature on top of MPICH > and other MPI implementations. But to use this, --disable-auto-cleanup must > be passed to mpiexec. We provide wrapper scripts to try to abstract the > back end (GASNet, MPI, OpenSHMEM etc.) in the form of a Fortran compiler > wrapper, and an executable launcher. So I'm wondering, since failed images > are part of the standard (2015) would it be dumb if we always pass > --disable-auto-cleanup to mpiexec and only turn off support when explicitly > asked for by the user, or is it safer/more performant to default to > requiring the user to pass an additional flag to our wrapper script that > results in --disable-auto-cleanup getting passed to mpiexec? > > Feedback would be much appreciated. Feel free to post responses at > https://github.com/sourceryinstitute/OpenCoarrays/issues/401 as well.. > > Thanks, > Zaak >