February 2014 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] turning off MPI abort messages
by Jeff Hammond 21 Feb '14

21 Feb '14

Barry: Can you tolerate the following workaround for Hydra's error cleanup or do you need it to be internal? I presume you know enough bash to generalize a.sh appropriately. alcfwl181:build jhammond$ cat a.sh #!/bin/sh $1 true alcfwl181:build jhammond$ mpiexec -n 1 -env MPIR_CVAR_SUPPRESS_ABORT_MESSAGE 0 ./a.sh ./a.out application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 alcfwl181:build jhammond$ mpiexec -n 1 -env MPIR_CVAR_SUPPRESS_ABORT_MESSAGE 1 ./a.sh ./a.out alcfwl181:build jhammond$ mpiexec -n 1 -env MPIR_CVAR_SUPPRESS_ABORT_MESSAGE 1 ./a.out =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 61123 RUNNING AT alcfwl181.alcf.anl.gov = EXIT CODE: 1 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== alcfwl181:build jhammond$ mpiexec -n 1 -env MPIR_CVAR_SUPPRESS_ABORT_MESSAGE 0 ./a.out application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 61126 RUNNING AT alcfwl181.alcf.anl.gov = EXIT CODE: 1 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== On Fri, Feb 21, 2014 at 3:10 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > Barry: > > Would the following behavior be acceptable to you? I have only made > the changes in MPI but am looking at the process manager now. > > Jeff > > > # Without the process manager > > alcfwl181:build jhammond$ export MPIR_CVAR_SUPPRESS_ABORT_MESSAGE=0 > alcfwl181:build jhammond$ ./a.out > application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 > alcfwl181:build jhammond$ export MPIR_CVAR_SUPPRESS_ABORT_MESSAGE=1 > alcfwl181:build jhammond$ ./a.out > > alcfwl181:build jhammond$ unset MPIR_CVAR_SUPPRESS_ABORT_MESSAGE > alcfwl181:build jhammond$ ./a.out > application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 > > # With the process manager > > alcfwl181:build jhammond$ mpiexec -n 1 -env > MPIR_CVAR_SUPPRESS_ABORT_MESSAGE 0 ./a.out > application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 > > =================================================================================== > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > = PID 61023 RUNNING AT alcfwl181.alcf.anl.gov > = EXIT CODE: 1 > = CLEANING UP REMAINING PROCESSES > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > =================================================================================== > alcfwl181:build jhammond$ mpiexec -n 1 -env > MPIR_CVAR_SUPPRESS_ABORT_MESSAGE 1 ./a.out > > > =================================================================================== > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > = PID 61026 RUNNING AT alcfwl181.alcf.anl.gov > = EXIT CODE: 1 > = CLEANING UP REMAINING PROCESSES > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > =================================================================================== > alcfwl181:build jhammond$ mpiexec -n 1 ./a.out > application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 > > =================================================================================== > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > = PID 61032 RUNNING AT alcfwl181.alcf.anl.gov > = EXIT CODE: 1 > = CLEANING UP REMAINING PROCESSES > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > =================================================================================== > > > > On Thu, Feb 20, 2014 at 11:33 AM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: >> >> Is there any way to turn off MPICH (and others) printing messages about MPI_Abort? We have already prepared and presented useful error messages to the user about the situation and would like to avoid having these additional messages printed (that often make the situation look worse than it is) >> >> Thanks >> >> Barry >> >> application called MPI_Abort(MPI_COMM_WORLD, 56) - process 0 >> [cli_0]: aborting job: >> application called MPI_Abort(MPI_COMM_WORLD, 56) - process 0 >> >> ==================================================================mailto:[email protected]================= >> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >> = EXIT CODE: 56 >> = CLEANING UP REMAINING PROCESSES >> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >> =================================================================================== >> >> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > > > -- > Jeff Hammond > jeff.science(a)gmail.com -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [mpich-discuss] turning off MPI abort messages
by Jeff Hammond 21 Feb '14

21 Feb '14

Barry: Would the following behavior be acceptable to you? I have only made the changes in MPI but am looking at the process manager now. Jeff # Without the process manager alcfwl181:build jhammond$ export MPIR_CVAR_SUPPRESS_ABORT_MESSAGE=0 alcfwl181:build jhammond$ ./a.out application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 alcfwl181:build jhammond$ export MPIR_CVAR_SUPPRESS_ABORT_MESSAGE=1 alcfwl181:build jhammond$ ./a.out alcfwl181:build jhammond$ unset MPIR_CVAR_SUPPRESS_ABORT_MESSAGE alcfwl181:build jhammond$ ./a.out application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 # With the process manager alcfwl181:build jhammond$ mpiexec -n 1 -env MPIR_CVAR_SUPPRESS_ABORT_MESSAGE 0 ./a.out application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 61023 RUNNING AT alcfwl181.alcf.anl.gov = EXIT CODE: 1 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== alcfwl181:build jhammond$ mpiexec -n 1 -env MPIR_CVAR_SUPPRESS_ABORT_MESSAGE 1 ./a.out =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 61026 RUNNING AT alcfwl181.alcf.anl.gov = EXIT CODE: 1 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== alcfwl181:build jhammond$ mpiexec -n 1 ./a.out application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 61032 RUNNING AT alcfwl181.alcf.anl.gov = EXIT CODE: 1 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== On Thu, Feb 20, 2014 at 11:33 AM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > Is there any way to turn off MPICH (and others) printing messages about MPI_Abort? We have already prepared and presented useful error messages to the user about the situation and would like to avoid having these additional messages printed (that often make the situation look worse than it is) > > Thanks > > Barry > > application called MPI_Abort(MPI_COMM_WORLD, 56) - process 0 > [cli_0]: aborting job: > application called MPI_Abort(MPI_COMM_WORLD, 56) - process 0 > > ==================================================================mailto:[email protected]================= > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > = EXIT CODE: 56 > = CLEANING UP REMAINING PROCESSES > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > =================================================================================== > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [mpich-discuss] turning off MPI abort messages
by Jeff Hammond 21 Feb '14

21 Feb '14

I have already implemented the CVAR logic to suppress the output in MPI_Abort so the necessary convincing is only as to whether or not MPICH folks accepts this patch. I'm not sure whether I will bother to do anything to Hydra. Seems like one could wrap the binary in a "nofail" script (we use these on BG a lot because of how the control system responds to error codes) to keep Hydra from complaining. However, this is shell magic and clearly I don't know anything about shell because I think nonzero error codes in MPI programs imply that ls crashes when looking for missing files. Jeff On Fri, Feb 21, 2014 at 1:28 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: > Barry, > > Do you need a portable solution that works across different MPI > implementations, or does a solution for making just MPICH silent address > your need? If the latter, you could probably convince someone on the MPICH > team to add an environment variable for MPICH and a command-line "quiet" > flag for hydra/mpiexec. > > ~Jim. > > > On Fri, Feb 21, 2014 at 2:13 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: >> >> >> Understood. But I would like to eliminate both sets of error messages >> and still provide a useful “return code”. Perhaps compile time options to >> the library? >> >> Barry >> >> On Feb 21, 2014, at 12:40 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: >> >> > A little more detail -- you're actually getting messages from two >> > sources: (1) the MPICH library ("application called MPI_Abort...") and (2) >> > the job launcher ("BAD TERMINATION..."). You can eliminate the messages >> > from the job launcher by providing an error code of 0 in MPI_Abort. >> > >> > ~Jim. >> > >> > >> > >> > >> > On Fri, Feb 21, 2014 at 1:19 PM, Jeff Hammond <jeff.science(a)gmail.com> >> > wrote: >> > >> Just configure MPICH such that snprintf isn't discovered by configure >> > >> and you won't see these messages. >> > >> >> > >> The other solution is to fix PETSc so that people can't crash it so >> > >> easily ;-) >> > > >> > > Here we go again. It is not CRASHING; it has detected an error >> > > conditioning and trying to appropriately and cleanly terminate. The reason >> > > it needs to use MPI_Abort() is that often detecting error conditions is not >> > > a uniformly collective thing. >> > > >> > > Printing a suitable error message and ending is not crashing. But >> > > with all the badly formatted “error messages” printed by MPICH I can not >> > > control at the end it looks like it is crashing. >> > >> > You're returning a non-zero exit code, which I consider crashing. I >> > apologize if this definition disagrees with yours. If this is just >> > gentle cleanup, why not exit with code=0 as Jim suggested already? >> > >> > Jeff >> > >> > >> On Thu, Feb 20, 2014 at 3:19 PM, Jim Dinan <james.dinan(a)gmail.com> >> > >> wrote: >> > >>> If you can find a way to call MPI_Finalize instead, you will >> > >>> portably >> > >>> eliminate these messages. >> > >>> >> > >>> A lesser solution would be to provide an error code of 0 (or >> > >>> MPI_SUCCESS) to >> > >>> MPI_Abort, e.g. MPI_Comm_abort(MPI_COMM_WORLD, MPI_SUCCESS). This >> > >>> would >> > >>> eliminate the error message that you are getting from the job >> > >>> launcher. >> > >>> MPICH could be modified to be quiet about the abort when the >> > >>> application >> > >>> aborts with an error code of MPI_SUCCESS. >> > >>> >> > >>> ~Jim. >> > >>> >> > >>> >> > >>> On Thu, Feb 20, 2014 at 12:33 PM, Barry Smith <bsmith(a)mcs.anl.gov> >> > >>> wrote: >> > >>>> >> > >>>> >> > >>>> Is there any way to turn off MPICH (and others) printing messages >> > >>>> about >> > >>>> MPI_Abort? We have already prepared and presented useful error >> > >>>> messages to >> > >>>> the user about the situation and would like to avoid having these >> > >>>> additional >> > >>>> messages printed (that often make the situation look worse than it >> > >>>> is) >> > >>>> >> > >>>> Thanks >> > >>>> >> > >>>> Barry >> > >>>> >> > >>>> application called MPI_Abort(MPI_COMM_WORLD, 56) - process 0 >> > >>>> [cli_0]: aborting job: >> > >>>> application called MPI_Abort(MPI_COMM_WORLD, 56) - process 0 >> > >>>> >> > >>>> >> > >>>> >> > >>>> ==================================================================mailto:[email protected]================= >> > >>>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >> > >>>> = EXIT CODE: 56 >> > >>>> = CLEANING UP REMAINING PROCESSES >> > >>>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >> > >>>> >> > >>>> >> > >>>> =================================================================================== >> > >>>> >> > >>>> >> > >>>> >> > >>>> >> > >>>> _______________________________________________ >> > >>>> discuss mailing list discuss(a)mpich.org >> > >>>> To manage subscription options or unsubscribe: >> > >>>> https://lists.mpich.org/mailman/listinfo/discuss >> > >>> >> > >>> >> > >>> >> > >>> _______________________________________________ >> > >>> discuss mailing list discuss(a)mpich.org >> > >>> To manage subscription options or unsubscribe: >> > >>> https://lists.mpich.org/mailman/listinfo/discuss >> > >> >> > >> >> > >> >> > >> -- >> > >> Jeff Hammond >> > >> jeff.science(a)gmail.com >> > >> _______________________________________________ >> > >> discuss mailing list discuss(a)mpich.org >> > >> To manage subscription options or unsubscribe: >> > >> https://lists.mpich.org/mailman/listinfo/discuss >> > > >> > > _______________________________________________ >> > > discuss mailing list discuss(a)mpich.org >> > > To manage subscription options or unsubscribe: >> > > https://lists.mpich.org/mailman/listinfo/discuss >> > >> > >> > >> > -- >> > Jeff Hammond >> > jeff.science(a)gmail.com >> > _______________________________________________ >> > discuss mailing list discuss(a)mpich.org >> > To manage subscription options or unsubscribe: >> > https://lists.mpich.org/mailman/listinfo/discuss >> > >> > _______________________________________________ >> > discuss mailing list discuss(a)mpich.org >> > To manage subscription options or unsubscribe: >> > https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [mpich-discuss] turning off MPI abort messages
by Jed Brown 21 Feb '14

21 Feb '14

Jim Dinan <james.dinan(a)gmail.com> writes: > Barry, > > Do you need a portable solution that works across different MPI > implementations, This is well outside the standard so I don't see this being viable. > or does a solution for making just MPICH silent address your need? If > the latter, you could probably convince someone on the MPICH team to > add an environment variable for MPICH and a command-line "quiet" flag > for hydra/mpiexec. Even better would be an MPIX_Set_error_verbosity() or MPIX_Abort_quietly().

1 0

Re: [mpich-discuss] turning off MPI abort messages
by Jim Dinan 21 Feb '14

21 Feb '14

Barry, Do you need a portable solution that works across different MPI implementations, or does a solution for making just MPICH silent address your need? If the latter, you could probably convince someone on the MPICH team to add an environment variable for MPICH and a command-line "quiet" flag for hydra/mpiexec. ~Jim. On Fri, Feb 21, 2014 at 2:13 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > Understood. But I would like to eliminate both sets of error messages > and still provide a useful "return code". Perhaps compile time options to > the library? > > Barry > > On Feb 21, 2014, at 12:40 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: > > > A little more detail -- you're actually getting messages from two > sources: (1) the MPICH library ("application called MPI_Abort...") and (2) > the job launcher ("BAD TERMINATION..."). You can eliminate the messages > from the job launcher by providing an error code of 0 in MPI_Abort. > > > > ~Jim. > > > > > > > > > > On Fri, Feb 21, 2014 at 1:19 PM, Jeff Hammond <jeff.science(a)gmail.com> > wrote: > > >> Just configure MPICH such that snprintf isn't discovered by configure > > >> and you won't see these messages. > > >> > > >> The other solution is to fix PETSc so that people can't crash it so > easily ;-) > > > > > > Here we go again. It is not CRASHING; it has detected an error > conditioning and trying to appropriately and cleanly terminate. The reason > it needs to use MPI_Abort() is that often detecting error conditions is not > a uniformly collective thing. > > > > > > Printing a suitable error message and ending is not crashing. But > with all the badly formatted "error messages" printed by MPICH I can not > control at the end it looks like it is crashing. > > > > You're returning a non-zero exit code, which I consider crashing. I > > apologize if this definition disagrees with yours. If this is just > > gentle cleanup, why not exit with code=0 as Jim suggested already? > > > > Jeff > > > > >> On Thu, Feb 20, 2014 at 3:19 PM, Jim Dinan <james.dinan(a)gmail.com> > wrote: > > >>> If you can find a way to call MPI_Finalize instead, you will portably > > >>> eliminate these messages. > > >>> > > >>> A lesser solution would be to provide an error code of 0 (or > MPI_SUCCESS) to > > >>> MPI_Abort, e.g. MPI_Comm_abort(MPI_COMM_WORLD, MPI_SUCCESS). This > would > > >>> eliminate the error message that you are getting from the job > launcher. > > >>> MPICH could be modified to be quiet about the abort when the > application > > >>> aborts with an error code of MPI_SUCCESS. > > >>> > > >>> ~Jim. > > >>> > > >>> > > >>> On Thu, Feb 20, 2014 at 12:33 PM, Barry Smith <bsmith(a)mcs.anl.gov> > wrote: > > >>>> > > >>>> > > >>>> Is there any way to turn off MPICH (and others) printing messages > about > > >>>> MPI_Abort? We have already prepared and presented useful error > messages to > > >>>> the user about the situation and would like to avoid having these > additional > > >>>> messages printed (that often make the situation look worse than it > is) > > >>>> > > >>>> Thanks > > >>>> > > >>>> Barry > > >>>> > > >>>> application called MPI_Abort(MPI_COMM_WORLD, 56) - process 0 > > >>>> [cli_0]: aborting job: > > >>>> application called MPI_Abort(MPI_COMM_WORLD, 56) - process 0 > > >>>> > > >>>> > > >>>> > ==================================================================mailto: > discuss(a)mpich.org================= > > >>>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > > >>>> = EXIT CODE: 56 > > >>>> = CLEANING UP REMAINING PROCESSES > > >>>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > > >>>> > > >>>> > =================================================================================== > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> _______________________________________________ > > >>>> discuss mailing list discuss(a)mpich.org > > >>>> To manage subscription options or unsubscribe: > > >>>> https://lists.mpich.org/mailman/listinfo/discuss > > >>> > > >>> > > >>> > > >>> _______________________________________________ > > >>> discuss mailing list discuss(a)mpich.org > > >>> To manage subscription options or unsubscribe: > > >>> https://lists.mpich.org/mailman/listinfo/discuss > > >> > > >> > > >> > > >> -- > > >> Jeff Hammond > > >> jeff.science(a)gmail.com > > >> _______________________________________________ > > >> discuss mailing list discuss(a)mpich.org > > >> To manage subscription options or unsubscribe: > > >> https://lists.mpich.org/mailman/listinfo/discuss > > > > > > _______________________________________________ > > > discuss mailing list discuss(a)mpich.org > > > To manage subscription options or unsubscribe: > > > https://lists.mpich.org/mailman/listinfo/discuss > > > > > > > > -- > > Jeff Hammond > > jeff.science(a)gmail.com > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] turning off MPI abort messages
by Jeff Hammond 21 Feb '14

21 Feb '14

Aron told me that I should have said "we look forward to your patch" earlier instead of causing you guys to go apoplectic over my different definition of the word crash. Jeff On Fri, Feb 21, 2014 at 1:13 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > Understood. But I would like to eliminate both sets of error messages and still provide a useful “return code”. Perhaps compile time options to the library? > > Barry > > On Feb 21, 2014, at 12:40 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: > >> A little more detail -- you're actually getting messages from two sources: (1) the MPICH library ("application called MPI_Abort...") and (2) the job launcher ("BAD TERMINATION..."). You can eliminate the messages from the job launcher by providing an error code of 0 in MPI_Abort. >> >> ~Jim. >> >> >> >> >> On Fri, Feb 21, 2014 at 1:19 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >> >> Just configure MPICH such that snprintf isn't discovered by configure >> >> and you won't see these messages. >> >> >> >> The other solution is to fix PETSc so that people can't crash it so easily ;-) >> > >> > Here we go again. It is not CRASHING; it has detected an error conditioning and trying to appropriately and cleanly terminate. The reason it needs to use MPI_Abort() is that often detecting error conditions is not a uniformly collective thing. >> > >> > Printing a suitable error message and ending is not crashing. But with all the badly formatted “error messages” printed by MPICH I can not control at the end it looks like it is crashing. >> >> You're returning a non-zero exit code, which I consider crashing. I >> apologize if this definition disagrees with yours. If this is just >> gentle cleanup, why not exit with code=0 as Jim suggested already? >> >> Jeff >> >> >> On Thu, Feb 20, 2014 at 3:19 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: >> >>> If you can find a way to call MPI_Finalize instead, you will portably >> >>> eliminate these messages. >> >>> >> >>> A lesser solution would be to provide an error code of 0 (or MPI_SUCCESS) to >> >>> MPI_Abort, e.g. MPI_Comm_abort(MPI_COMM_WORLD, MPI_SUCCESS). This would >> >>> eliminate the error message that you are getting from the job launcher. >> >>> MPICH could be modified to be quiet about the abort when the application >> >>> aborts with an error code of MPI_SUCCESS. >> >>> >> >>> ~Jim. >> >>> >> >>> >> >>> On Thu, Feb 20, 2014 at 12:33 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: >> >>>> >> >>>> >> >>>> Is there any way to turn off MPICH (and others) printing messages about >> >>>> MPI_Abort? We have already prepared and presented useful error messages to >> >>>> the user about the situation and would like to avoid having these additional >> >>>> messages printed (that often make the situation look worse than it is) >> >>>> >> >>>> Thanks >> >>>> >> >>>> Barry >> >>>> >> >>>> application called MPI_Abort(MPI_COMM_WORLD, 56) - process 0 >> >>>> [cli_0]: aborting job: >> >>>> application called MPI_Abort(MPI_COMM_WORLD, 56) - process 0 >> >>>> >> >>>> >> >>>> ==================================================================mailto:[email protected]================= >> >>>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >> >>>> = EXIT CODE: 56 >> >>>> = CLEANING UP REMAINING PROCESSES >> >>>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >> >>>> >> >>>> =================================================================================== >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> _______________________________________________ >> >>>> discuss mailing list discuss(a)mpich.org >> >>>> To manage subscription options or unsubscribe: >> >>>> https://lists.mpich.org/mailman/listinfo/discuss >> >>> >> >>> >> >>> >> >>> _______________________________________________ >> >>> discuss mailing list discuss(a)mpich.org >> >>> To manage subscription options or unsubscribe: >> >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> >> >> >> >> >> >> -- >> >> Jeff Hammond >> >> jeff.science(a)gmail.com >> >> _______________________________________________ >> >> discuss mailing list discuss(a)mpich.org >> >> To manage subscription options or unsubscribe: >> >> https://lists.mpich.org/mailman/listinfo/discuss >> > >> > _______________________________________________ >> > discuss mailing list discuss(a)mpich.org >> > To manage subscription options or unsubscribe: >> > https://lists.mpich.org/mailman/listinfo/discuss >> >> >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [mpich-discuss] turning off MPI abort messages
by Barry Smith 21 Feb '14

21 Feb '14

Understood. But I would like to eliminate both sets of error messages and still provide a useful “return code”. Perhaps compile time options to the library? Barry On Feb 21, 2014, at 12:40 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: > A little more detail -- you're actually getting messages from two sources: (1) the MPICH library ("application called MPI_Abort...") and (2) the job launcher ("BAD TERMINATION..."). You can eliminate the messages from the job launcher by providing an error code of 0 in MPI_Abort. > > ~Jim. > > > > > On Fri, Feb 21, 2014 at 1:19 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > >> Just configure MPICH such that snprintf isn't discovered by configure > >> and you won't see these messages. > >> > >> The other solution is to fix PETSc so that people can't crash it so easily ;-) > > > > Here we go again. It is not CRASHING; it has detected an error conditioning and trying to appropriately and cleanly terminate. The reason it needs to use MPI_Abort() is that often detecting error conditions is not a uniformly collective thing. > > > > Printing a suitable error message and ending is not crashing. But with all the badly formatted “error messages” printed by MPICH I can not control at the end it looks like it is crashing. > > You're returning a non-zero exit code, which I consider crashing. I > apologize if this definition disagrees with yours. If this is just > gentle cleanup, why not exit with code=0 as Jim suggested already? > > Jeff > > >> On Thu, Feb 20, 2014 at 3:19 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: > >>> If you can find a way to call MPI_Finalize instead, you will portably > >>> eliminate these messages. > >>> > >>> A lesser solution would be to provide an error code of 0 (or MPI_SUCCESS) to > >>> MPI_Abort, e.g. MPI_Comm_abort(MPI_COMM_WORLD, MPI_SUCCESS). This would > >>> eliminate the error message that you are getting from the job launcher. > >>> MPICH could be modified to be quiet about the abort when the application > >>> aborts with an error code of MPI_SUCCESS. > >>> > >>> ~Jim. > >>> > >>> > >>> On Thu, Feb 20, 2014 at 12:33 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > >>>> > >>>> > >>>> Is there any way to turn off MPICH (and others) printing messages about > >>>> MPI_Abort? We have already prepared and presented useful error messages to > >>>> the user about the situation and would like to avoid having these additional > >>>> messages printed (that often make the situation look worse than it is) > >>>> > >>>> Thanks > >>>> > >>>> Barry > >>>> > >>>> application called MPI_Abort(MPI_COMM_WORLD, 56) - process 0 > >>>> [cli_0]: aborting job: > >>>> application called MPI_Abort(MPI_COMM_WORLD, 56) - process 0 > >>>> > >>>> > >>>> ==================================================================mailto:[email protected]================= > >>>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > >>>> = EXIT CODE: 56 > >>>> = CLEANING UP REMAINING PROCESSES > >>>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > >>>> > >>>> =================================================================================== > >>>> > >>>> > >>>> > >>>> > >>>> _______________________________________________ > >>>> discuss mailing list discuss(a)mpich.org > >>>> To manage subscription options or unsubscribe: > >>>> https://lists.mpich.org/mailman/listinfo/discuss > >>> > >>> > >>> > >>> _______________________________________________ > >>> discuss mailing list discuss(a)mpich.org > >>> To manage subscription options or unsubscribe: > >>> https://lists.mpich.org/mailman/listinfo/discuss > >> > >> > >> > >> -- > >> Jeff Hammond > >> jeff.science(a)gmail.com > >> _______________________________________________ > >> discuss mailing list discuss(a)mpich.org > >> To manage subscription options or unsubscribe: > >> https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] turning off MPI abort messages
by Barry Smith 21 Feb '14

21 Feb '14

~/Src/petsc/src/snes/examples/tutorials barry/fix-petsc $ grep joe ex1.c ~/Src/petsc/src/snes/examples/tutorials barry/fix-petsc $ echo $? 1 Well jeepers I just make grep “CRASH”. A nonzero return code denotes some condition, it doesn’t denote a crash. Barry On Feb 21, 2014, at 12:19 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >>> Just configure MPICH such that snprintf isn't discovered by configure >>> and you won't see these messages. >>> >>> The other solution is to fix PETSc so that people can't crash it so easily ;-) >> >> Here we go again. It is not CRASHING; it has detected an error conditioning and trying to appropriately and cleanly terminate. The reason it needs to use MPI_Abort() is that often detecting error conditions is not a uniformly collective thing. >> >> Printing a suitable error message and ending is not crashing. But with all the badly formatted “error messages” printed by MPICH I can not control at the end it looks like it is crashing. > > You're returning a non-zero exit code, which I consider crashing. I > apologize if this definition disagrees with yours. If this is just > gentle cleanup, why not exit with code=0 as Jim suggested already? > > Jeff > >>> On Thu, Feb 20, 2014 at 3:19 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: >>>> If you can find a way to call MPI_Finalize instead, you will portably >>>> eliminate these messages. >>>> >>>> A lesser solution would be to provide an error code of 0 (or MPI_SUCCESS) to >>>> MPI_Abort, e.g. MPI_Comm_abort(MPI_COMM_WORLD, MPI_SUCCESS). This would >>>> eliminate the error message that you are getting from the job launcher. >>>> MPICH could be modified to be quiet about the abort when the application >>>> aborts with an error code of MPI_SUCCESS. >>>> >>>> ~Jim. >>>> >>>> >>>> On Thu, Feb 20, 2014 at 12:33 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: >>>>> >>>>> >>>>> Is there any way to turn off MPICH (and others) printing messages about >>>>> MPI_Abort? We have already prepared and presented useful error messages to >>>>> the user about the situation and would like to avoid having these additional >>>>> messages printed (that often make the situation look worse than it is) >>>>> >>>>> Thanks >>>>> >>>>> Barry >>>>> >>>>> application called MPI_Abort(MPI_COMM_WORLD, 56) - process 0 >>>>> [cli_0]: aborting job: >>>>> application called MPI_Abort(MPI_COMM_WORLD, 56) - process 0 >>>>> >>>>> >>>>> ==================================================================mailto:[email protected]================= >>>>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >>>>> = EXIT CODE: 56 >>>>> = CLEANING UP REMAINING PROCESSES >>>>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >>>>> >>>>> =================================================================================== >>>>> >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> discuss mailing list discuss(a)mpich.org >>>>> To manage subscription options or unsubscribe: >>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> >>>> >>>> >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>> >>> >>> >>> -- >>> Jeff Hammond >>> jeff.science(a)gmail.com >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Help with Cray MPICH
by Marcin Zalewski 21 Feb '14

21 Feb '14

I forgot to ask directly if you have any suggestions to work around this before we have the new version of MPT installed on our machine. Thanks, Marcin On Fri, Feb 21, 2014 at 2:02 PM, Marcin Zalewski <zalewski(a)indiana.edu> wrote: > Nick, > > I have been trying my code to run using an older version of MPI > (cray-mpich2/5.6.1), and I am getting a similar stack trace: > > #0 0x00002aaaada445d5 in memcpy () from /lib64/libc.so.6 > #1 0x00002aaaac5493e4 in MPID_Segment_blkidx_m2m () from > /opt/cray/lib64/libmpich_gnu_47.so.1 > #2 0x00002aaaac547d10 in DLOOP_Leaf_contig_count_block () from > /opt/cray/lib64/libmpich_gnu_47.so.1 > #3 0x00002aaaac5501f1 in MPID_Datatype_free_contents () from > /opt/cray/lib64/libmpich_gnu_47.so.1 > #4 0x00002aaaac538c18 in MPID_nem_gni_progress_send_handle_cookie () > from /opt/cray/lib64/libmpich_gnu_47.so.1 > #5 0x00002aaaac52cacf in MPID_nem_gni_poll () from > /opt/cray/lib64/libmpich_gnu_47.so.1 > #6 0x00002aaaac52d65a in nem_gni_display_pkt.isra.1 () from > /opt/cray/lib64/libmpich_gnu_47.so.1 > #7 0x00002aaaac51285f in MPIDI_CH3I_Progress_init () from > /opt/cray/lib64/libmpich_gnu_47.so.1 > #8 0x00002aaaac5ecc19 in PMPI_Wait () from /opt/cray/lib64/libmpich_gnu_47.so.1 > > Do you know if the bug you have in mind is limited to MPI versions > 6.x.x, or has it occurred in earlier versions too? > > Thank you, > Marcin > > On Mon, Feb 3, 2014 at 2:46 PM, Nick Radcliffe <nradclif(a)cray.com> wrote: >> Hi Marcin, >> >> I can't be certain, but this looks like a bug we fixed recently in Cray MPICH. The fix should be available in MPT 6.2.2 by Feb. 20. >> >> -Nick Radcliffe, >> Cray MPT Team >> >> ________________________________________ >> From: discuss-bounces(a)mpich.org [discuss-bounces(a)mpich.org] on behalf of Marcin Zalewski [marcin.zalewski(a)gmail.com] >> Sent: Monday, February 03, 2014 1:27 PM >> To: discuss(a)mpich.org >> Subject: [mpich-discuss] Help with Cray MPICH >> >> I have an application I am trying to run on a Cray machine composed of >> XE6 nodes. I have run this application previously using Open MPI and >> MVAPICH on a few different machines, so I think it should be more or >> less free of major bugs. However, when I run it on the Cray machine, I >> get a segmentation fault with a stack trace that ends in this: >> >> #0 0x00002aaaafe265e6 in memcpy () from /lib64/libc.so.6 >> #1 0x00002aaaae8fe023 in MPID_Segment_index_m2m () from >> /opt/cray/lib64/libmpich_gnu_48.so.2 >> #2 0x00002aaaae8fca18 in MPID_Segment_manipulate () from >> /opt/cray/lib64/libmpich_gnu_48.so.2 >> #3 0x00002aaaae9049d1 in MPID_Segment_unpack () from >> /opt/cray/lib64/libmpich_gnu_48.so.2 >> #4 0x00002aaaae8edd38 in MPID_nem_gni_complete_rdma_get () from >> /opt/cray/lib64/libmpich_gnu_48.so.2 >> #5 0x00002aaaae8e11c8 in MPID_nem_gni_check_localCQ () from >> /opt/cray/lib64/libmpich_gnu_48.so.2 >> #6 0x00002aaaae8e29fa in MPID_nem_gni_poll () from >> /opt/cray/lib64/libmpich_gnu_48.so.2 >> #7 0x00002aaaae8c3515 in MPIDI_CH3I_Progress () from >> /opt/cray/lib64/libmpich_gnu_48.so.2 >> #8 0x00002aaaae9a3d4d in PMPI_Testsome () from >> /opt/cray/lib64/libmpich_gnu_48.so.2 >> >> I understand that this is very little information to go on, but I >> really cannot deal with this problem, and I thought I would try this >> list while I am trying to solicit help from our support team. >> Currently, I do not have access to a debug version of the library (or >> the source code). Has anyone here seen a similar error? What would be >> a reason to get an error on memcpy in MPICH while calling MPI_Testsome >> in general? I do have asserts in my code to make sure that the output >> arrays are large enough, and I got it to work on other implementations >> of MPI, so I am really clueless to what the problem might be. >> >> Thank you, >> Marcin >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Help with Cray MPICH
by Marcin Zalewski 21 Feb '14

21 Feb '14

Nick, I have been trying my code to run using an older version of MPI (cray-mpich2/5.6.1), and I am getting a similar stack trace: #0 0x00002aaaada445d5 in memcpy () from /lib64/libc.so.6 #1 0x00002aaaac5493e4 in MPID_Segment_blkidx_m2m () from /opt/cray/lib64/libmpich_gnu_47.so.1 #2 0x00002aaaac547d10 in DLOOP_Leaf_contig_count_block () from /opt/cray/lib64/libmpich_gnu_47.so.1 #3 0x00002aaaac5501f1 in MPID_Datatype_free_contents () from /opt/cray/lib64/libmpich_gnu_47.so.1 #4 0x00002aaaac538c18 in MPID_nem_gni_progress_send_handle_cookie () from /opt/cray/lib64/libmpich_gnu_47.so.1 #5 0x00002aaaac52cacf in MPID_nem_gni_poll () from /opt/cray/lib64/libmpich_gnu_47.so.1 #6 0x00002aaaac52d65a in nem_gni_display_pkt.isra.1 () from /opt/cray/lib64/libmpich_gnu_47.so.1 #7 0x00002aaaac51285f in MPIDI_CH3I_Progress_init () from /opt/cray/lib64/libmpich_gnu_47.so.1 #8 0x00002aaaac5ecc19 in PMPI_Wait () from /opt/cray/lib64/libmpich_gnu_47.so.1 Do you know if the bug you have in mind is limited to MPI versions 6.x.x, or has it occurred in earlier versions too? Thank you, Marcin On Mon, Feb 3, 2014 at 2:46 PM, Nick Radcliffe <nradclif(a)cray.com> wrote: > Hi Marcin, > > I can't be certain, but this looks like a bug we fixed recently in Cray MPICH. The fix should be available in MPT 6.2.2 by Feb. 20. > > -Nick Radcliffe, > Cray MPT Team > > ________________________________________ > From: discuss-bounces(a)mpich.org [discuss-bounces(a)mpich.org] on behalf of Marcin Zalewski [marcin.zalewski(a)gmail.com] > Sent: Monday, February 03, 2014 1:27 PM > To: discuss(a)mpich.org > Subject: [mpich-discuss] Help with Cray MPICH > > I have an application I am trying to run on a Cray machine composed of > XE6 nodes. I have run this application previously using Open MPI and > MVAPICH on a few different machines, so I think it should be more or > less free of major bugs. However, when I run it on the Cray machine, I > get a segmentation fault with a stack trace that ends in this: > > #0 0x00002aaaafe265e6 in memcpy () from /lib64/libc.so.6 > #1 0x00002aaaae8fe023 in MPID_Segment_index_m2m () from > /opt/cray/lib64/libmpich_gnu_48.so.2 > #2 0x00002aaaae8fca18 in MPID_Segment_manipulate () from > /opt/cray/lib64/libmpich_gnu_48.so.2 > #3 0x00002aaaae9049d1 in MPID_Segment_unpack () from > /opt/cray/lib64/libmpich_gnu_48.so.2 > #4 0x00002aaaae8edd38 in MPID_nem_gni_complete_rdma_get () from > /opt/cray/lib64/libmpich_gnu_48.so.2 > #5 0x00002aaaae8e11c8 in MPID_nem_gni_check_localCQ () from > /opt/cray/lib64/libmpich_gnu_48.so.2 > #6 0x00002aaaae8e29fa in MPID_nem_gni_poll () from > /opt/cray/lib64/libmpich_gnu_48.so.2 > #7 0x00002aaaae8c3515 in MPIDI_CH3I_Progress () from > /opt/cray/lib64/libmpich_gnu_48.so.2 > #8 0x00002aaaae9a3d4d in PMPI_Testsome () from > /opt/cray/lib64/libmpich_gnu_48.so.2 > > I understand that this is very little information to go on, but I > really cannot deal with this problem, and I thought I would try this > list while I am trying to solicit help from our support team. > Currently, I do not have access to a debug version of the library (or > the source code). Has anyone here seen a similar error? What would be > a reason to get an error on memcpy in MPICH while calling MPI_Testsome > in general? I do have asserts in my code to make sure that the output > arrays are large enough, and I got it to work on other implementations > of MPI, so I am really clueless to what the problem might be. > > Thank you, > Marcin > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0