February 2014 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] [petsc-dev] turning off MPI abort messages
by Junchao Zhang 22 Feb '14

22 Feb '14

See the wiki here<http://wiki.mpich.org/mpich/index.php/Tool_interfaces_(MPI-T),_MPICH_parame…> for how to add a cvar in MPICH. See MPI 3.0 Chapter 14.3.6 for how to access cvars in MPI. --Junchao Zhang On Sat, Feb 22, 2014 at 7:56 AM, William Gropp <wgropp(a)illinois.edu> wrote: > Yes, CVAR already reads env variables, and PETSc could use the MPI_T > interface to set the value at any time. > > Bill > > William Gropp > Director, Parallel Computing Institute > Deputy Director for Research > Institute for Advanced Computing Applications and Technologies > Thomas M. Siebel Chair in Computer Science > University of Illinois Urbana-Champaign > > > > > On Feb 22, 2014, at 5:41 AM, Jeff Hammond wrote: > > I'll look into a general setenv-like mechanism for CVARs rather than an > abort-specific one-off. Might be there already through MPI_T interface > (which is standard, unlike MPIX). > > Jeff > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] [petsc-dev] turning off MPI abort messages
by William Gropp 22 Feb '14

22 Feb '14

Yes, CVAR already reads env variables, and PETSc could use the MPI_T interface to set the value at any time. Bill William Gropp Director, Parallel Computing Institute Deputy Director for Research Institute for Advanced Computing Applications and Technologies Thomas M. Siebel Chair in Computer Science University of Illinois Urbana-Champaign On Feb 22, 2014, at 5:41 AM, Jeff Hammond wrote: > I'll look into a general setenv-like mechanism for CVARs rather than an abort-specific one-off. Might be there already through MPI_T interface (which is standard, unlike MPIX). > > Jeff

1 0

Re: [mpich-discuss] turning off MPI abort messages
by Jeff Hammond 22 Feb '14

22 Feb '14

I'll look into a general setenv-like mechanism for CVARs rather than an abort-specific one-off. Might be there already through MPI_T interface (which is standard, unlike MPIX). Jeff Sent from my iPhone > On Feb 21, 2014, at 11:35 PM, Jed Brown <jed(a)jedbrown.org> wrote: > > Jeff Hammond <jeff.science(a)gmail.com> writes: > >> Did you look at my patch or the demonstration yet? I posted all the details this afternoon. > > Yeah, I wrote the message on the plane before I could read it. > >> I tried very hard to support verbosity suppression in a reasonable way at runtime. >> >> Do you really want an MPIX call that is equivalent to setenv("<see >> patch>")? Is the extra code worth it? (These are serious questions.) > > My understanding of these variables is that they are processed in > MPI_Init. PETSc may not have access to MPI_Init, so we're too late to > influence the environment variable. But if the user gets a return code > and calls our error-handler (as we encourage them to do if they have > nothing better to do with it), we'd like to be able to exit cleanly. A > global setting isn't as good because if the user encounters an error > condition and "crashes", it probably makes sense for MPICH to print the > information. > > > The reason this issue came up is that we have a sizable fraction of > support email in which the exact question is answered in the error > message we print when they call our error handler. But a lot of users > don't read the error message and worse, they don't copy the whole thing > into the email. This happens almost every day and requires an extra > round-trip on the list the resolve. Our thinking was that if we can > make the error message visually cleaner, they may be more likely to read > it and answer their own question or copy the message into an email in > which case we can ask them to read the relevant lines of the error > message. (A large fraction of these issues are because the user has > configured something incompatible, usually via run-time options. It is > very analogous to asking for a file that doesn't exist. A > straightforward change of run-time options will get them running > correctly.) > > The MPICH Abort messages add a lot of visual clutter that we hypothesize > makes people less likely to read our messages (telling them how to fix > the problem), or to believe the output does not have useful information. > > I think something along the lines of MPIX_Abort_quietly(comm,err) would > be the right granularity since it allows our error handler to suppress > the clutter without influencing the behavior observed in other libraries > or the application.

1 0

Re: [mpich-discuss] turning off MPI abort messages
by Jed Brown 21 Feb '14

21 Feb '14

Jeff Hammond <jeff.science(a)gmail.com> writes: > Did you look at my patch or the demonstration yet? I posted all the details this afternoon. Yeah, I wrote the message on the plane before I could read it. > I tried very hard to support verbosity suppression in a reasonable way at runtime. > > Do you really want an MPIX call that is equivalent to setenv("<see > patch>")? Is the extra code worth it? (These are serious questions.) My understanding of these variables is that they are processed in MPI_Init. PETSc may not have access to MPI_Init, so we're too late to influence the environment variable. But if the user gets a return code and calls our error-handler (as we encourage them to do if they have nothing better to do with it), we'd like to be able to exit cleanly. A global setting isn't as good because if the user encounters an error condition and "crashes", it probably makes sense for MPICH to print the information. The reason this issue came up is that we have a sizable fraction of support email in which the exact question is answered in the error message we print when they call our error handler. But a lot of users don't read the error message and worse, they don't copy the whole thing into the email. This happens almost every day and requires an extra round-trip on the list the resolve. Our thinking was that if we can make the error message visually cleaner, they may be more likely to read it and answer their own question or copy the message into an email in which case we can ask them to read the relevant lines of the error message. (A large fraction of these issues are because the user has configured something incompatible, usually via run-time options. It is very analogous to asking for a file that doesn't exist. A straightforward change of run-time options will get them running correctly.) The MPICH Abort messages add a lot of visual clutter that we hypothesize makes people less likely to read our messages (telling them how to fix the problem), or to believe the output does not have useful information. I think something along the lines of MPIX_Abort_quietly(comm,err) would be the right granularity since it allows our error handler to suppress the clutter without influencing the behavior observed in other libraries or the application.

1 0

Re: [mpich-discuss] turning off MPI abort messages
by Jeff Hammond 21 Feb '14

21 Feb '14

Jed, Did you look at my patch or the demonstration yet? I posted all the details this afternoon. I tried very hard to support verbosity suppression in a reasonable way at runtime. Do you really want an MPIX call that is equivalent to setenv("<see patch>")? Is the extra code worth it? (These are serious questions.) Jeff Sent from my iPhone > On Feb 21, 2014, at 1:33 PM, Jed Brown <jed(a)jedbrown.org> wrote: > > Jim Dinan <james.dinan(a)gmail.com> writes: > >> Barry, >> >> Do you need a portable solution that works across different MPI >> implementations, > > This is well outside the standard so I don't see this being viable. > >> or does a solution for making just MPICH silent address your need? If >> the latter, you could probably convince someone on the MPICH team to >> add an environment variable for MPICH and a command-line "quiet" flag >> for hydra/mpiexec. > > Even better would be an MPIX_Set_error_verbosity() or MPIX_Abort_quietly(). > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] turning off MPI abort messages
by Barry Smith 21 Feb '14

21 Feb '14

On Feb 21, 2014, at 5:01 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > On Fri, Feb 21, 2014 at 4:32 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: >> >> Jeff, >> >> Thanks. This is certainly a useful thing. > > It's only half a solution right now. Hacking Hydra is a bit more > difficult for me. Not sure how long before I can solve that in a > manner that the MPICH folks find acceptable. > >> I never meant to kick a hornet’s nest with my initial email. I was taught by my postdoctoral advisor that any library or package that had stdout or stderr output hardwired that could not be turned off without losing functionality was rude and poorly thought out but then that guy probably never amounted to anything I guess so I should just ignore him since he doesn’t represent main stream thought. > > A colleague of mine suggested that libraries shouldn't be calling > MPI_Abort but rather return an error code to the application and let > them decide how to handle it, but he learned MPI from Bill Gropp, so > he might not know anything ;-) Actually "the library" isn’t “calling” MPI_Abort, the library’s default error handler is eventually calling MPI_Abort(). The library returns error codes to the application code and the application code is free to handle them anyway it likes as well as set its own error handlers. Barry > > I apologize for being unpleasant earlier. > > Best, > > Jeff > > >> >> Barry >> >> On Feb 21, 2014, at 3:10 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >> >>> Barry: >>> >>> Would the following behavior be acceptable to you? I have only made >>> the changes in MPI but am looking at the process manager now. >>> >>> Jeff >>> >>> >>> # Without the process manager >>> >>> alcfwl181:build jhammond$ export MPIR_CVAR_SUPPRESS_ABORT_MESSAGE=0 >>> alcfwl181:build jhammond$ ./a.out >>> application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 >>> alcfwl181:build jhammond$ export MPIR_CVAR_SUPPRESS_ABORT_MESSAGE=1 >>> alcfwl181:build jhammond$ ./a.out >>> >>> alcfwl181:build jhammond$ unset MPIR_CVAR_SUPPRESS_ABORT_MESSAGE >>> alcfwl181:build jhammond$ ./a.out >>> application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 >>> >>> # With the process manager >>> >>> alcfwl181:build jhammond$ mpiexec -n 1 -env >>> MPIR_CVAR_SUPPRESS_ABORT_MESSAGE 0 ./a.out >>> application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 >>> >>> =================================================================================== >>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >>> = PID 61023 RUNNING AT alcfwl181.alcf.anl.gov >>> = EXIT CODE: 1 >>> = CLEANING UP REMAINING PROCESSES >>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >>> =================================================================================== >>> alcfwl181:build jhammond$ mpiexec -n 1 -env >>> MPIR_CVAR_SUPPRESS_ABORT_MESSAGE 1 ./a.out >>> >>> >>> =================================================================================== >>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >>> = PID 61026 RUNNING AT alcfwl181.alcf.anl.gov >>> = EXIT CODE: 1 >>> = CLEANING UP REMAINING PROCESSES >>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >>> =================================================================================== >>> alcfwl181:build jhammond$ mpiexec -n 1 ./a.out >>> application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 >>> >>> =================================================================================== >>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >>> = PID 61032 RUNNING AT alcfwl181.alcf.anl.gov >>> = EXIT CODE: 1 >>> = CLEANING UP REMAINING PROCESSES >>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >>> =================================================================================== >>> >>> >>> >>> On Thu, Feb 20, 2014 at 11:33 AM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: >>>> >>>> Is there any way to turn off MPICH (and others) printing messages about MPI_Abort? We have already prepared and presented useful error messages to the user about the situation and would like to avoid having these additional messages printed (that often make the situation look worse than it is) >>>> >>>> Thanks >>>> >>>> Barry >>>> >>>> application called MPI_Abort(MPI_COMM_WORLD, 56) - process 0 >>>> [cli_0]: aborting job: >>>> application called MPI_Abort(MPI_COMM_WORLD, 56) - process 0 >>>> >>>> ==================================================================mailto:[email protected]================= >>>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >>>> = EXIT CODE: 56 >>>> = CLEANING UP REMAINING PROCESSES >>>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >>>> =================================================================================== >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>> >>> >>> >>> -- >>> Jeff Hammond >>> jeff.science(a)gmail.com >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] [petsc-dev] turning off MPI abort messages
by William Gropp 21 Feb '14

21 Feb '14

My thanks too. My position, as Barry notes, is that if you ever use | (pipe) in a shell command, you agree with me, even if you don't know it :). One of the fundamental strengths of Unix is the easy ability to build tools by composing other tools with pipe, and programs and systems that insist on gratuitous output break this model; they become second class citizens in the Unix world. Bill William Gropp Director, Parallel Computing Institute Deputy Director for Research Institute for Advanced Computing Applications and Technologies Thomas M. Siebel Chair in Computer Science University of Illinois Urbana-Champaign On Feb 21, 2014, at 2:32 PM, Barry Smith wrote: > > Jeff, > > Thanks. This is certainly a useful thing. > > I never meant to kick a hornet’s nest with my initial email. I was taught by my postdoctoral advisor that any library or package that had stdout or stderr output hardwired that could not be turned off without losing functionality was rude and poorly thought out but then that guy probably never amounted to anything I guess so I should just ignore him since he doesn’t represent main stream thought. > > Barry > > On Feb 21, 2014, at 3:10 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > >> Barry: >> >> Would the following behavior be acceptable to you? I have only made >> the changes in MPI but am looking at the process manager now. >> >> Jeff >> >> >> # Without the process manager >> >> alcfwl181:build jhammond$ export MPIR_CVAR_SUPPRESS_ABORT_MESSAGE=0 >> alcfwl181:build jhammond$ ./a.out >> application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 >> alcfwl181:build jhammond$ export MPIR_CVAR_SUPPRESS_ABORT_MESSAGE=1 >> alcfwl181:build jhammond$ ./a.out >> >> alcfwl181:build jhammond$ unset MPIR_CVAR_SUPPRESS_ABORT_MESSAGE >> alcfwl181:build jhammond$ ./a.out >> application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 >> >> # With the process manager >> >> alcfwl181:build jhammond$ mpiexec -n 1 -env >> MPIR_CVAR_SUPPRESS_ABORT_MESSAGE 0 ./a.out >> application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 >> >> =================================================================================== >> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >> = PID 61023 RUNNING AT alcfwl181.alcf.anl.gov >> = EXIT CODE: 1 >> = CLEANING UP REMAINING PROCESSES >> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >> =================================================================================== >> alcfwl181:build jhammond$ mpiexec -n 1 -env >> MPIR_CVAR_SUPPRESS_ABORT_MESSAGE 1 ./a.out >> >> >> =================================================================================== >> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >> = PID 61026 RUNNING AT alcfwl181.alcf.anl.gov >> = EXIT CODE: 1 >> = CLEANING UP REMAINING PROCESSES >> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >> =================================================================================== >> alcfwl181:build jhammond$ mpiexec -n 1 ./a.out >> application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 >> >> =================================================================================== >> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >> = PID 61032 RUNNING AT alcfwl181.alcf.anl.gov >> = EXIT CODE: 1 >> = CLEANING UP REMAINING PROCESSES >> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >> =================================================================================== >> >> >> >> On Thu, Feb 20, 2014 at 11:33 AM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: >>> >>> Is there any way to turn off MPICH (and others) printing messages about MPI_Abort? We have already prepared and presented useful error messages to the user about the situation and would like to avoid having these additional messages printed (that often make the situation look worse than it is) >>> >>> Thanks >>> >>> Barry >>> >>> application called MPI_Abort(MPI_COMM_WORLD, 56) - process 0 >>> [cli_0]: aborting job: >>> application called MPI_Abort(MPI_COMM_WORLD, 56) - process 0 >>> >>> ==================================================================mailto:[email protected]================= >>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >>> = EXIT CODE: 56 >>> = CLEANING UP REMAINING PROCESSES >>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >>> =================================================================================== >>> >>> >>> >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] turning off MPI abort messages
by Jeff Hammond 21 Feb '14

21 Feb '14

On Fri, Feb 21, 2014 at 4:32 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > Jeff, > > Thanks. This is certainly a useful thing. It's only half a solution right now. Hacking Hydra is a bit more difficult for me. Not sure how long before I can solve that in a manner that the MPICH folks find acceptable. > I never meant to kick a hornet’s nest with my initial email. I was taught by my postdoctoral advisor that any library or package that had stdout or stderr output hardwired that could not be turned off without losing functionality was rude and poorly thought out but then that guy probably never amounted to anything I guess so I should just ignore him since he doesn’t represent main stream thought. A colleague of mine suggested that libraries shouldn't be calling MPI_Abort but rather return an error code to the application and let them decide how to handle it, but he learned MPI from Bill Gropp, so he might not know anything ;-) I apologize for being unpleasant earlier. Best, Jeff > > Barry > > On Feb 21, 2014, at 3:10 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > >> Barry: >> >> Would the following behavior be acceptable to you? I have only made >> the changes in MPI but am looking at the process manager now. >> >> Jeff >> >> >> # Without the process manager >> >> alcfwl181:build jhammond$ export MPIR_CVAR_SUPPRESS_ABORT_MESSAGE=0 >> alcfwl181:build jhammond$ ./a.out >> application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 >> alcfwl181:build jhammond$ export MPIR_CVAR_SUPPRESS_ABORT_MESSAGE=1 >> alcfwl181:build jhammond$ ./a.out >> >> alcfwl181:build jhammond$ unset MPIR_CVAR_SUPPRESS_ABORT_MESSAGE >> alcfwl181:build jhammond$ ./a.out >> application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 >> >> # With the process manager >> >> alcfwl181:build jhammond$ mpiexec -n 1 -env >> MPIR_CVAR_SUPPRESS_ABORT_MESSAGE 0 ./a.out >> application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 >> >> =================================================================================== >> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >> = PID 61023 RUNNING AT alcfwl181.alcf.anl.gov >> = EXIT CODE: 1 >> = CLEANING UP REMAINING PROCESSES >> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >> =================================================================================== >> alcfwl181:build jhammond$ mpiexec -n 1 -env >> MPIR_CVAR_SUPPRESS_ABORT_MESSAGE 1 ./a.out >> >> >> =================================================================================== >> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >> = PID 61026 RUNNING AT alcfwl181.alcf.anl.gov >> = EXIT CODE: 1 >> = CLEANING UP REMAINING PROCESSES >> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >> =================================================================================== >> alcfwl181:build jhammond$ mpiexec -n 1 ./a.out >> application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 >> >> =================================================================================== >> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >> = PID 61032 RUNNING AT alcfwl181.alcf.anl.gov >> = EXIT CODE: 1 >> = CLEANING UP REMAINING PROCESSES >> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >> =================================================================================== >> >> >> >> On Thu, Feb 20, 2014 at 11:33 AM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: >>> >>> Is there any way to turn off MPICH (and others) printing messages about MPI_Abort? We have already prepared and presented useful error messages to the user about the situation and would like to avoid having these additional messages printed (that often make the situation look worse than it is) >>> >>> Thanks >>> >>> Barry >>> >>> application called MPI_Abort(MPI_COMM_WORLD, 56) - process 0 >>> [cli_0]: aborting job: >>> application called MPI_Abort(MPI_COMM_WORLD, 56) - process 0 >>> >>> ==================================================================mailto:[email protected]================= >>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >>> = EXIT CODE: 56 >>> = CLEANING UP REMAINING PROCESSES >>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >>> =================================================================================== >>> >>> >>> >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [mpich-discuss] turning off MPI abort messages
by Barry Smith 21 Feb '14

21 Feb '14

Jeff, Thanks. This is certainly a useful thing. I never meant to kick a hornet’s nest with my initial email. I was taught by my postdoctoral advisor that any library or package that had stdout or stderr output hardwired that could not be turned off without losing functionality was rude and poorly thought out but then that guy probably never amounted to anything I guess so I should just ignore him since he doesn’t represent main stream thought. Barry On Feb 21, 2014, at 3:10 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > Barry: > > Would the following behavior be acceptable to you? I have only made > the changes in MPI but am looking at the process manager now. > > Jeff > > > # Without the process manager > > alcfwl181:build jhammond$ export MPIR_CVAR_SUPPRESS_ABORT_MESSAGE=0 > alcfwl181:build jhammond$ ./a.out > application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 > alcfwl181:build jhammond$ export MPIR_CVAR_SUPPRESS_ABORT_MESSAGE=1 > alcfwl181:build jhammond$ ./a.out > > alcfwl181:build jhammond$ unset MPIR_CVAR_SUPPRESS_ABORT_MESSAGE > alcfwl181:build jhammond$ ./a.out > application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 > > # With the process manager > > alcfwl181:build jhammond$ mpiexec -n 1 -env > MPIR_CVAR_SUPPRESS_ABORT_MESSAGE 0 ./a.out > application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 > > =================================================================================== > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > = PID 61023 RUNNING AT alcfwl181.alcf.anl.gov > = EXIT CODE: 1 > = CLEANING UP REMAINING PROCESSES > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > =================================================================================== > alcfwl181:build jhammond$ mpiexec -n 1 -env > MPIR_CVAR_SUPPRESS_ABORT_MESSAGE 1 ./a.out > > > =================================================================================== > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > = PID 61026 RUNNING AT alcfwl181.alcf.anl.gov > = EXIT CODE: 1 > = CLEANING UP REMAINING PROCESSES > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > =================================================================================== > alcfwl181:build jhammond$ mpiexec -n 1 ./a.out > application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 > > =================================================================================== > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > = PID 61032 RUNNING AT alcfwl181.alcf.anl.gov > = EXIT CODE: 1 > = CLEANING UP REMAINING PROCESSES > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > =================================================================================== > > > > On Thu, Feb 20, 2014 at 11:33 AM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: >> >> Is there any way to turn off MPICH (and others) printing messages about MPI_Abort? We have already prepared and presented useful error messages to the user about the situation and would like to avoid having these additional messages printed (that often make the situation look worse than it is) >> >> Thanks >> >> Barry >> >> application called MPI_Abort(MPI_COMM_WORLD, 56) - process 0 >> [cli_0]: aborting job: >> application called MPI_Abort(MPI_COMM_WORLD, 56) - process 0 >> >> ==================================================================mailto:[email protected]================= >> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >> = EXIT CODE: 56 >> = CLEANING UP REMAINING PROCESSES >> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >> =================================================================================== >> >> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] turning off MPI abort messages
by Jeff Hammond 21 Feb '14

21 Feb '14

https://trac.mpich.org/projects/mpich/ticket/2038 has the patches. Jeff On Fri, Feb 21, 2014 at 3:47 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > Barry: > > Can you tolerate the following workaround for Hydra's error cleanup or > do you need it to be internal? I presume you know enough bash to > generalize a.sh appropriately. > > alcfwl181:build jhammond$ cat a.sh > #!/bin/sh > $1 > true > alcfwl181:build jhammond$ mpiexec -n 1 -env > MPIR_CVAR_SUPPRESS_ABORT_MESSAGE 0 ./a.sh ./a.out > application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 > alcfwl181:build jhammond$ mpiexec -n 1 -env > MPIR_CVAR_SUPPRESS_ABORT_MESSAGE 1 ./a.sh ./a.out > > alcfwl181:build jhammond$ mpiexec -n 1 -env > MPIR_CVAR_SUPPRESS_ABORT_MESSAGE 1 ./a.out > > > =================================================================================== > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > = PID 61123 RUNNING AT alcfwl181.alcf.anl.gov > = EXIT CODE: 1 > = CLEANING UP REMAINING PROCESSES > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > =================================================================================== > alcfwl181:build jhammond$ mpiexec -n 1 -env > MPIR_CVAR_SUPPRESS_ABORT_MESSAGE 0 ./a.out > application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 > > =================================================================================== > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > = PID 61126 RUNNING AT alcfwl181.alcf.anl.gov > = EXIT CODE: 1 > = CLEANING UP REMAINING PROCESSES > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > =================================================================================== > > On Fri, Feb 21, 2014 at 3:10 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >> Barry: >> >> Would the following behavior be acceptable to you? I have only made >> the changes in MPI but am looking at the process manager now. >> >> Jeff >> >> >> # Without the process manager >> >> alcfwl181:build jhammond$ export MPIR_CVAR_SUPPRESS_ABORT_MESSAGE=0 >> alcfwl181:build jhammond$ ./a.out >> application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 >> alcfwl181:build jhammond$ export MPIR_CVAR_SUPPRESS_ABORT_MESSAGE=1 >> alcfwl181:build jhammond$ ./a.out >> >> alcfwl181:build jhammond$ unset MPIR_CVAR_SUPPRESS_ABORT_MESSAGE >> alcfwl181:build jhammond$ ./a.out >> application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 >> >> # With the process manager >> >> alcfwl181:build jhammond$ mpiexec -n 1 -env >> MPIR_CVAR_SUPPRESS_ABORT_MESSAGE 0 ./a.out >> application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 >> >> =================================================================================== >> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >> = PID 61023 RUNNING AT alcfwl181.alcf.anl.gov >> = EXIT CODE: 1 >> = CLEANING UP REMAINING PROCESSES >> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >> =================================================================================== >> alcfwl181:build jhammond$ mpiexec -n 1 -env >> MPIR_CVAR_SUPPRESS_ABORT_MESSAGE 1 ./a.out >> >> >> =================================================================================== >> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >> = PID 61026 RUNNING AT alcfwl181.alcf.anl.gov >> = EXIT CODE: 1 >> = CLEANING UP REMAINING PROCESSES >> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >> =================================================================================== >> alcfwl181:build jhammond$ mpiexec -n 1 ./a.out >> application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 >> >> =================================================================================== >> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >> = PID 61032 RUNNING AT alcfwl181.alcf.anl.gov >> = EXIT CODE: 1 >> = CLEANING UP REMAINING PROCESSES >> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >> =================================================================================== >> >> >> >> On Thu, Feb 20, 2014 at 11:33 AM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: >>> >>> Is there any way to turn off MPICH (and others) printing messages about MPI_Abort? We have already prepared and presented useful error messages to the user about the situation and would like to avoid having these additional messages printed (that often make the situation look worse than it is) >>> >>> Thanks >>> >>> Barry >>> >>> application called MPI_Abort(MPI_COMM_WORLD, 56) - process 0 >>> [cli_0]: aborting job: >>> application called MPI_Abort(MPI_COMM_WORLD, 56) - process 0 >>> >>> ==================================================================mailto:[email protected]================= >>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >>> = EXIT CODE: 56 >>> = CLEANING UP REMAINING PROCESSES >>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >>> =================================================================================== >>> >>> >>> >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com > > > > -- > Jeff Hammond > jeff.science(a)gmail.com -- Jeff Hammond jeff.science(a)gmail.com

1 0