- discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] [EXTERNAL] Re: Abnormal termination on Linux
by Mccall, Kurt E. (MSFC-EV41) 07 Apr '20

07 Apr '20

Thanks for your insights, Hui and Joachim. I'm using the methods in https://www.mcs.anl.gov/~lusk/papers/fault-tolerance.pdf to attempt to create fault tolerance in my MPI code. I'm going to have to reread that paper to see if the issues you have raised are addressed. Kurt -----Original Message----- From: Joachim Protze <protze(a)itc.rwth-aachen.de> Sent: Tuesday, April 7, 2020 3:37 AM To: discuss(a)mpich.org; Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Cc: Zhou, Hui <zhouh(a)anl.gov> Subject: [EXTERNAL] Re: [mpich-discuss] Abnormal termination on Linux As long as you use a standard MPI implementation, there is no resilience and the MPI job will not survive a segfault or any other MPI error. Probably you should target your question towards "Why does my application segfault in the first place?" If it's only one process, that segfaults probably use the core file for debugging. By default I assume corefile size is limited on most clusters, so increase it in your batch job: ulimit -c unlimited gdb a.out -c corefile.hostname.timestamp backtrace quit Am 07.04.20 um 00:23 schrieb Zhou, Hui via discuss: > I see. MPICH doesn’t trap any signal as I checked 😊. So back to your issue, let’s assume your trap did work, but after the signal handler, which after it called `MPI_Finalize`, the code is still going to run the same code that causes segfault in the first place, right? Wouldn’t that endup in infinite signal loop or maybe the kernel smart out and by-passed your handler? Just a thought experiment. > Calling MPI_Finalize would not help, because MPI_Finalize is a collective call and will not finish before all other processes have called MPI_Finalize. Furthermore, in such error situation you won't have any guarantees, that the process does not have any open blocking or non-blocking communication. Such communication would need to be completed before calling MPI_Finalize. - Joachim > Beside, MPICH is thread-safe but not interrupt safe. So you should not really call MPI functions inside signal handlers. > > -- > Hui Zhou > > > From: "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> > Date: Monday, April 6, 2020 at 4:50 PM > To: "Zhou, Hui" <zhouh(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> > Subject: RE: [mpich-discuss] Abnormal termination on Linux > > Hui, > > I’d like to trap segfaults so that the process that raised them can be shut down gracefully/finalized without taking down the whole MPI job. Maybe you are right and I shouldn’t trap them. > > However, when I install my signal handlers, I check if MPI already has a signal handler installed for each of them. It isn’t reporting that MPI has done so. Perhaps my code is incorrect? > > setUpOneHandler(const int signum) > { > struct sigaction sa_new, sa_old; > > sa_new.sa_handler = mpiSignalHandler; > sa_new.sa_flags = 0; > sigemptyset(&sa_new.sa_mask); > > if (sigaction(signum, NULL, &sa_old) < 0) > { > // ERROR: could not query old handler > > } else if (sa_old.sa_handler == SIG_IGN || > sa_old.sa_handler == SIG_DFL) > { > // MPI hasn't set its own handler for this signal, so we > // will install our own. > > if (sigaction(signum, &sa_new, NULL) < 0) > { > // ERROR: could not set new handler > } > } else > { > // MPI already has a handler installed for this signal. Do nothing > } > } > > > > > From: Zhou, Hui <zhouh(a)anl.gov> > Sent: Monday, April 6, 2020 2:08 PM > To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org > Subject: [EXTERNAL] Re: [mpich-discuss] Abnormal termination on Linux > > Thanks, Kurt. I think the reason your signal trap didn’t work is because `mpiexec` is trapping it first. Segfault is code error. Why would you want to trap it? > > -- > Hui Zhou > > > From: "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> > Date: Monday, April 6, 2020 at 1:52 PM > To: "Zhou, Hui" <zhouh(a)anl.gov<mailto:[email protected]>>, "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> > Subject: RE: [mpich-discuss] Abnormal termination on Linux > > Hui, > > Sorry for not mentioning that. MPICH 3.3.2 compiled with pgc++ 19.5. > > Kurt > > From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> > Sent: Monday, April 6, 2020 12:58 PM > To: discuss(a)mpich.org<mailto:[email protected]> > Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> > Subject: [EXTERNAL] Re: [mpich-discuss] Abnormal termination on Linux > > Which version of MPICH were you running? > > -- > Hui Zhou > > > From: "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org<mailto:[email protected]>> > Reply-To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> > Date: Monday, April 6, 2020 at 12:45 PM > To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> > Cc: "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> > Subject: Re: [mpich-discuss] Abnormal termination on Linux > > I should mention that I am unable to predict in which node or process the abnormal termination occurs, so I can’t practically attach a debugger and try to intercept the error. > > Kurt > > From: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> > Sent: Monday, April 6, 2020 11:50 AM > To: discuss(a)mpich.org<mailto:[email protected]> > Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> > Subject: Abnormal termination on Linux > > I have a couple of questions about abnormal termination. The EXIT CODE below is 11, which could be signal SIGSEGV, or is it something defined by MPICH? If it is SIGSEGV, it is strange because my signal handler isn’t catching it and cleaning up properly (the signal handler calls MPI_Finalize()). Is there any way to get more information about the location of the error? > > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > = PID 14385 RUNNING AT n020.cluster.com > = EXIT CODE: 11 > = CLEANING UP REMAINING PROCESSES > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > > Thanks, > Kurt > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Dipl.-Inf. Joachim Protze IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 protze(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0

Re: [mpich-discuss] Abnormal termination on Linux
by Joachim Protze 07 Apr '20

07 Apr '20

As long as you use a standard MPI implementation, there is no resilience and the MPI job will not survive a segfault or any other MPI error. Probably you should target your question towards "Why does my application segfault in the first place?" If it's only one process, that segfaults probably use the core file for debugging. By default I assume corefile size is limited on most clusters, so increase it in your batch job: ulimit -c unlimited gdb a.out -c corefile.hostname.timestamp backtrace quit Am 07.04.20 um 00:23 schrieb Zhou, Hui via discuss: > I see. MPICH doesn’t trap any signal as I checked 😊. So back to your issue, let’s assume your trap did work, but after the signal handler, which after it called `MPI_Finalize`, the code is still going to run the same code that causes segfault in the first place, right? Wouldn’t that endup in infinite signal loop or maybe the kernel smart out and by-passed your handler? Just a thought experiment. > Calling MPI_Finalize would not help, because MPI_Finalize is a collective call and will not finish before all other processes have called MPI_Finalize. Furthermore, in such error situation you won't have any guarantees, that the process does not have any open blocking or non-blocking communication. Such communication would need to be completed before calling MPI_Finalize. - Joachim > Beside, MPICH is thread-safe but not interrupt safe. So you should not really call MPI functions inside signal handlers. > > -- > Hui Zhou > > > From: "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> > Date: Monday, April 6, 2020 at 4:50 PM > To: "Zhou, Hui" <zhouh(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> > Subject: RE: [mpich-discuss] Abnormal termination on Linux > > Hui, > > I’d like to trap segfaults so that the process that raised them can be shut down gracefully/finalized without taking down the whole MPI job. Maybe you are right and I shouldn’t trap them. > > However, when I install my signal handlers, I check if MPI already has a signal handler installed for each of them. It isn’t reporting that MPI has done so. Perhaps my code is incorrect? > > setUpOneHandler(const int signum) > { > struct sigaction sa_new, sa_old; > > sa_new.sa_handler = mpiSignalHandler; > sa_new.sa_flags = 0; > sigemptyset(&sa_new.sa_mask); > > if (sigaction(signum, NULL, &sa_old) < 0) > { > // ERROR: could not query old handler > > } else if (sa_old.sa_handler == SIG_IGN || > sa_old.sa_handler == SIG_DFL) > { > // MPI hasn't set its own handler for this signal, so we > // will install our own. > > if (sigaction(signum, &sa_new, NULL) < 0) > { > // ERROR: could not set new handler > } > } else > { > // MPI already has a handler installed for this signal. Do nothing > } > } > > > > > From: Zhou, Hui <zhouh(a)anl.gov> > Sent: Monday, April 6, 2020 2:08 PM > To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org > Subject: [EXTERNAL] Re: [mpich-discuss] Abnormal termination on Linux > > Thanks, Kurt. I think the reason your signal trap didn’t work is because `mpiexec` is trapping it first. Segfault is code error. Why would you want to trap it? > > -- > Hui Zhou > > > From: "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> > Date: Monday, April 6, 2020 at 1:52 PM > To: "Zhou, Hui" <zhouh(a)anl.gov<mailto:[email protected]>>, "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> > Subject: RE: [mpich-discuss] Abnormal termination on Linux > > Hui, > > Sorry for not mentioning that. MPICH 3.3.2 compiled with pgc++ 19.5. > > Kurt > > From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> > Sent: Monday, April 6, 2020 12:58 PM > To: discuss(a)mpich.org<mailto:[email protected]> > Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> > Subject: [EXTERNAL] Re: [mpich-discuss] Abnormal termination on Linux > > Which version of MPICH were you running? > > -- > Hui Zhou > > > From: "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org<mailto:[email protected]>> > Reply-To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> > Date: Monday, April 6, 2020 at 12:45 PM > To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> > Cc: "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> > Subject: Re: [mpich-discuss] Abnormal termination on Linux > > I should mention that I am unable to predict in which node or process the abnormal termination occurs, so I can’t practically attach a debugger and try to intercept the error. > > Kurt > > From: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> > Sent: Monday, April 6, 2020 11:50 AM > To: discuss(a)mpich.org<mailto:[email protected]> > Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> > Subject: Abnormal termination on Linux > > I have a couple of questions about abnormal termination. The EXIT CODE below is 11, which could be signal SIGSEGV, or is it something defined by MPICH? If it is SIGSEGV, it is strange because my signal handler isn’t catching it and cleaning up properly (the signal handler calls MPI_Finalize()). Is there any way to get more information about the location of the error? > > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > = PID 14385 RUNNING AT n020.cluster.com > = EXIT CODE: 11 > = CLEANING UP REMAINING PROCESSES > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > > Thanks, > Kurt > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Dipl.-Inf. Joachim Protze IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 protze(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0

Re: [mpich-discuss] Abnormal termination on Linux
by Zhou, Hui 06 Apr '20

06 Apr '20

I see. MPICH doesn’t trap any signal as I checked 😊. So back to your issue, let’s assume your trap did work, but after the signal handler, which after it called `MPI_Finalize`, the code is still going to run the same code that causes segfault in the first place, right? Wouldn’t that endup in infinite signal loop or maybe the kernel smart out and by-passed your handler? Just a thought experiment. Beside, MPICH is thread-safe but not interrupt safe. So you should not really call MPI functions inside signal handlers. -- Hui Zhou From: "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> Date: Monday, April 6, 2020 at 4:50 PM To: "Zhou, Hui" <zhouh(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: RE: [mpich-discuss] Abnormal termination on Linux Hui, I’d like to trap segfaults so that the process that raised them can be shut down gracefully/finalized without taking down the whole MPI job. Maybe you are right and I shouldn’t trap them. However, when I install my signal handlers, I check if MPI already has a signal handler installed for each of them. It isn’t reporting that MPI has done so. Perhaps my code is incorrect? setUpOneHandler(const int signum) { struct sigaction sa_new, sa_old; sa_new.sa_handler = mpiSignalHandler; sa_new.sa_flags = 0; sigemptyset(&sa_new.sa_mask); if (sigaction(signum, NULL, &sa_old) < 0) { // ERROR: could not query old handler } else if (sa_old.sa_handler == SIG_IGN || sa_old.sa_handler == SIG_DFL) { // MPI hasn't set its own handler for this signal, so we // will install our own. if (sigaction(signum, &sa_new, NULL) < 0) { // ERROR: could not set new handler } } else { // MPI already has a handler installed for this signal. Do nothing } } From: Zhou, Hui <zhouh(a)anl.gov> Sent: Monday, April 6, 2020 2:08 PM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: [EXTERNAL] Re: [mpich-discuss] Abnormal termination on Linux Thanks, Kurt. I think the reason your signal trap didn’t work is because `mpiexec` is trapping it first. Segfault is code error. Why would you want to trap it? -- Hui Zhou From: "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Date: Monday, April 6, 2020 at 1:52 PM To: "Zhou, Hui" <zhouh(a)anl.gov<mailto:[email protected]>>, "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Subject: RE: [mpich-discuss] Abnormal termination on Linux Hui, Sorry for not mentioning that. MPICH 3.3.2 compiled with pgc++ 19.5. Kurt From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> Sent: Monday, April 6, 2020 12:58 PM To: discuss(a)mpich.org<mailto:[email protected]> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: [EXTERNAL] Re: [mpich-discuss] Abnormal termination on Linux Which version of MPICH were you running? -- Hui Zhou From: "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org<mailto:[email protected]>> Reply-To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Date: Monday, April 6, 2020 at 12:45 PM To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Cc: "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: Re: [mpich-discuss] Abnormal termination on Linux I should mention that I am unable to predict in which node or process the abnormal termination occurs, so I can’t practically attach a debugger and try to intercept the error. Kurt From: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Sent: Monday, April 6, 2020 11:50 AM To: discuss(a)mpich.org<mailto:[email protected]> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: Abnormal termination on Linux I have a couple of questions about abnormal termination. The EXIT CODE below is 11, which could be signal SIGSEGV, or is it something defined by MPICH? If it is SIGSEGV, it is strange because my signal handler isn’t catching it and cleaning up properly (the signal handler calls MPI_Finalize()). Is there any way to get more information about the location of the error? = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 14385 RUNNING AT n020.cluster.com = EXIT CODE: 11 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES Thanks, Kurt

1 0

Re: [mpich-discuss] Abnormal termination on Linux
by Mccall, Kurt E. (MSFC-EV41) 06 Apr '20

06 Apr '20

Hui, I’d like to trap segfaults so that the process that raised them can be shut down gracefully/finalized without taking down the whole MPI job. Maybe you are right and I shouldn’t trap them. However, when I install my signal handlers, I check if MPI already has a signal handler installed for each of them. It isn’t reporting that MPI has done so. Perhaps my code is incorrect? setUpOneHandler(const int signum) { struct sigaction sa_new, sa_old; sa_new.sa_handler = mpiSignalHandler; sa_new.sa_flags = 0; sigemptyset(&sa_new.sa_mask); if (sigaction(signum, NULL, &sa_old) < 0) { // ERROR: could not query old handler } else if (sa_old.sa_handler == SIG_IGN || sa_old.sa_handler == SIG_DFL) { // MPI hasn't set its own handler for this signal, so we // will install our own. if (sigaction(signum, &sa_new, NULL) < 0) { // ERROR: could not set new handler } } else { // MPI already has a handler installed for this signal. Do nothing } } From: Zhou, Hui <zhouh(a)anl.gov> Sent: Monday, April 6, 2020 2:08 PM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: [EXTERNAL] Re: [mpich-discuss] Abnormal termination on Linux Thanks, Kurt. I think the reason your signal trap didn’t work is because `mpiexec` is trapping it first. Segfault is code error. Why would you want to trap it? -- Hui Zhou From: "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Date: Monday, April 6, 2020 at 1:52 PM To: "Zhou, Hui" <zhouh(a)anl.gov<mailto:[email protected]>>, "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Subject: RE: [mpich-discuss] Abnormal termination on Linux Hui, Sorry for not mentioning that. MPICH 3.3.2 compiled with pgc++ 19.5. Kurt From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> Sent: Monday, April 6, 2020 12:58 PM To: discuss(a)mpich.org<mailto:[email protected]> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: [EXTERNAL] Re: [mpich-discuss] Abnormal termination on Linux Which version of MPICH were you running? -- Hui Zhou From: "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org<mailto:[email protected]>> Reply-To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Date: Monday, April 6, 2020 at 12:45 PM To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Cc: "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: Re: [mpich-discuss] Abnormal termination on Linux I should mention that I am unable to predict in which node or process the abnormal termination occurs, so I can’t practically attach a debugger and try to intercept the error. Kurt From: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Sent: Monday, April 6, 2020 11:50 AM To: discuss(a)mpich.org<mailto:[email protected]> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: Abnormal termination on Linux I have a couple of questions about abnormal termination. The EXIT CODE below is 11, which could be signal SIGSEGV, or is it something defined by MPICH? If it is SIGSEGV, it is strange because my signal handler isn’t catching it and cleaning up properly (the signal handler calls MPI_Finalize()). Is there any way to get more information about the location of the error? = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 14385 RUNNING AT n020.cluster.com = EXIT CODE: 11 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES Thanks, Kurt

1 0

Re: [mpich-discuss] Abnormal termination on Linux
by Zhou, Hui 06 Apr '20

06 Apr '20

Thanks, Kurt. I think the reason your signal trap didn’t work is because `mpiexec` is trapping it first. Segfault is code error. Why would you want to trap it? -- Hui Zhou From: "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> Date: Monday, April 6, 2020 at 1:52 PM To: "Zhou, Hui" <zhouh(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: RE: [mpich-discuss] Abnormal termination on Linux Hui, Sorry for not mentioning that. MPICH 3.3.2 compiled with pgc++ 19.5. Kurt From: Zhou, Hui <zhouh(a)anl.gov> Sent: Monday, April 6, 2020 12:58 PM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [EXTERNAL] Re: [mpich-discuss] Abnormal termination on Linux Which version of MPICH were you running? -- Hui Zhou From: "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org<mailto:[email protected]>> Reply-To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Date: Monday, April 6, 2020 at 12:45 PM To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Cc: "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: Re: [mpich-discuss] Abnormal termination on Linux I should mention that I am unable to predict in which node or process the abnormal termination occurs, so I can’t practically attach a debugger and try to intercept the error. Kurt From: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Sent: Monday, April 6, 2020 11:50 AM To: discuss(a)mpich.org<mailto:[email protected]> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: Abnormal termination on Linux I have a couple of questions about abnormal termination. The EXIT CODE below is 11, which could be signal SIGSEGV, or is it something defined by MPICH? If it is SIGSEGV, it is strange because my signal handler isn’t catching it and cleaning up properly (the signal handler calls MPI_Finalize()). Is there any way to get more information about the location of the error? = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 14385 RUNNING AT n020.cluster.com = EXIT CODE: 11 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES Thanks, Kurt

1 0

Re: [mpich-discuss] Abnormal termination on Linux
by Mccall, Kurt E. (MSFC-EV41) 06 Apr '20

06 Apr '20

Hui, Sorry for not mentioning that. MPICH 3.3.2 compiled with pgc++ 19.5. Kurt From: Zhou, Hui <zhouh(a)anl.gov> Sent: Monday, April 6, 2020 12:58 PM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [EXTERNAL] Re: [mpich-discuss] Abnormal termination on Linux Which version of MPICH were you running? -- Hui Zhou From: "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org<mailto:[email protected]>> Reply-To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Date: Monday, April 6, 2020 at 12:45 PM To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Cc: "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: Re: [mpich-discuss] Abnormal termination on Linux I should mention that I am unable to predict in which node or process the abnormal termination occurs, so I can’t practically attach a debugger and try to intercept the error. Kurt From: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Sent: Monday, April 6, 2020 11:50 AM To: discuss(a)mpich.org<mailto:[email protected]> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: Abnormal termination on Linux I have a couple of questions about abnormal termination. The EXIT CODE below is 11, which could be signal SIGSEGV, or is it something defined by MPICH? If it is SIGSEGV, it is strange because my signal handler isn’t catching it and cleaning up properly (the signal handler calls MPI_Finalize()). Is there any way to get more information about the location of the error? = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 14385 RUNNING AT n020.cluster.com = EXIT CODE: 11 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES Thanks, Kurt

1 0

Re: [mpich-discuss] Abnormal termination on Linux
by Zhou, Hui 06 Apr '20

06 Apr '20

Which version of MPICH were you running? -- Hui Zhou From: "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Monday, April 6, 2020 at 12:45 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> Subject: Re: [mpich-discuss] Abnormal termination on Linux I should mention that I am unable to predict in which node or process the abnormal termination occurs, so I can’t practically attach a debugger and try to intercept the error. Kurt From: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Sent: Monday, April 6, 2020 11:50 AM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: Abnormal termination on Linux I have a couple of questions about abnormal termination. The EXIT CODE below is 11, which could be signal SIGSEGV, or is it something defined by MPICH? If it is SIGSEGV, it is strange because my signal handler isn’t catching it and cleaning up properly (the signal handler calls MPI_Finalize()). Is there any way to get more information about the location of the error? = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 14385 RUNNING AT n020.cluster.com = EXIT CODE: 11 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES Thanks, Kurt

1 0

Re: [mpich-discuss] Abnormal termination on Linux
by Mccall, Kurt E. (MSFC-EV41) 06 Apr '20

06 Apr '20

I should mention that I am unable to predict in which node or process the abnormal termination occurs, so I can't practically attach a debugger and try to intercept the error. Kurt From: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Sent: Monday, April 6, 2020 11:50 AM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: Abnormal termination on Linux I have a couple of questions about abnormal termination. The EXIT CODE below is 11, which could be signal SIGSEGV, or is it something defined by MPICH? If it is SIGSEGV, it is strange because my signal handler isn't catching it and cleaning up properly (the signal handler calls MPI_Finalize()). Is there any way to get more information about the location of the error? = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 14385 RUNNING AT n020.cluster.com = EXIT CODE: 11 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES Thanks, Kurt

1 0

Abnormal termination on Linux
by Mccall, Kurt E. (MSFC-EV41) 06 Apr '20

06 Apr '20

I have a couple of questions about abnormal termination. The EXIT CODE below is 11, which could be signal SIGSEGV, or is it something defined by MPICH? If it is SIGSEGV, it is strange because my signal handler isn't catching it and cleaning up properly (the signal handler calls MPI_Finalize()). Is there any way to get more information about the location of the error? = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 14385 RUNNING AT n020.cluster.com = EXIT CODE: 11 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES Thanks, Kurt

1 0

Re: [mpich-discuss] MPICH INSTALLATION ERRORS
by Zhou, Hui 04 Apr '20

04 Apr '20

Are you building it on Windows? MPICH 3 series does not support Windows. Try Microsoft MPI : https://docs.microsoft.com/en-us/message-passing-interface/microsoft-mpi -- Hui Zhou From: Mbaabu Brian via discuss <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Saturday, April 4, 2020 at 8:42 AM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: Mbaabu Brian <mbaabubrian(a)gmail.com> Subject: [mpich-discuss] MPICH INSTALLATION ERRORS Hello, My installation produced such errors as are documented in the attached log files. Please help me resolve them.

1 0