Hui,

I’d like to trap segfaults so that the process that raised them can be shut down gracefully/finalized without taking down the whole MPI job.   Maybe you are right and  I shouldn’t trap them.

However, when I install my signal handlers, I check if MPI already has a signal handler installed for each of them.   It isn’t reporting that MPI has done so.  Perhaps my code is incorrect?

setUpOneHandler(const int signum)

{

    struct sigaction sa_new, sa_old;

    sa_new.sa_handler = mpiSignalHandler;

    sa_new.sa_flags = 0;

    sigemptyset(&sa_new.sa_mask);

    if (sigaction(signum, NULL, &sa_old) < 0)

    {

          // ERROR: could not query old handler

    } else if (sa_old.sa_handler == SIG_IGN ||

                  sa_old.sa_handler == SIG_DFL)

    {

        // MPI hasn't set its own handler for this signal, so we

        // will install our own.

        if (sigaction(signum, &sa_new, NULL) < 0)

        {

            // ERROR: could not set new handler

        }

    } else

    {

        // MPI already has a handler installed for this signal.  Do nothing

    }

}

From: Zhou, Hui <zhouh@anl.gov>
Sent: Monday, April 6, 2020 2:08 PM
To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall@nasa.gov>; discuss@mpich.org
Subject: [EXTERNAL] Re: [mpich-discuss] Abnormal termination on Linux

Thanks, Kurt. I think the reason your signal trap didn’t work is because `mpiexec` is trapping it first. Segfault is code error. Why would you want to trap it?

-- 
Hui Zhou

From: "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall@nasa.gov>
Date: Monday, April 6, 2020 at 1:52 PM
To: "Zhou, Hui" <zhouh@anl.gov>, "discuss@mpich.org" <discuss@mpich.org>
Subject: RE: [mpich-discuss] Abnormal termination on Linux

Hui,

Sorry for not mentioning that.  MPICH 3.3.2 compiled with pgc++ 19.5.

Kurt

From: Zhou, Hui <zhouh@anl.gov>
Sent: Monday, April 6, 2020 12:58 PM
To: discuss@mpich.org
Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall@nasa.gov>
Subject: [EXTERNAL] Re: [mpich-discuss] Abnormal termination on Linux

Which version of MPICH were you running?

-- 
Hui Zhou

From: "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss@mpich.org>
Reply-To: "discuss@mpich.org" <discuss@mpich.org>
Date: Monday, April 6, 2020 at 12:45 PM
To: "discuss@mpich.org" <discuss@mpich.org>
Cc: "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall@nasa.gov>
Subject: Re: [mpich-discuss] Abnormal termination on Linux

I should mention that I am unable to predict in which node or process the abnormal termination occurs, so I can’t practically attach a debugger and try to intercept the error.

Kurt

From: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall@nasa.gov>
Sent: Monday, April 6, 2020 11:50 AM
To: discuss@mpich.org
Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall@nasa.gov>
Subject: Abnormal termination on Linux

I have a couple of questions about abnormal termination.   The EXIT CODE below is 11, which could be signal SIGSEGV, or is it something defined by MPICH?  If it is SIGSEGV, it is strange because my signal handler isn’t catching it and cleaning up properly (the signal handler calls MPI_Finalize()).   Is there any way to get more information about the location of the error?

=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES

=   PID 14385 RUNNING AT n020.cluster.com

=   EXIT CODE: 11

=   CLEANING UP REMAINING PROCESSES

=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES

Thanks,

Kurt