- discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Should MPI_Abort() call exit or _exit?
by John Peterson 21 Apr '20

21 Apr '20

Hi, Thanks for your reply. Since in this case it's PETSc's PetscSignalHandlerDefault signal handler which actually calls MPI_Abort(), are you saying they should not do that? Any suggestion for what they should do instead? It seems that there is then no clean way of terminating an MPI process that receives SIGTERM? -- John On Tue, Apr 21, 2020 at 1:18 PM Zhou, Hui <zhouh(a)anl.gov> wrote: > Let’s make it clear – inside a signal handler, one is not allowed to call > any interrupt unsafe functions, that includes all MPI functions including > `MPI_Abort`, which means `MPL_exit` should never be called inside a signal > handler. Back to the question, since `MPL_exit` is not inside a signal > handler, it is allowed to call `exit`. > > > > I don’t think it is possible to do any real clean up inside a signal > handler. The best a signal handler can do is to flip some atomic flags, the > your applications should have checkpoints check those flags and do your > graceful exit if necessary. > > > > -- > Hui Zhou > > > > > > *From: *John Peterson via discuss <discuss(a)mpich.org> > *Reply-To: *"discuss(a)mpich.org" <discuss(a)mpich.org> > *Date: *Tuesday, April 21, 2020 at 1:06 PM > *To: *"discuss(a)mpich.org" <discuss(a)mpich.org> > *Cc: *John Peterson <jwpeterson(a)gmail.com> > *Subject: *[mpich-discuss] Should MPI_Abort() call exit or _exit? > > > > Hi, > > I have what is a fairly convoluted question involving several different > libraries, so I'll try to make it as succinct as possible. The issue is > that we have an MPI job that is canceled by slurm's "scancel", but instead > of exiting cleanly, the job (sometimes) hangs. I think we've tracked it > down to the job being canceled while in the middle of a call to "free" and > then "free" subsequently being called again from a function called by the > signal handler, which leads to the deadlock. The general rule seems to be > that only "asynchronous-safe" functions (abort(), _Exit(), etc.) are > allowed to be called in signal handlers [0]. > > > > In our specific case, the stack trace of one of the hung jobs is: > > > > #0 __lll_lock_wait_private () at > ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95 > #1 0x00007ff175223114 in _int_free (have_lock=0, p=0x55cd9120f2a0, > av=0x7ff175576c40 <main_arena>) at malloc.c:4266 > #2 __GI___libc_free (mem=0x55cd9120f2b0) at malloc.c:3124 > #3 0x00007ff1714938aa in H5MM_xfree () > #4 0x00007ff17147e5da in H5L_term_package () > #5 0x00007ff17133a766 in H5_term_library () > #6 0x00007ff1751ce041 in __run_exit_handlers (status=59, > listp=0x7ff175576718 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, > run_dtors=run_dtors@entry=true) at exit.c:108 > #7 0x00007ff1751ce13a in __GI_exit (status=<optimized out>) at exit.c:139 > #8 0x00007ff176f3c809 in MPL_exit () from > /usr/lib/x86_64-linux-gnu/libmpich.so.0 > #9 0x00007ff176eefd4c in ?? () from > /usr/lib/x86_64-linux-gnu/libmpich.so.0 > #10 0x00007ff176e3ea59 in PMPI_Abort () from > /usr/lib/x86_64-linux-gnu/libmpich.so.0 > #11 0x00007ff177635531 in PetscSignalHandlerDefault () > > #12 0x00007ff177635270 in PetscSignalHandler_Private () > #13 <signal handler called> > #14 0x00007ff175222c6f in _int_free (have_lock=0, p=0x55cd9345e360, > av=0x7ff175576c40 <main_arena>) at malloc.c:4280 > #15 __GI___libc_free (mem=0x55cd9345e370) at malloc.c:3124 > > > > To summarize this: > > 1.) We are in a call to "free" > > 2.) The process receives a sigterm/sigkill signal, which petsc handles > 3.) petsc calls mpich's abort function > 4.) mpich calls "exit", which causes the "atexit" functions to run > > 5.) HDF5 registers an "atexit" function which also calls "free" > > 6.) Deadlock > > > > This could definitely be seen as an HDF5 issue: I'm not sure of the wisdom > of registering "atexit" functions which free memory -- the program is > exiting after all. But, I also wanted to confirm whether calling exit() > from MPL_exit() is a deliberate choice, or if it could perhaps be changed > to _exit, which I think would avoid this particular problem. > > > > Thanks, > > John > > > > [0]: > https://wiki.sei.cmu.edu/confluence/display/c/SIG30-C.+Call+only+asynchrono… > > > > >

1 0

Re: [mpich-discuss] Should MPI_Abort() call exit or _exit?
by Zhou, Hui 21 Apr '20

21 Apr '20

Let’s make it clear – inside a signal handler, one is not allowed to call any interrupt unsafe functions, that includes all MPI functions including `MPI_Abort`, which means `MPL_exit` should never be called inside a signal handler. Back to the question, since `MPL_exit` is not inside a signal handler, it is allowed to call `exit`. I don’t think it is possible to do any real clean up inside a signal handler. The best a signal handler can do is to flip some atomic flags, the your applications should have checkpoints check those flags and do your graceful exit if necessary. -- Hui Zhou From: John Peterson via discuss <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Tuesday, April 21, 2020 at 1:06 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: John Peterson <jwpeterson(a)gmail.com> Subject: [mpich-discuss] Should MPI_Abort() call exit or _exit? Hi, I have what is a fairly convoluted question involving several different libraries, so I'll try to make it as succinct as possible. The issue is that we have an MPI job that is canceled by slurm's "scancel", but instead of exiting cleanly, the job (sometimes) hangs. I think we've tracked it down to the job being canceled while in the middle of a call to "free" and then "free" subsequently being called again from a function called by the signal handler, which leads to the deadlock. The general rule seems to be that only "asynchronous-safe" functions (abort(), _Exit(), etc.) are allowed to be called in signal handlers [0]. In our specific case, the stack trace of one of the hung jobs is: #0 __lll_lock_wait_private () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95 #1 0x00007ff175223114 in _int_free (have_lock=0, p=0x55cd9120f2a0, av=0x7ff175576c40 <main_arena>) at malloc.c:4266 #2 __GI___libc_free (mem=0x55cd9120f2b0) at malloc.c:3124 #3 0x00007ff1714938aa in H5MM_xfree () #4 0x00007ff17147e5da in H5L_term_package () #5 0x00007ff17133a766 in H5_term_library () #6 0x00007ff1751ce041 in __run_exit_handlers (status=59, listp=0x7ff175576718 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:108 #7 0x00007ff1751ce13a in __GI_exit (status=<optimized out>) at exit.c:139 #8 0x00007ff176f3c809 in MPL_exit () from /usr/lib/x86_64-linux-gnu/libmpich.so.0 #9 0x00007ff176eefd4c in ?? () from /usr/lib/x86_64-linux-gnu/libmpich.so.0 #10 0x00007ff176e3ea59 in PMPI_Abort () from /usr/lib/x86_64-linux-gnu/libmpich.so.0 #11 0x00007ff177635531 in PetscSignalHandlerDefault () #12 0x00007ff177635270 in PetscSignalHandler_Private () #13 <signal handler called> #14 0x00007ff175222c6f in _int_free (have_lock=0, p=0x55cd9345e360, av=0x7ff175576c40 <main_arena>) at malloc.c:4280 #15 __GI___libc_free (mem=0x55cd9345e370) at malloc.c:3124 To summarize this: 1.) We are in a call to "free" 2.) The process receives a sigterm/sigkill signal, which petsc handles 3.) petsc calls mpich's abort function 4.) mpich calls "exit", which causes the "atexit" functions to run 5.) HDF5 registers an "atexit" function which also calls "free" 6.) Deadlock This could definitely be seen as an HDF5 issue: I'm not sure of the wisdom of registering "atexit" functions which free memory -- the program is exiting after all. But, I also wanted to confirm whether calling exit() from MPL_exit() is a deliberate choice, or if it could perhaps be changed to _exit, which I think would avoid this particular problem. Thanks, John [0]: https://wiki.sei.cmu.edu/confluence/display/c/SIG30-C.+Call+only+asynchrono…

1 0

Should MPI_Abort() call exit or _exit?
by John Peterson 21 Apr '20

21 Apr '20

Hi, I have what is a fairly convoluted question involving several different libraries, so I'll try to make it as succinct as possible. The issue is that we have an MPI job that is canceled by slurm's "scancel", but instead of exiting cleanly, the job (sometimes) hangs. I think we've tracked it down to the job being canceled while in the middle of a call to "free" and then "free" subsequently being called again from a function called by the signal handler, which leads to the deadlock. The general rule seems to be that only "asynchronous-safe" functions (abort(), _Exit(), etc.) are allowed to be called in signal handlers [0]. In our specific case, the stack trace of one of the hung jobs is: #0 __lll_lock_wait_private () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95 #1 0x00007ff175223114 in _int_free (have_lock=0, p=0x55cd9120f2a0, av=0x7ff175576c40 <main_arena>) at malloc.c:4266 #2 __GI___libc_free (mem=0x55cd9120f2b0) at malloc.c:3124 #3 0x00007ff1714938aa in H5MM_xfree () #4 0x00007ff17147e5da in H5L_term_package () #5 0x00007ff17133a766 in H5_term_library () #6 0x00007ff1751ce041 in __run_exit_handlers (status=59, listp=0x7ff175576718 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:108 #7 0x00007ff1751ce13a in __GI_exit (status=<optimized out>) at exit.c:139 #8 0x00007ff176f3c809 in MPL_exit () from /usr/lib/x86_64-linux-gnu/libmpich.so.0 #9 0x00007ff176eefd4c in ?? () from /usr/lib/x86_64-linux-gnu/libmpich.so.0 #10 0x00007ff176e3ea59 in PMPI_Abort () from /usr/lib/x86_64-linux-gnu/libmpich.so.0 #11 0x00007ff177635531 in PetscSignalHandlerDefault () #12 0x00007ff177635270 in PetscSignalHandler_Private () #13 <signal handler called> #14 0x00007ff175222c6f in _int_free (have_lock=0, p=0x55cd9345e360, av=0x7ff175576c40 <main_arena>) at malloc.c:4280 #15 __GI___libc_free (mem=0x55cd9345e370) at malloc.c:3124 To summarize this: 1.) We are in a call to "free" 2.) The process receives a sigterm/sigkill signal, which petsc handles 3.) petsc calls mpich's abort function 4.) mpich calls "exit", which causes the "atexit" functions to run 5.) HDF5 registers an "atexit" function which also calls "free" 6.) Deadlock This could definitely be seen as an HDF5 issue: I'm not sure of the wisdom of registering "atexit" functions which free memory -- the program is exiting after all. But, I also wanted to confirm whether calling exit() from MPL_exit() is a deliberate choice, or if it could perhaps be changed to _exit, which I think would avoid this particular problem. Thanks, John [0]: https://wiki.sei.cmu.edu/confluence/display/c/SIG30-C.+Call+only+asynchrono…

1 0

Re: [mpich-discuss] MPICH configure
by Zhou, Hui 20 Apr '20

20 Apr '20

The error is from `hydra`, which should not have changed much between the versions. Could you verify that 3.3.1 still works for you? -- Hui Zhou From: "Palmer, Bruce J via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Thursday, April 16, 2020 at 5:48 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov> Subject: [mpich-discuss] MPICH configure Hi, I’ve been building MPICH on are aging Infiniband cluster using the following formula ./configure --prefix=/people/d3g293/mpich/mpich-3.3.2/install --with-device=ch4:ofi:sockets --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++ It’s been working pretty well but I recently tried to build mpich-3.3.2 and mpich-3.4a2 and although the build seems to work okay, I’m having problems actually running anything. If I run on 2 nodes the code seems to hang on MPI_Init and it looks like it is producing the error message [proxy:0:[email protected]] HYDU_sock_connect (utils/sock/sock.c:145): unable to connect from "node013.l ocal" to "node012.local" (Connection refused) [proxy:0:[email protected]] main (pm/pmiserv/pmip.c:183): unable to connect to server node012.local at p ort 37769 (check for firewalls!) srun: error: node013: task 1: Exited with exit code 5 If I run on a single node, things seem to work. Any idea what is going on here? I’ve got a working build of mpich-3.3, so things were okay up until recently. Has something in MPICH changed and my configuration formula is no good, or is this more likely to be due to some system modification? Bruce Palmer Senior Research Scientist Pacific Northwest National Laboratory Richland, WA 99352 (509) 375-3899

1 0

Running issue on CENTOS Cluster with SLURM
by Yaser Afshar 20 Apr '20

20 Apr '20

Hi, System: - AMD cluster (AMD Opteron(tm) Processor 6344) with InfiniPath_QLE7240 - CentOS Linux release 7.7.1908 (Core) Scheduler: - slurm MPICH version: 3.3.2 MPICH configuration: ./configure --prefix=/home/yaser/bin/mpich/3.3.2/intel CC=icc CXX=icpc FC=ifort --with-hwloc=/opt/HWLOC/2.2.0 --with-ucx=/opt/UCX/1.8.0 --with-knem=/opt/KNEM/1.1.3 --with-device=ch4:ucx --enable-mpi-cxx --enable-mpi1-compatibility --enable-threads=multiple --with-pmi --with-pm I am testing using the OSU microbenchmark ( http://mvapich.cse.ohio-state.edu/benchmarks) running the job on `srun` would succeed with no problem. ``` srun --mpi=pmi2 ./osu_bw ``` When I use the `mpiexec`, or `mpirun` it fails with: ``` [proxy:0:[email protected]] HYDU_sock_connect (utils/sock/sock.c:145): unable to connect from "pd-compute-3-40.local" to "pd-compute-1-6.local" (Connection refused) [proxy:0:[email protected]] main (pm/pmiserv/pmip.c:183): unable to connect to server pd-compute-1-6.local at port 42216 (check for firewalls!) srun: error: pd-compute-3-40: task 1: Exited with exit code 5 ``` The firewall is off, so it is not the reason. ``` > systemctl status firewalld ● firewalld.service Loaded: masked (/dev/null; bad) Active: inactive (dead) ``` I could not find any hint on MPICH FAQ nor anything useful anywhere else. Would you help me to resolve this issue? Many thanks.

1 0

MPICH configure
by Palmer, Bruce J 16 Apr '20

16 Apr '20

Hi, I’ve been building MPICH on are aging Infiniband cluster using the following formula ./configure --prefix=/people/d3g293/mpich/mpich-3.3.2/install --with-device=ch4:ofi:sockets --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++ It’s been working pretty well but I recently tried to build mpich-3.3.2 and mpich-3.4a2 and although the build seems to work okay, I’m having problems actually running anything. If I run on 2 nodes the code seems to hang on MPI_Init and it looks like it is producing the error message [proxy:0:[email protected]] HYDU_sock_connect (utils/sock/sock.c:145): unable to connect from "node013.l ocal" to "node012.local" (Connection refused) [proxy:0:[email protected]] main (pm/pmiserv/pmip.c:183): unable to connect to server node012.local at p ort 37769 (check for firewalls!) srun: error: node013: task 1: Exited with exit code 5 If I run on a single node, things seem to work. Any idea what is going on here? I’ve got a working build of mpich-3.3, so things were okay up until recently. Has something in MPICH changed and my configuration formula is no good, or is this more likely to be due to some system modification? Bruce Palmer Senior Research Scientist Pacific Northwest National Laboratory Richland, WA 99352 (509) 375-3899

1 0

Re: [mpich-discuss] Abnormal termination on Linux
by Zhou, Hui 07 Apr '20

07 Apr '20

I see. So I read a bit more. It appears that `exit()` is not interrupt-safe either. Let's try something basic first: in your signal handler, call `_exit(1)` or `_Exit(1)` , which should instantly terminate your process, no fuss. Don't call `MPI_Finalize` or any other non-safe functions before exit. Make sure that works for you; then to clean up states, you need work with your actual process and figure out synchronizations between your normal code and your signal handler, or even between multiple invocations of your signal handler -- remember, your code can be interrupted multiple times, any time, any place. Now, once the process abruptly terminates, the process manager will receive signal. The default action is to stop all processes. I guess that's why you want to call `MPI_Finalize`. -- I guess I talked myself into understanding what you are trying to do. __ Well the problem is `MPI_Finalize` can't run safely at any point -- imagine you are in the middle of an MPI operation that is changing some states half-way, and when `MPI_Finalize` try to cleanup, it may access memory that is freed by yet the pointer or state hasn't reset yet, which certainly will result in segfault. To make it work, you need some kind of coordination. A typical strategy is to raise a flag in your signal handler and have your main program check that flag frequently and gracefully exit when flag is raised. That way, you are always running `MPI_Finalize` outside MPI functions, which should work. But this won't work with segfaults. Segfaults is your code problem and won't go away. It will repeatedly segfault after your signal handler returns. But hopefully, the segfaults you are seeing is caused by your signal handler itself, in which case, the synchronized strategy should work. -- Hui Zhou On 4/7/20, 5:43 PM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Hui, I am definitely calling exit in my signal handler after calling MPI_Finalize. This has worked in the past for small jobs, but as I scale up the size of the job, I've been seeing strange errors.

1 0

Re: [mpich-discuss] Abnormal termination on Linux
by Mccall, Kurt E. (MSFC-EV41) 07 Apr '20

07 Apr '20

Hui, I am definitely calling exit in my signal handler after calling MPI_Finalize. This has worked in the past for small jobs, but as I scale up the size of the job, I've been seeing strange errors. -----Original Message----- From: Zhou, Hui <zhouh(a)anl.gov> Sent: Tuesday, April 7, 2020 5:35 PM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: [EXTERNAL] Re: [mpich-discuss] Abnormal termination on Linux > I'm using the methods in the paper https://urldefense.proofpoint.com/v2/url?u=https-3A__www.mcs.anl.gov_-7Elus… to make my Monte Carlo jobs fault tolerant (using inter-communicators rather than intra-communicators). The processes have a manager/worker arrangement, with the workers being single instances of a simulation, each with unique randomized inputs. These workers are created using MPI_Comm_spawn(). When a worker finishes, it must exit, and up to now I've been calling MPI_Finalize in the worker before exiting. > > From what you've said, this is incorrect because MPI_Finalize is collective over all of the processes in the job? Obviously my managers do not exit when the workers exit. Can you suggest any way to do cleanup before a worker exists, so that the job can continue? I think the first question should be answered is: what happens when you exit your signal handler? `MPI_Finalize` won't exit the process. I believe your process will simply try to continue at where it gets interrupted, which is the same place where It segfaults -- or maybe the segfaults is caused by your code continue trying to run MPI-operations but you have "Finalized" the state in the interrupt handler? Since you are going to exit your process, why don't you simply exit? I am not sure whether you are allowed to exit in signal handler -- curious to know 😊. But if exit works, you are leaving to the kernel to clean up your process, which may be sufficient for your goal. -- Hui Zhou

1 0

Re: [mpich-discuss] Abnormal termination on Linux
by Zhou, Hui 07 Apr '20

07 Apr '20

> I'm using the methods in the paper https://www.mcs.anl.gov/~lusk/papers/fault-tolerance.pdf to make my Monte Carlo jobs fault tolerant (using inter-communicators rather than intra-communicators). The processes have a manager/worker arrangement, with the workers being single instances of a simulation, each with unique randomized inputs. These workers are created using MPI_Comm_spawn(). When a worker finishes, it must exit, and up to now I've been calling MPI_Finalize in the worker before exiting. > > From what you've said, this is incorrect because MPI_Finalize is collective over all of the processes in the job? Obviously my managers do not exit when the workers exit. Can you suggest any way to do cleanup before a worker exists, so that the job can continue? I think the first question should be answered is: what happens when you exit your signal handler? `MPI_Finalize` won't exit the process. I believe your process will simply try to continue at where it gets interrupted, which is the same place where It segfaults -- or maybe the segfaults is caused by your code continue trying to run MPI-operations but you have "Finalized" the state in the interrupt handler? Since you are going to exit your process, why don't you simply exit? I am not sure whether you are allowed to exit in signal handler -- curious to know 😊. But if exit works, you are leaving to the kernel to clean up your process, which may be sufficient for your goal. -- Hui Zhou

1 0

Re: [mpich-discuss] Abnormal termination on Linux
by Mccall, Kurt E. (MSFC-EV41) 07 Apr '20

07 Apr '20

I'm using the methods in the paper https://www.mcs.anl.gov/~lusk/papers/fault-tolerance.pdf to make my Monte Carlo jobs fault tolerant (using inter-communicators rather than intra-communicators). The processes have a manager/worker arrangement, with the workers being single instances of a simulation, each with unique randomized inputs. These workers are created using MPI_Comm_spawn(). When a worker finishes, it must exit, and up to now I've been calling MPI_Finalize in the worker before exiting. >From what you've said, this is incorrect because MPI_Finalize is collective over all of the processes in the job? Obviously my managers do not exit when the workers exit. Can you suggest any way to do cleanup before a worker exists, so that the job can continue? Thanks, Kurt -----Original Message----- From: Joachim Protze <protze(a)itc.rwth-aachen.de> Sent: Tuesday, April 7, 2020 3:37 AM To: discuss(a)mpich.org; Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Cc: Zhou, Hui <zhouh(a)anl.gov> Subject: [EXTERNAL] Re: [mpich-discuss] Abnormal termination on Linux As long as you use a standard MPI implementation, there is no resilience and the MPI job will not survive a segfault or any other MPI error. Probably you should target your question towards "Why does my application segfault in the first place?" If it's only one process, that segfaults probably use the core file for debugging. By default I assume corefile size is limited on most clusters, so increase it in your batch job: ulimit -c unlimited gdb a.out -c corefile.hostname.timestamp backtrace quit Am 07.04.20 um 00:23 schrieb Zhou, Hui via discuss: > I see. MPICH doesn’t trap any signal as I checked 😊. So back to your issue, let’s assume your trap did work, but after the signal handler, which after it called `MPI_Finalize`, the code is still going to run the same code that causes segfault in the first place, right? Wouldn’t that endup in infinite signal loop or maybe the kernel smart out and by-passed your handler? Just a thought experiment. > Calling MPI_Finalize would not help, because MPI_Finalize is a collective call and will not finish before all other processes have called MPI_Finalize. Furthermore, in such error situation you won't have any guarantees, that the process does not have any open blocking or non-blocking communication. Such communication would need to be completed before calling MPI_Finalize. - Joachim > Beside, MPICH is thread-safe but not interrupt safe. So you should not really call MPI functions inside signal handlers. > > -- > Hui Zhou > > > From: "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> > Date: Monday, April 6, 2020 at 4:50 PM > To: "Zhou, Hui" <zhouh(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> > Subject: RE: [mpich-discuss] Abnormal termination on Linux > > Hui, > > I’d like to trap segfaults so that the process that raised them can be shut down gracefully/finalized without taking down the whole MPI job. Maybe you are right and I shouldn’t trap them. > > However, when I install my signal handlers, I check if MPI already has a signal handler installed for each of them. It isn’t reporting that MPI has done so. Perhaps my code is incorrect? > > setUpOneHandler(const int signum) > { > struct sigaction sa_new, sa_old; > > sa_new.sa_handler = mpiSignalHandler; > sa_new.sa_flags = 0; > sigemptyset(&sa_new.sa_mask); > > if (sigaction(signum, NULL, &sa_old) < 0) > { > // ERROR: could not query old handler > > } else if (sa_old.sa_handler == SIG_IGN || > sa_old.sa_handler == SIG_DFL) > { > // MPI hasn't set its own handler for this signal, so we > // will install our own. > > if (sigaction(signum, &sa_new, NULL) < 0) > { > // ERROR: could not set new handler > } > } else > { > // MPI already has a handler installed for this signal. Do nothing > } > } > > > > > From: Zhou, Hui <zhouh(a)anl.gov> > Sent: Monday, April 6, 2020 2:08 PM > To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org > Subject: [EXTERNAL] Re: [mpich-discuss] Abnormal termination on Linux > > Thanks, Kurt. I think the reason your signal trap didn’t work is because `mpiexec` is trapping it first. Segfault is code error. Why would you want to trap it? > > -- > Hui Zhou > > > From: "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> > Date: Monday, April 6, 2020 at 1:52 PM > To: "Zhou, Hui" <zhouh(a)anl.gov<mailto:[email protected]>>, "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> > Subject: RE: [mpich-discuss] Abnormal termination on Linux > > Hui, > > Sorry for not mentioning that. MPICH 3.3.2 compiled with pgc++ 19.5. > > Kurt > > From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> > Sent: Monday, April 6, 2020 12:58 PM > To: discuss(a)mpich.org<mailto:[email protected]> > Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> > Subject: [EXTERNAL] Re: [mpich-discuss] Abnormal termination on Linux > > Which version of MPICH were you running? > > -- > Hui Zhou > > > From: "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org<mailto:[email protected]>> > Reply-To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> > Date: Monday, April 6, 2020 at 12:45 PM > To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> > Cc: "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> > Subject: Re: [mpich-discuss] Abnormal termination on Linux > > I should mention that I am unable to predict in which node or process the abnormal termination occurs, so I can’t practically attach a debugger and try to intercept the error. > > Kurt > > From: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> > Sent: Monday, April 6, 2020 11:50 AM > To: discuss(a)mpich.org<mailto:[email protected]> > Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> > Subject: Abnormal termination on Linux > > I have a couple of questions about abnormal termination. The EXIT CODE below is 11, which could be signal SIGSEGV, or is it something defined by MPICH? If it is SIGSEGV, it is strange because my signal handler isn’t catching it and cleaning up properly (the signal handler calls MPI_Finalize()). Is there any way to get more information about the location of the error? > > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > = PID 14385 RUNNING AT n020.cluster.com > = EXIT CODE: 11 > = CLEANING UP REMAINING PROCESSES > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > > Thanks, > Kurt > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Dipl.-Inf. Joachim Protze IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 protze(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0