April 2020 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Preloading the libraries "LD_PRELOAD" Effect
by Zhou, Hui 22 Apr '20

22 Apr '20

HI Kihang, “LD_PRELOAD” mainly is used to tell dynamic linker to pre-bind symbols rather than to cache the loading. I don’t think it can gain you the benefits as you described. As for tips for accelerating parallel programs, that is too broad a topic. Any tips I gave here would be just some random tips that may or may not be relevant to you. If you describe your actual applications, we may be able to give some more specific tips. -- Hui Zhou From: Kihang Youn via discuss <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Tuesday, April 21, 2020 at 11:26 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: Kihang Youn <kyoun(a)lenovo.com> Subject: [mpich-discuss] Preloading the libraries "LD_PRELOAD" Effect Hi, everyone. I'd like to know about the MPICH mpirun environment option "LD_PRELOAD". Is the function of this option used to put the library on Cache (or RAM?) at startup and to gain benefits from multiple calls? 3. For Fortran programs, you must also modify LD_PRELOAD export LD_PRELOAD=/path/to/intel/lib/libmpi.so:/path/to/intel/lib/libmpigf.so:$LD_PRELOAD Our team is looking for a way to load the library used by programs running through the "vmtouch" into the cache when performing the model, and we would like to know if it can be implemented through the "LD_PRELOAD" option. The second question may be a lazy one. Is there any options on tuning factor that can adjust parallel performance with MPICH? I'm actually only familiar with using MPI because I'm a newbie and I don't know much about performance enhancement, but I'm going to organize the list of recommendable performance-related options with MPICH and test them, so I'd like to know if there's anything organized(PDF or URL or something). Thank you. Kihang Youn(윤기항) - Application Analyst | Lenovo DCG Professional Services Mobile: +82-10-9374-9396 E-mail: kyoun(a)lenovo.com

1 0

Re: [mpich-discuss] MPICH configure
by Zhou, Hui 22 Apr '20

22 Apr '20

Hi Bruce, Thanks for the effort checking these versions. What is the slurm versions that you have on cluster? -- Hui Zhou From: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov> Date: Wednesday, April 22, 2020 at 11:02 AM To: "Zhou, Hui" <zhouh(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH configure Hi Hui, I rebuilt everything from scratch and tried running several versions of mpich. Release 3.3.1 seems to work okay but 3.3.2 hangs. Here is a complete summary of the versions I ran 3.3rc1: Works 3.3: Works 3.3.1: Works 3.3.2: Hangs 3.4a2: Hangs I’m not seeing the error message from hydra anymore (I have no idea why not), but I logged into one of the hung processes when running with 3.3.2 and got the following listing from gdb (gdb) where #0 0x0000003d7ce0e810 in __read_nocancel () from /lib64/libpthread.so.0 #1 0x00002aaaac6e936e in PMIU_readline () from /people/d3g293/mpich/mpich-3.3.2/install/lib/libmpi.so.12 #2 0x00002aaaac6e985b in GetResponse.part.0 () from /people/d3g293/mpich/mpich-3.3.2/install/lib/libmpi.so.12 #3 0x00002aaaac6e4e36 in MPIDU_shm_seg_commit () from /people/d3g293/mpich/mpich-3.3.2/install/lib/libmpi.so.12 #4 0x00002aaaabc541dc in MPIR_Init_thread () from /people/d3g293/mpich/mpich-3.3.2/install/lib/libmpi.so.12 #5 0x00002aaaabc3db7e in PMPI_Init () from /people/d3g293/mpich/mpich-3.3.2/install/lib/libmpi.so.12 #6 0x0000000000408d0e in main () Bruce From: "Zhou, Hui" <zhouh(a)anl.gov> Date: Monday, April 20, 2020 at 10:25 AM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov> Subject: Re: [mpich-discuss] MPICH configure The error is from `hydra`, which should not have changed much between the versions. Could you verify that 3.3.1 still works for you? -- Hui Zhou From: "Palmer, Bruce J via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Thursday, April 16, 2020 at 5:48 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov> Subject: [mpich-discuss] MPICH configure Hi, I’ve been building MPICH on are aging Infiniband cluster using the following formula ./configure --prefix=/people/d3g293/mpich/mpich-3.3.2/install --with-device=ch4:ofi:sockets --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++ It’s been working pretty well but I recently tried to build mpich-3.3.2 and mpich-3.4a2 and although the build seems to work okay, I’m having problems actually running anything. If I run on 2 nodes the code seems to hang on MPI_Init and it looks like it is producing the error message [proxy:0:[email protected]] HYDU_sock_connect (utils/sock/sock.c:145): unable to connect from "node013.l ocal" to "node012.local" (Connection refused) [proxy:0:[email protected]] main (pm/pmiserv/pmip.c:183): unable to connect to server node012.local at p ort 37769 (check for firewalls!) srun: error: node013: task 1: Exited with exit code 5 If I run on a single node, things seem to work. Any idea what is going on here? I’ve got a working build of mpich-3.3, so things were okay up until recently. Has something in MPICH changed and my configuration formula is no good, or is this more likely to be due to some system modification? Bruce Palmer Senior Research Scientist Pacific Northwest National Laboratory Richland, WA 99352 (509) 375-3899

1 0

Re: [mpich-discuss] MPICH configure
by Palmer, Bruce J 22 Apr '20

22 Apr '20

Hi Hui, I rebuilt everything from scratch and tried running several versions of mpich. Release 3.3.1 seems to work okay but 3.3.2 hangs. Here is a complete summary of the versions I ran 3.3rc1: Works 3.3: Works 3.3.1: Works 3.3.2: Hangs 3.4a2: Hangs I’m not seeing the error message from hydra anymore (I have no idea why not), but I logged into one of the hung processes when running with 3.3.2 and got the following listing from gdb (gdb) where #0 0x0000003d7ce0e810 in __read_nocancel () from /lib64/libpthread.so.0 #1 0x00002aaaac6e936e in PMIU_readline () from /people/d3g293/mpich/mpich-3.3.2/install/lib/libmpi.so.12 #2 0x00002aaaac6e985b in GetResponse.part.0 () from /people/d3g293/mpich/mpich-3.3.2/install/lib/libmpi.so.12 #3 0x00002aaaac6e4e36 in MPIDU_shm_seg_commit () from /people/d3g293/mpich/mpich-3.3.2/install/lib/libmpi.so.12 #4 0x00002aaaabc541dc in MPIR_Init_thread () from /people/d3g293/mpich/mpich-3.3.2/install/lib/libmpi.so.12 #5 0x00002aaaabc3db7e in PMPI_Init () from /people/d3g293/mpich/mpich-3.3.2/install/lib/libmpi.so.12 #6 0x0000000000408d0e in main () Bruce From: "Zhou, Hui" <zhouh(a)anl.gov> Date: Monday, April 20, 2020 at 10:25 AM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov> Subject: Re: [mpich-discuss] MPICH configure The error is from `hydra`, which should not have changed much between the versions. Could you verify that 3.3.1 still works for you? -- Hui Zhou From: "Palmer, Bruce J via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Thursday, April 16, 2020 at 5:48 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov> Subject: [mpich-discuss] MPICH configure Hi, I’ve been building MPICH on are aging Infiniband cluster using the following formula ./configure --prefix=/people/d3g293/mpich/mpich-3.3.2/install --with-device=ch4:ofi:sockets --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++ It’s been working pretty well but I recently tried to build mpich-3.3.2 and mpich-3.4a2 and although the build seems to work okay, I’m having problems actually running anything. If I run on 2 nodes the code seems to hang on MPI_Init and it looks like it is producing the error message [proxy:0:[email protected]] HYDU_sock_connect (utils/sock/sock.c:145): unable to connect from "node013.l ocal" to "node012.local" (Connection refused) [proxy:0:[email protected]] main (pm/pmiserv/pmip.c:183): unable to connect to server node012.local at p ort 37769 (check for firewalls!) srun: error: node013: task 1: Exited with exit code 5 If I run on a single node, things seem to work. Any idea what is going on here? I’ve got a working build of mpich-3.3, so things were okay up until recently. Has something in MPICH changed and my configuration formula is no good, or is this more likely to be due to some system modification? Bruce Palmer Senior Research Scientist Pacific Northwest National Laboratory Richland, WA 99352 (509) 375-3899

1 0

Preloading the libraries "LD_PRELOAD" Effect
by Kihang Youn 21 Apr '20

21 Apr '20

Hi, everyone. I'd like to know about the MPICH mpirun environment option "LD_PRELOAD". Is the function of this option used to put the library on Cache (or RAM?) at startup and to gain benefits from multiple calls? 3. For Fortran programs, you must also modify LD_PRELOAD export LD_PRELOAD=/path/to/intel/lib/libmpi.so:/path/to/intel/lib/libmpigf.so:$LD_PRELOAD Our team is looking for a way to load the library used by programs running through the "vmtouch" into the cache when performing the model, and we would like to know if it can be implemented through the "LD_PRELOAD" option. The second question may be a lazy one. Is there any options on tuning factor that can adjust parallel performance with MPICH? I'm actually only familiar with using MPI because I'm a newbie and I don't know much about performance enhancement, but I'm going to organize the list of recommendable performance-related options with MPICH and test them, so I'd like to know if there's anything organized(PDF or URL or something). Thank you. Kihang Youn(윤기항) - Application Analyst | Lenovo DCG Professional Services Mobile: +82-10-9374-9396 E-mail: kyoun(a)lenovo.com

1 0

Re: [mpich-discuss] Should MPI_Abort() call exit or _exit?
by Zhou, Hui 21 Apr '20

21 Apr '20

Yeah, I think `PETSc` should not really do that. Depend on the applications and process manager, calling `_exit` in the signal handler maybe clean enough – especially when the SIGTERM is sent by the process manager so it doesn’t need the process tell it back that “I’m aborting”. -- Hui Zhou From: John Peterson <jwpeterson(a)gmail.com> Date: Tuesday, April 21, 2020 at 2:06 PM To: "Zhou, Hui" <zhouh(a)anl.gov> Cc: "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] Should MPI_Abort() call exit or _exit? Hi, Thanks for your reply. Since in this case it's PETSc's PetscSignalHandlerDefault signal handler which actually calls MPI_Abort(), are you saying they should not do that? Any suggestion for what they should do instead? It seems that there is then no clean way of terminating an MPI process that receives SIGTERM? -- John On Tue, Apr 21, 2020 at 1:18 PM Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> wrote: Let’s make it clear – inside a signal handler, one is not allowed to call any interrupt unsafe functions, that includes all MPI functions including `MPI_Abort`, which means `MPL_exit` should never be called inside a signal handler. Back to the question, since `MPL_exit` is not inside a signal handler, it is allowed to call `exit`. I don’t think it is possible to do any real clean up inside a signal handler. The best a signal handler can do is to flip some atomic flags, the your applications should have checkpoints check those flags and do your graceful exit if necessary. -- Hui Zhou From: John Peterson via discuss <discuss(a)mpich.org<mailto:[email protected]>> Reply-To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Date: Tuesday, April 21, 2020 at 1:06 PM To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Cc: John Peterson <jwpeterson(a)gmail.com<mailto:[email protected]>> Subject: [mpich-discuss] Should MPI_Abort() call exit or _exit? Hi, I have what is a fairly convoluted question involving several different libraries, so I'll try to make it as succinct as possible. The issue is that we have an MPI job that is canceled by slurm's "scancel", but instead of exiting cleanly, the job (sometimes) hangs. I think we've tracked it down to the job being canceled while in the middle of a call to "free" and then "free" subsequently being called again from a function called by the signal handler, which leads to the deadlock. The general rule seems to be that only "asynchronous-safe" functions (abort(), _Exit(), etc.) are allowed to be called in signal handlers [0]. In our specific case, the stack trace of one of the hung jobs is: #0 __lll_lock_wait_private () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95 #1 0x00007ff175223114 in _int_free (have_lock=0, p=0x55cd9120f2a0, av=0x7ff175576c40 <main_arena>) at malloc.c:4266 #2 __GI___libc_free (mem=0x55cd9120f2b0) at malloc.c:3124 #3 0x00007ff1714938aa in H5MM_xfree () #4 0x00007ff17147e5da in H5L_term_package () #5 0x00007ff17133a766 in H5_term_library () #6 0x00007ff1751ce041 in __run_exit_handlers (status=59, listp=0x7ff175576718 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:108 #7 0x00007ff1751ce13a in __GI_exit (status=<optimized out>) at exit.c:139 #8 0x00007ff176f3c809 in MPL_exit () from /usr/lib/x86_64-linux-gnu/libmpich.so.0 #9 0x00007ff176eefd4c in ?? () from /usr/lib/x86_64-linux-gnu/libmpich.so.0 #10 0x00007ff176e3ea59 in PMPI_Abort () from /usr/lib/x86_64-linux-gnu/libmpich.so.0 #11 0x00007ff177635531 in PetscSignalHandlerDefault () #12 0x00007ff177635270 in PetscSignalHandler_Private () #13 <signal handler called> #14 0x00007ff175222c6f in _int_free (have_lock=0, p=0x55cd9345e360, av=0x7ff175576c40 <main_arena>) at malloc.c:4280 #15 __GI___libc_free (mem=0x55cd9345e370) at malloc.c:3124 To summarize this: 1.) We are in a call to "free" 2.) The process receives a sigterm/sigkill signal, which petsc handles 3.) petsc calls mpich's abort function 4.) mpich calls "exit", which causes the "atexit" functions to run 5.) HDF5 registers an "atexit" function which also calls "free" 6.) Deadlock This could definitely be seen as an HDF5 issue: I'm not sure of the wisdom of registering "atexit" functions which free memory -- the program is exiting after all. But, I also wanted to confirm whether calling exit() from MPL_exit() is a deliberate choice, or if it could perhaps be changed to _exit, which I think would avoid this particular problem. Thanks, John [0]: https://wiki.sei.cmu.edu/confluence/display/c/SIG30-C.+Call+only+asynchrono…

1 0

Re: [mpich-discuss] Should MPI_Abort() call exit or _exit?
by John Peterson 21 Apr '20

21 Apr '20

Hi, Thanks for your reply. Since in this case it's PETSc's PetscSignalHandlerDefault signal handler which actually calls MPI_Abort(), are you saying they should not do that? Any suggestion for what they should do instead? It seems that there is then no clean way of terminating an MPI process that receives SIGTERM? -- John On Tue, Apr 21, 2020 at 1:18 PM Zhou, Hui <zhouh(a)anl.gov> wrote: > Let’s make it clear – inside a signal handler, one is not allowed to call > any interrupt unsafe functions, that includes all MPI functions including > `MPI_Abort`, which means `MPL_exit` should never be called inside a signal > handler. Back to the question, since `MPL_exit` is not inside a signal > handler, it is allowed to call `exit`. > > > > I don’t think it is possible to do any real clean up inside a signal > handler. The best a signal handler can do is to flip some atomic flags, the > your applications should have checkpoints check those flags and do your > graceful exit if necessary. > > > > -- > Hui Zhou > > > > > > *From: *John Peterson via discuss <discuss(a)mpich.org> > *Reply-To: *"discuss(a)mpich.org" <discuss(a)mpich.org> > *Date: *Tuesday, April 21, 2020 at 1:06 PM > *To: *"discuss(a)mpich.org" <discuss(a)mpich.org> > *Cc: *John Peterson <jwpeterson(a)gmail.com> > *Subject: *[mpich-discuss] Should MPI_Abort() call exit or _exit? > > > > Hi, > > I have what is a fairly convoluted question involving several different > libraries, so I'll try to make it as succinct as possible. The issue is > that we have an MPI job that is canceled by slurm's "scancel", but instead > of exiting cleanly, the job (sometimes) hangs. I think we've tracked it > down to the job being canceled while in the middle of a call to "free" and > then "free" subsequently being called again from a function called by the > signal handler, which leads to the deadlock. The general rule seems to be > that only "asynchronous-safe" functions (abort(), _Exit(), etc.) are > allowed to be called in signal handlers [0]. > > > > In our specific case, the stack trace of one of the hung jobs is: > > > > #0 __lll_lock_wait_private () at > ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95 > #1 0x00007ff175223114 in _int_free (have_lock=0, p=0x55cd9120f2a0, > av=0x7ff175576c40 <main_arena>) at malloc.c:4266 > #2 __GI___libc_free (mem=0x55cd9120f2b0) at malloc.c:3124 > #3 0x00007ff1714938aa in H5MM_xfree () > #4 0x00007ff17147e5da in H5L_term_package () > #5 0x00007ff17133a766 in H5_term_library () > #6 0x00007ff1751ce041 in __run_exit_handlers (status=59, > listp=0x7ff175576718 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, > run_dtors=run_dtors@entry=true) at exit.c:108 > #7 0x00007ff1751ce13a in __GI_exit (status=<optimized out>) at exit.c:139 > #8 0x00007ff176f3c809 in MPL_exit () from > /usr/lib/x86_64-linux-gnu/libmpich.so.0 > #9 0x00007ff176eefd4c in ?? () from > /usr/lib/x86_64-linux-gnu/libmpich.so.0 > #10 0x00007ff176e3ea59 in PMPI_Abort () from > /usr/lib/x86_64-linux-gnu/libmpich.so.0 > #11 0x00007ff177635531 in PetscSignalHandlerDefault () > > #12 0x00007ff177635270 in PetscSignalHandler_Private () > #13 <signal handler called> > #14 0x00007ff175222c6f in _int_free (have_lock=0, p=0x55cd9345e360, > av=0x7ff175576c40 <main_arena>) at malloc.c:4280 > #15 __GI___libc_free (mem=0x55cd9345e370) at malloc.c:3124 > > > > To summarize this: > > 1.) We are in a call to "free" > > 2.) The process receives a sigterm/sigkill signal, which petsc handles > 3.) petsc calls mpich's abort function > 4.) mpich calls "exit", which causes the "atexit" functions to run > > 5.) HDF5 registers an "atexit" function which also calls "free" > > 6.) Deadlock > > > > This could definitely be seen as an HDF5 issue: I'm not sure of the wisdom > of registering "atexit" functions which free memory -- the program is > exiting after all. But, I also wanted to confirm whether calling exit() > from MPL_exit() is a deliberate choice, or if it could perhaps be changed > to _exit, which I think would avoid this particular problem. > > > > Thanks, > > John > > > > [0]: > https://wiki.sei.cmu.edu/confluence/display/c/SIG30-C.+Call+only+asynchrono… > > > > >

1 0

Re: [mpich-discuss] Should MPI_Abort() call exit or _exit?
by Zhou, Hui 21 Apr '20

21 Apr '20

Let’s make it clear – inside a signal handler, one is not allowed to call any interrupt unsafe functions, that includes all MPI functions including `MPI_Abort`, which means `MPL_exit` should never be called inside a signal handler. Back to the question, since `MPL_exit` is not inside a signal handler, it is allowed to call `exit`. I don’t think it is possible to do any real clean up inside a signal handler. The best a signal handler can do is to flip some atomic flags, the your applications should have checkpoints check those flags and do your graceful exit if necessary. -- Hui Zhou From: John Peterson via discuss <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Tuesday, April 21, 2020 at 1:06 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: John Peterson <jwpeterson(a)gmail.com> Subject: [mpich-discuss] Should MPI_Abort() call exit or _exit? Hi, I have what is a fairly convoluted question involving several different libraries, so I'll try to make it as succinct as possible. The issue is that we have an MPI job that is canceled by slurm's "scancel", but instead of exiting cleanly, the job (sometimes) hangs. I think we've tracked it down to the job being canceled while in the middle of a call to "free" and then "free" subsequently being called again from a function called by the signal handler, which leads to the deadlock. The general rule seems to be that only "asynchronous-safe" functions (abort(), _Exit(), etc.) are allowed to be called in signal handlers [0]. In our specific case, the stack trace of one of the hung jobs is: #0 __lll_lock_wait_private () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95 #1 0x00007ff175223114 in _int_free (have_lock=0, p=0x55cd9120f2a0, av=0x7ff175576c40 <main_arena>) at malloc.c:4266 #2 __GI___libc_free (mem=0x55cd9120f2b0) at malloc.c:3124 #3 0x00007ff1714938aa in H5MM_xfree () #4 0x00007ff17147e5da in H5L_term_package () #5 0x00007ff17133a766 in H5_term_library () #6 0x00007ff1751ce041 in __run_exit_handlers (status=59, listp=0x7ff175576718 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:108 #7 0x00007ff1751ce13a in __GI_exit (status=<optimized out>) at exit.c:139 #8 0x00007ff176f3c809 in MPL_exit () from /usr/lib/x86_64-linux-gnu/libmpich.so.0 #9 0x00007ff176eefd4c in ?? () from /usr/lib/x86_64-linux-gnu/libmpich.so.0 #10 0x00007ff176e3ea59 in PMPI_Abort () from /usr/lib/x86_64-linux-gnu/libmpich.so.0 #11 0x00007ff177635531 in PetscSignalHandlerDefault () #12 0x00007ff177635270 in PetscSignalHandler_Private () #13 <signal handler called> #14 0x00007ff175222c6f in _int_free (have_lock=0, p=0x55cd9345e360, av=0x7ff175576c40 <main_arena>) at malloc.c:4280 #15 __GI___libc_free (mem=0x55cd9345e370) at malloc.c:3124 To summarize this: 1.) We are in a call to "free" 2.) The process receives a sigterm/sigkill signal, which petsc handles 3.) petsc calls mpich's abort function 4.) mpich calls "exit", which causes the "atexit" functions to run 5.) HDF5 registers an "atexit" function which also calls "free" 6.) Deadlock This could definitely be seen as an HDF5 issue: I'm not sure of the wisdom of registering "atexit" functions which free memory -- the program is exiting after all. But, I also wanted to confirm whether calling exit() from MPL_exit() is a deliberate choice, or if it could perhaps be changed to _exit, which I think would avoid this particular problem. Thanks, John [0]: https://wiki.sei.cmu.edu/confluence/display/c/SIG30-C.+Call+only+asynchrono…

1 0

Should MPI_Abort() call exit or _exit?
by John Peterson 21 Apr '20

21 Apr '20

Hi, I have what is a fairly convoluted question involving several different libraries, so I'll try to make it as succinct as possible. The issue is that we have an MPI job that is canceled by slurm's "scancel", but instead of exiting cleanly, the job (sometimes) hangs. I think we've tracked it down to the job being canceled while in the middle of a call to "free" and then "free" subsequently being called again from a function called by the signal handler, which leads to the deadlock. The general rule seems to be that only "asynchronous-safe" functions (abort(), _Exit(), etc.) are allowed to be called in signal handlers [0]. In our specific case, the stack trace of one of the hung jobs is: #0 __lll_lock_wait_private () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95 #1 0x00007ff175223114 in _int_free (have_lock=0, p=0x55cd9120f2a0, av=0x7ff175576c40 <main_arena>) at malloc.c:4266 #2 __GI___libc_free (mem=0x55cd9120f2b0) at malloc.c:3124 #3 0x00007ff1714938aa in H5MM_xfree () #4 0x00007ff17147e5da in H5L_term_package () #5 0x00007ff17133a766 in H5_term_library () #6 0x00007ff1751ce041 in __run_exit_handlers (status=59, listp=0x7ff175576718 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:108 #7 0x00007ff1751ce13a in __GI_exit (status=<optimized out>) at exit.c:139 #8 0x00007ff176f3c809 in MPL_exit () from /usr/lib/x86_64-linux-gnu/libmpich.so.0 #9 0x00007ff176eefd4c in ?? () from /usr/lib/x86_64-linux-gnu/libmpich.so.0 #10 0x00007ff176e3ea59 in PMPI_Abort () from /usr/lib/x86_64-linux-gnu/libmpich.so.0 #11 0x00007ff177635531 in PetscSignalHandlerDefault () #12 0x00007ff177635270 in PetscSignalHandler_Private () #13 <signal handler called> #14 0x00007ff175222c6f in _int_free (have_lock=0, p=0x55cd9345e360, av=0x7ff175576c40 <main_arena>) at malloc.c:4280 #15 __GI___libc_free (mem=0x55cd9345e370) at malloc.c:3124 To summarize this: 1.) We are in a call to "free" 2.) The process receives a sigterm/sigkill signal, which petsc handles 3.) petsc calls mpich's abort function 4.) mpich calls "exit", which causes the "atexit" functions to run 5.) HDF5 registers an "atexit" function which also calls "free" 6.) Deadlock This could definitely be seen as an HDF5 issue: I'm not sure of the wisdom of registering "atexit" functions which free memory -- the program is exiting after all. But, I also wanted to confirm whether calling exit() from MPL_exit() is a deliberate choice, or if it could perhaps be changed to _exit, which I think would avoid this particular problem. Thanks, John [0]: https://wiki.sei.cmu.edu/confluence/display/c/SIG30-C.+Call+only+asynchrono…

1 0

Re: [mpich-discuss] MPICH configure
by Zhou, Hui 20 Apr '20

20 Apr '20

The error is from `hydra`, which should not have changed much between the versions. Could you verify that 3.3.1 still works for you? -- Hui Zhou From: "Palmer, Bruce J via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Thursday, April 16, 2020 at 5:48 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov> Subject: [mpich-discuss] MPICH configure Hi, I’ve been building MPICH on are aging Infiniband cluster using the following formula ./configure --prefix=/people/d3g293/mpich/mpich-3.3.2/install --with-device=ch4:ofi:sockets --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++ It’s been working pretty well but I recently tried to build mpich-3.3.2 and mpich-3.4a2 and although the build seems to work okay, I’m having problems actually running anything. If I run on 2 nodes the code seems to hang on MPI_Init and it looks like it is producing the error message [proxy:0:[email protected]] HYDU_sock_connect (utils/sock/sock.c:145): unable to connect from "node013.l ocal" to "node012.local" (Connection refused) [proxy:0:[email protected]] main (pm/pmiserv/pmip.c:183): unable to connect to server node012.local at p ort 37769 (check for firewalls!) srun: error: node013: task 1: Exited with exit code 5 If I run on a single node, things seem to work. Any idea what is going on here? I’ve got a working build of mpich-3.3, so things were okay up until recently. Has something in MPICH changed and my configuration formula is no good, or is this more likely to be due to some system modification? Bruce Palmer Senior Research Scientist Pacific Northwest National Laboratory Richland, WA 99352 (509) 375-3899

1 0

Running issue on CENTOS Cluster with SLURM
by Yaser Afshar 20 Apr '20

20 Apr '20

Hi, System: - AMD cluster (AMD Opteron(tm) Processor 6344) with InfiniPath_QLE7240 - CentOS Linux release 7.7.1908 (Core) Scheduler: - slurm MPICH version: 3.3.2 MPICH configuration: ./configure --prefix=/home/yaser/bin/mpich/3.3.2/intel CC=icc CXX=icpc FC=ifort --with-hwloc=/opt/HWLOC/2.2.0 --with-ucx=/opt/UCX/1.8.0 --with-knem=/opt/KNEM/1.1.3 --with-device=ch4:ucx --enable-mpi-cxx --enable-mpi1-compatibility --enable-threads=multiple --with-pmi --with-pm I am testing using the OSU microbenchmark ( http://mvapich.cse.ohio-state.edu/benchmarks) running the job on `srun` would succeed with no problem. ``` srun --mpi=pmi2 ./osu_bw ``` When I use the `mpiexec`, or `mpirun` it fails with: ``` [proxy:0:[email protected]] HYDU_sock_connect (utils/sock/sock.c:145): unable to connect from "pd-compute-3-40.local" to "pd-compute-1-6.local" (Connection refused) [proxy:0:[email protected]] main (pm/pmiserv/pmip.c:183): unable to connect to server pd-compute-1-6.local at port 42216 (check for firewalls!) srun: error: pd-compute-3-40: task 1: Exited with exit code 5 ``` The firewall is off, so it is not the reason. ``` > systemctl status firewalld ● firewalld.service Loaded: masked (/dev/null; bad) Active: inactive (dead) ``` I could not find any hint on MPICH FAQ nor anything useful anywhere else. Would you help me to resolve this issue? Many thanks.

1 0